Energy Stability for a Class of Semilinear Elliptic Problems

Abstract

In this paper, we consider semilinear elliptic problems in a bounded domain $\Omega$ contained in a given unbounded Lipschitz domain ${\mathcal {C}} \subset {\mathbb {R}}^N$. Our aim is to study how the energy of a solution behaves with respect to volume-preserving variations of the domain $\Omega$ inside ${\mathcal {C}}$. Once a rigorous variational approach to this question is set, we focus on the cases when ${\mathcal {C}}$ is a cone or a cylinder and we consider spherical sectors and radial solutions or bounded cylinders and special one-dimensional solutions, respectively. In these cases, we show both stability and instability results, which have connections with related overdetermined problems.

1 Introduction

Let ${\mathcal {C}} \subset {\mathbb {R}}^N$, $N \ge 2$, be an unbounded uniformly Lipschitz domain and let $\Omega \subset {\mathcal {C}}$ be a bounded Lipschitz domain with smooth relative boundary $\Gamma _\Omega \, {:}{=}\, \partial \Omega \cap {\mathcal {C}}$. More precisely, we assume that $\Gamma _\Omega$ is a smooth manifold of dimension $N-1$ with smooth boundary $\partial \Gamma _\Omega$. We set $\Gamma _{1, \Omega } \, {:}{=}\, \partial \Omega \setminus \overline{\Gamma }_\Omega$ and assume that ${\mathcal {H}}^{N - 1}(\Gamma _{1, \Omega }) > 0$, where ${\mathcal {H}}^{N - 1}$ denotes the $(N - 1)$-dimensional Hausdorff measure. Hence $\partial \Omega = \Gamma _\Omega \cup \Gamma _{1, \Omega } \cup \partial \Gamma _\Omega$.

We consider the following semilinear elliptic problem:

$$\begin{aligned} \left\{ \begin{array}{rcll} - \Delta u &{} = &{} f(u) &{} \quad \text { in } \Omega \\ u &{} = &{} 0 &{} \quad \text { on } \Gamma _\Omega \\ \displaystyle \frac{\partial u}{\partial \nu } &{} = &{} 0 &{} \quad \text { on } \Gamma _{1, \Omega } \end{array} \right. \end{aligned}$$

(1.1)

where $f: {\mathbb {R}} \rightarrow {\mathbb {R}}$ is a locally $C^{1,\alpha }$ nonlinearity and $\nu$ denotes the exterior unit normal vector to $\partial \Omega$.

Let $u_\Omega$ be a positive weak solution of (1.1) in the Sobolev space $H_0^1(\Omega \cup \Gamma _{1, \Omega })$, which is the space of functions in $H^1(\Omega )$ whose trace vanishes on $\Gamma _\Omega$. By standard variational methods we have that under suitable hypotheses on f such a solution exists and is a critical point of the energy functional

$$\begin{aligned} J(v) = \frac{1}{2} \int _\Omega |\nabla v|^2 \ dx - \int _\Omega F(v) \ dx, \quad v \in H_0^1(\Omega \cup \Gamma _{1, \Omega }), \end{aligned}$$

(1.2)

where $F(s) = \int _0^s f(\tau ) \ d\tau$.

A classical example of a nonlinearity for which a positive solution exists for any domain $\Omega$ in ${\mathcal {C}}$ is the Lane-Emden nonlinearity, namely

$$\begin{aligned} f(u) = u^p,\ \text {with} {\left\{ \begin{array}{ll} 1< p< \frac{N + 2}{N - 2} &{} \text {if} \, N \ge 3,\\ 1< p < + \infty &{}\text {if} \, N = 2.\end{array}\right. } \end{aligned}$$

(1.3)

In this case, $u_\Omega$ can be obtained, for instance, by minimizing the functional J on the Nehari manifold

$$\begin{aligned} {\mathcal {N}}(\Omega ) = \{v \in H_0^1(\Omega \cup \Gamma _{1, \Omega }) \setminus \{0\} \ : \ J'(v)[v] = 0\} . \end{aligned}$$

Given the unbounded region ${\mathcal {C}}$, an interesting question is to understand how the energy $J(u_\Omega )$ behaves with respect to variations of a domain $\Omega$ inside ${\mathcal {C}}$. In particular, one could ask whether the energy $J(u_\Omega )$ increases or decreases by deforming $\Omega$ into a domain ${{\widetilde{\Omega }}}$ sufficiently close to $\Omega$ and with the same measure.

Loosely speaking, one could consider the function $\Omega \mapsto T(\Omega ) = J(u_\Omega )$ and study it in a suitable “neighborhood” of $\Omega$. Under this aspect, domains $\Omega$ which are local minima of T could be particularly interesting. This question could be attacked by differentiating $T(\Omega )$ with respect to variations of $\Omega$ which leave the volume invariant and studying the stability or instability of its critical points. However, since (1.1) is a nonlinear problem and solutions of (1.1) are not unique in general, it is not clear a priori how to well define the functional $T(\Omega )$.

We will show in Sect. 2 that for nondegenerate solutions $u_\Omega$ of (1.1) the energy functional $T(\Omega )$ is well defined for domains obtained by small deformations of $\Omega$ induced by vector fields which leave ${\mathcal {C}}$ invariant.

We remark that the study of the stationary domains of the energy functional $T(\Omega )$ with a volume constraint is strictly related to the overdetermined problem obtained from (1.1) by adding the condition that the normal derivative $\frac{\partial u}{\partial \nu }$ is constant on $\Gamma _\Omega$, see Proposition 2.6. This is well-known for a Dirichlet problem in ${\mathbb {R}}^N$ and when $T(\Omega )$ is globally defined for all domains $\Omega \subset {\mathbb {R}}^N$ (as in the case of the torsion problem, i.e. $f \equiv 1$). It has been observed in [21] and [17] in the relative setting of the cone.

The existence or not of domains that are local minimizers of the energy and their shapes obviously depend on the unbounded region ${\mathcal {C}}$ where the domains $\Omega$ are contained. In this paper, we consider unbounded cones and cylinders, in which there are some particular domains that, for symmetry or other geometric reasons, could be natural candidates for being local minimizers of the energy.

Let us first describe the case when ${\mathcal {C}}$ is a cone $\Sigma _D$ defined as

$$\begin{aligned} \Sigma _D \, {:}{=}\, \{x \in {\mathbb {R}}^N \ : \ x = tq, \ q \in D, \ t > 0\}, \end{aligned}$$

(1.4)

where D is a smooth domain on the unit sphere ${\mathbb {S}^{N - 1}}$.

In $\Sigma _D$ we consider the spherical sector $\Omega _D$ obtained by intersecting the cone with the unit ball centered at the origin, i.e. $\Omega _D=\Sigma _D\cap B_1$. In $\Omega _D$ we can consider a radially symmetric solution $u_D$ of problem (1.1), for the nonlinearities f for which they exist. Obviously, $u_D$ is a radial solution of the analogous Dirichlet problem in the unit ball $B_1$.

In Sect. 3 we show that, whenever $u_D$ is a nondegenerate solution of (1.1), then the pair $(\Omega _D, u_D)$ is energy-stationary in the sense of Definition 2.4 and investigate its “stability” as a critical point of the energy functional T, which is well defined for small perturbations of $\Omega _D$. This means to investigate the sign of the quadratic form corresponding to the second domain derivative of T (see Sects. 2 and 3).

The main result we get is that the stability of $(\Omega _D, u_D)$ depends on the first nontrivial Neumann eigenvalue $\lambda _1(D)$ of the Laplace-Beltrami operator $- \Delta _{\mathbb {S}^{N - 1}}$ on the domain $D \subset \mathbb {S}^{N - 1}$ which spans the cone. In particular, we obtain a precise threshold for stability/instability which is independent of the nonlinearity, and on the radial positive solution considered, whenever multiple radial positive solutions exist. Let us remark that for several nonlinearities the radial positive solution is unique (see [19]). For example, this is the case if $f(u) = u^p$, $p > 1$.

To state precisely our result we need to introduce the first eigenvalue ${\widehat{\nu }}_1$ of the following singular eigenvalue problem:

$$\begin{aligned} {\left\{ \begin{array}{ll} - z'' - \frac{N - 1}{r}z' - f'(u_D)z = \frac{{\widehat{\nu }}}{r^2} z \quad \text {in } (0, 1) \\ z(1) = 0 \end{array}\right. } \end{aligned}$$

(1.5)

This problem arises naturally when studying the spectrum of the linearized operator $- \Delta - f'(u_D)$. We refer to Sect. 3 for more details.

Theorem 1.1

Let $\Sigma _D$ be the cone spanned by the smooth domain $D \subset \mathbb {S}^{N - 1}$, $N \ge 3$, and let $\lambda _1(D)$ be the first nonzero Neumann eigenvalue of the Laplace-Beltrami operator $- \Delta _{\mathbb {S}^{N - 1}}$ on D. Let $u_D$ be a radial positive solution of (1.1) in the spherical sector $\Omega _D$. We have:

• (i) if $- {\widehat{\nu }}_1< \lambda _1(D) < N - 1$, then the pair $(\Omega _D, u_D)$ is an unstable energy-stationary pair;

• (ii) if $\lambda _1(D) > N - 1$, then $(\Omega _D, u_D)$ is a stable energy-stationary pair.

Remark 1.2

The case $N = 2$ is special and in this case, the overdetermined torsion problem has been completely solved in [20] using that the boundary of any cone in dimension 2 is flat. In the nonlinear case, the condition $N \ge 3$ arises from the study of an auxiliary singular problem (see Proposition 3.12). It is important to observe that the singular eigenvalue ${\widehat{\nu }}_1$ which appears in (i) is larger than $-(N - 1)$ for all autonomous nonlinearities f(u) (see [7, Proposition 3.4]). Thus the condition $\lambda _1(D) \in (- {\widehat{\nu }}_1, N - 1)$ is consistent.

Remark 1.3

It is known that if D is a convex domain in ${\mathbb {S}}^{N - 1}$, then $\lambda _1(D) > N - 1$ (see [12, Theorem 4.3] or [2, Theorem 4.1]) and the same holds if D is almost convex ( [5]). On the other side, examples of domains D in the sphere for which (i) holds are provided in [17], Appendix A.

Let us comment on the meaning of Theorem 1.1. The statement (ii) will be proved by showing that the quadratic form corresponding to the second derivative of the energy functional, with a fixed volume constraint, is positive definite in all directions. This means that the spherical sector locally minimizes the energy among small volume preserving perturbations of $\Omega _D$ and of the corresponding radial solution $u_D$.

On the contrary, when $- {\widehat{\nu }}_1< \lambda _1(D) < N - 1$, by (i) we have that the pair $(\Omega _D, u_D)$ is unstable and therefore $\Omega _D$ is not a local minimizer of the energy. This means that there exist small volume preserving deformations of the spherical sector $\Omega _D$ which produce domains $\Omega _t$ and solutions $u_t$ of (1.1) in $\Omega _t$ whose energy $J(u_t)$ is smaller than the energy $J(u_D)$ of the positive radial solution $u_D$ in the spherical sector $\Omega _D$.

Moreover, observe that the function $f = f(s)$ could satisfy suitable hypotheses such that problem (1.1) has a unique positive solution $u_\Omega$ in any domain $\Omega \subset \Sigma _D$ (or more generally in $\Omega \subset {\mathcal {C}}$). This is the case, for example, when $f \equiv 1$, i.e., (1.1) is a “relative” torsion problem. Then the energy functional $T(\Omega ) = J(u_\Omega )$ is well defined for any domain $\Omega \subset \Sigma _D$. Hence we may ask whether a global minimum for T exists, once the volume of $\Omega$ is fixed, and is given by the spherical sector $\Omega _D$. This question has been addressed in [20, 21] and [17] when $f \equiv 1$, showing that $\Omega _D$ is a global minimizer if $\Sigma _D$ is a convex cone ( [21]), as a consequence of an isoperimetric inequality introduced in [18], see also [6, 14, 22]. Instead, in [17] it is proved that $\Omega _D$ is not a local minimizer whenever $\lambda _1(D) < N - 1$, which is the same threshold we get in Theorem 1.1 for general nonlinearities. It would be very interesting to find a domain $\Omega$ in $\Sigma _D$ which is a local minimizer for T, but not a global minimizer, at least for some nonlinearity for which T is globally defined. This seems to be a challenging question.

The other example of an unbounded domain we consider in the present paper is a half-cylinder, defined as

$$\begin{aligned} \Sigma _\omega \, {:}{=}\, \omega \times (0, + \infty ) \subset {\mathbb {R}}^N, \end{aligned}$$

(1.6)

where $\omega \subset {\mathbb {R}}^{N - 1}$ is a smooth bounded domain. We denote the points in $\Sigma _\omega$ by $x = (x', x_N)$, $x' \in \omega$. In this case, a geometrically simple domain we consider is the bounded cylinder

$$\begin{aligned} \Omega _\omega \, {:}{=}\, \{(x', x_N) \in {\mathbb {R}}^{N - 1} \ : \ x' \in \omega , \ 0< x_N < 1 \}. \end{aligned}$$

(1.7)

In $\Omega _\omega$ we consider a positive solution

$$\begin{aligned} u_\omega (x) = u_\omega (x_N) \end{aligned}$$

(1.8)

which is obtained by trivially extending to $\Omega _\omega$ a positive one-dimensional solution of the problem

$$\begin{aligned} {\left\{ \begin{array}{ll} - u'' = f(u) \quad \text { in } (0, 1) \\ u'(0) = u(1)=0 \end{array}\right. } \end{aligned}$$

(1.9)

for a nonlinearity f for which such a solution exists.

Before stating the results concerning the stability of the pair $(\Omega _\omega , u_\omega )$ we again consider an auxiliary eigenvalue problem (but not singular):

$$\begin{aligned} {\left\{ \begin{array}{ll} -z'' - f'(u_\omega ) z = \alpha z \quad \text { in } (0, 1) \\ z'(0) = z(1) = 0 \end{array}\right. } \end{aligned}$$

(1.10)

The problem (1.10) is considered in Sect. 4 to study the spectrum of the linearized operator $- \Delta - f'(u_\omega )$. We denote by $\alpha _1$ the first eigenvalue of (1.10).

We start by stating a sharp stability/instability result for the torsion problem, i.e., taking $f \equiv 1$ in (1.1).

Theorem 1.4

Let $\Sigma _\omega \subset {\mathbb {R}}^N$, $N \ge 2$, and $\Omega _\omega$ be respectively, as in (1.6) and (1.7), and let $u_\omega$ be the one-dimensional positive solution of (1.1) in $\Omega _\omega$ obtained by (1.9) for $f \equiv 1$. Let $\lambda _1(\omega )$ be the first nontrivial Neumann eigenvalue of the Laplace operator $- \Delta _{{\mathbb {R}}^{N - 1}}$ in the domain $\omega \subset {\mathbb {R}}^{N - 1}$. Then there exists a number $\beta \approx 1,439$ such that

• (i) if $\lambda _1(\omega ) < \beta$, then the pair $(\Omega _\omega , u_\omega )$ is an unstable energy-stationary pair;

• (ii) if $\lambda _1(\omega ) > \beta$, then the pair $(\Omega _\omega , u_\omega )$ is a stable energy-stationary pair.

Note that the number $\beta$ that gives the threshold for the stability is independent of the dimension N. Its value is obtained by solving numerically the equation $\sqrt{\lambda _1} \tanh (\sqrt{\lambda _1}) - 1 = 0$ (see (4.44) in the proof of Theorem 1.4).

It is interesting to observe that the instability result of Theorem 1.4 is related to a bifurcation theorem obtained in [13]. Indeed, if we consider the cylinder $\Sigma _\omega$ in ${\mathbb {R}}^2$, in which case $\omega$ is simply an interval in ${\mathbb {R}}$ and $\Omega _\omega$ is a rectangle, a byproduct of Theorem 1.1 of [13] is the existence of a domain $\widetilde{\Omega }_\omega$ in $\Sigma _\omega$ that is a small deformation of the rectangle $\Omega _\omega$ and in which the overdetermined problem

$$\begin{aligned} \left\{ \begin{array}{rcll} - \Delta u &{} = &{} 1 &{} \quad \text { in } {{\widetilde{\Omega }}}_\omega \\ u &{} = &{} 0 &{} \quad \text { on } \Gamma _{{{\widetilde{\Omega }}}_\omega } \\ \frac{\partial u}{\partial \nu } &{} = &{} c < 0 &{} \quad \text { on } \Gamma _{{{\widetilde{\Omega }}}_\omega } \\ \frac{\partial u}{\partial \nu } &{} = &{} 0 &{} \quad \text { on } \Gamma _{1, \widetilde{\Omega }_\omega } \end{array} \right. \end{aligned}$$

has a solution.

By looking at the proof of [13] and relating it to our instability result it is clear that the bifurcation should occur when the eigenvalue $\lambda _1(\omega )$ crosses the value $\beta$ provided by Theorem 1.4.

The proof of Theorem 1.4 can be derived from a general condition for the stability of the pair $(\Omega _\omega , u_\omega )$ in the nonlinear case, which is obtained in Theorem 4.11. The proof of Theorem 4.11 involves auxiliary functions that appear naturally in the study of derivatives of the energy functional T, see Sect. 4.

Let us remark that in the case when $f \equiv 1$ we succeed in obtaining the sharp bound of Theorem 1.4 because the solution given by (1.8) and (1.9) is explicit:

$$\begin{aligned} u_\omega (x) = u_\omega (x_N) = \frac{1 - x_N^2}{2}, \end{aligned}$$

and so are the auxiliary functions which are solutions of simple linear ODEs. This allows us to use the condition of Theorem 4.11 to obtain Theorem 1.4.

The result of Theorem 1.4 gives a striking difference between the torsional energy problem and the isoperimetric problem in cylinders. Indeed, Proposition 2.1 of [1] shows that the only stationary cartesian graphs for the perimeter functional are the flat ones. Instead, Theorem 1.4 (as well as the result of [13]) indicate that there are domains for which the overdetermined problem relative to (1.1), with $f \equiv 1$, has a solution and whose relative boundary is a non-flat cartesian graph.

For the semilinear problem, we obtain a stability result for a large class of nonlinearities as soon as the eigenvalue $\lambda _1(\omega )$ is sufficiently large. Indeed, we have

Theorem 1.5

Let $\Sigma _\omega$ and $\Omega _\omega$ be as in (1.6) and (1.7), and let $u_\omega$ be a positive one-dimensional solution of (1.1) in $\Omega _\omega$. Let $\alpha _1$ be the first eigenvalue of (1.10) and let $\lambda _1(\omega )$ be as in Theorem 1.4. If the nonlinearity f satisfies $f(0) = 0$ and

$$\begin{aligned} \lambda _1(\omega ) > \max \{- \alpha _1, \Vert f'(u_\omega )\Vert _\infty \}, \end{aligned}$$

(1.11)

then the pair $(\Omega _\omega , u_\omega )$ is a stable energy-stationary pair.

The condition (1.11) shows that the stability depends on an interplay between the geometry of the cylinder $\Sigma _\omega$ (through the eigenvalue $\lambda _1(\omega )$) and the nonlinearity f. On the contrary, numerical evidence shows, for the Lane-Emden nonlinearity (1.3), that, if $\lambda _1$ is sufficiently close to $-\alpha _1$, instability occurs, see Remark 4.13.

Concerning the eigenvalue $\alpha _1$ in the bound (1.11), as well as the analogous one, $\lambda _1(D) > - {\widehat{\nu }}_1$, of Theorem 1.1, we point out that they are used in the proofs of both theorems to deduce the positivity of some auxiliary functions. It is an open problem to understand if they really play a role in the stability/instability result.

We delay further comments on the results and their proofs to the respective sections.

The paper is organized as follows. In Sect. 2 we study problem (1.1) in domains $\Omega$ contained in a general unbounded set $\mathcal C$. We define the energy functional and its derivative with respect to variations of $\Omega$ which leave ${\mathcal {C}}$ invariant and preserve the measure of $\Omega$. This is done by considering nondegenerate solutions of (1.1) in $\Omega$.

In Sect. 3 we consider the case when ${\mathcal {C}}$ is a cone $\Sigma _D$. In this setting we take domains which are defined by smooth radial graphs over D, in particular we consider the spherical sector $\Omega _D$ and a corresponding radial solution $u_D$ for which we prove the stability/instability result.

Finally in Sect. 4 we study the case of the cylinder $\Sigma _\omega$ and prove the corresponding stability/instability result for the pair $(\Omega _\omega , u_\omega )$ when $\Omega _\omega$ is a bounded cylinder and $u_\omega$ is as in (1.8) and (1.9).

2 Semilinear Elliptic Problems in Unbounded Sets

In this section we consider problem (1.1) in a bounded Lipschitz domain $\Omega$ contained in an unbounded open set ${\mathcal {C}}$ which we assume to be (uniformly) Lipschitz regular.

Starting from a positive nondegenerate solution of (1.1) in $\Omega$ we show how to define an energy functional for small variations of $\Omega$ which preserve the volume.

2.1 Nondegenerate Solutions

Let $\Omega \subset {\mathcal {C}}$ be a bounded domain whose relative boundary $\Gamma _\Omega = \partial \Omega \cap {\mathcal {C}}$ is a smooth manifold (with boundary). As in Sect. 1 we set $\Gamma _{1, \Omega } = \partial \Omega \setminus {\overline{\Gamma }_\Omega }$.

We consider a positive weak solution $u_\Omega$ of (1.1) in the Sobolev space $H_0^1(\Omega \cup \Gamma _{1, \Omega })$, which is the subspace of $H^1(\Omega )$ of functions whose trace vanishes on $\Gamma _\Omega$. By standard variational methods, such as constrained minimization, Mountain-Pass Theorem etc, it is easy to exhibit many nonlinearities $f = f(s)$ for which such a solution exists. Moreover, with suitable assumptions on the growth of f we also have, by regularity results, that $u_\Omega$ is a classical solution of (1.1) inside $\Omega$ and at any regular point of $\partial \Omega$, and that $u_\Omega$ is bounded (see also [7, Proposition 3.1]).

We assume that $u_\Omega$ is nondegenerate, i.e., the linearized operator

$$\begin{aligned} L_{u_\Omega } = - \Delta - f'(u_\Omega ) \end{aligned}$$

(2.1)

does not have zero as an eigenvalue in $H_0^1(\Omega \cup \Gamma _{1, \Omega })$ or, in other words, $L_{u_\Omega }$ defines an isomorphism between $H_0^1(\Omega \cup \Gamma _{1, \Omega })$ and its dual space. We consider small deformations of $\Omega$ which leave ${\mathcal {C}}$ invariant and would like to show that the nondegeneracy of $u_\Omega$ induces a local uniqueness result for solutions of (1.1) in the deformed domains. Thus we take a one-parameter family of diffeomorphisms $\xi _t$, for $t \in (- \eta , \eta )$, $\eta > 0$, associated to a smooth vector field V such that $V(x) \in T_x\partial {\mathcal {C}}$ for every $x \in \partial \mathcal C^{\textrm{reg}}$, $V(x)=0$ for $x\in \partial \mathcal C\setminus \partial {\mathcal {C}}^{\textrm{reg}}$, and set $\Omega _t:= \xi _t(\Omega )$, where $T_x\partial {\mathcal {C}}$ denotes the tangent space to $\partial {\mathcal {C}}$ at the point x, and $\partial {\mathcal {C}}^{\textrm{reg}}$ denotes the regular part of $\partial {\mathcal {C}}$. In particular $\Omega _0=\Omega$ and in order to simplify the notations we set

$$\begin{aligned} \Gamma _t \, {:}{=}\, \Gamma _{\Omega _t}, \quad \Gamma _{1, t} \, {:}{=}\, \Gamma _{1, \Omega _t}. \end{aligned}$$

(2.2)

Proposition 2.1

Let $u_\Omega$ be a positive nondegenerate solution of (1.1) which belongs to $W^{1, \infty }(\Omega ) \cap W^{2, 2}(\Omega )$. Let V be a smooth vector field and let $\xi _t$ be the associated family of diffeomorphisms. Then there exists $\delta > 0$ such that for any $t \in (- \delta , \delta )$ there is a unique solution $u_t$ of the problem

$$\begin{aligned} \left\{ \begin{array}{rcll} - \Delta u &{} = &{} f(u) &{} \quad \text { in } \Omega _t \\ u &{} = &{} 0 &{} \quad \text { on } \Gamma _t \\ \displaystyle \frac{\partial u}{\partial \nu } &{} = &{} 0 &{} \quad \text { on } \Gamma _{1, t} \end{array} \right. \end{aligned}$$

(2.3)

in a neighborhood of the function $u_\Omega \circ \xi _t^{-1}$ in the space $H_0^1(\Omega _t \cup \Gamma _{1, t})$. Moreover, the map $t \mapsto u_t$ is differentiable.

Proof

By using the diffeomorphism $\xi _t$ we can pass from the space $H_0^1(\Omega \cup \Gamma _{1, \Omega })$ to the space $H_0^1(\Omega _t \cup \Gamma _{1, t})$. Indeed,

$$\begin{aligned} H_0^1(\Omega \cup \Gamma _{1, \Omega }) = \{v \circ \xi _t \ : \ v \in H_0^1(\Omega _t \cup \Gamma _{1, t})\}. \end{aligned}$$

(2.4)

Moreover, $u_t$ is a weak solution of (2.3), i.e.,

$$\begin{aligned} \int _{\Omega _t} \nabla u_t \cdot \nabla v \ dx - \int _{\Omega _t} f(u_t) v \ dx = 0 \quad \forall v \in H_0^1(\Omega _t \cup \Gamma _{1, t}) \end{aligned}$$

if and only if the function ${\widehat{u}}_t = u_t \circ \xi _t \in H_0^1(\Omega \cup \Gamma _{1, \Omega })$ satisfies

$$\begin{aligned} \int _\Omega (M_t \nabla \widehat{u}_t)\cdot \nabla w J_t \ dx - \int _\Omega f({\widehat{u}}_t) w J_t \ dx = 0 \quad \forall w \in H_0^1(\Omega \cup \Gamma _{1, \Omega }) \end{aligned}$$

(2.5)

where

$$\begin{aligned} J_t(x) = \left| \det ({{\,\textrm{Jac}\,}}\xi _t(x))\right| \end{aligned}$$

and

$$\begin{aligned} M_t = [{{\,\textrm{Jac}\,}}\xi _t^{-1}(\xi _t(x))][{{\,\textrm{Jac}\,}}\xi _t^{-1}(\xi _t(x))]^T. \end{aligned}$$

(2.6)

In other words, setting ${\widehat{M}}_t:=M_t J_t$, we have that ${\widehat{u}}_t$ is a solution of

$$\begin{aligned} - {{\,\textrm{div}\,}}({\widehat{M}}_t \nabla {\widehat{u}}_t) - f(\widehat{u}_t)J_t = 0 \end{aligned}$$

in the space $H_0^1(\Omega \cup \Gamma _{1, \Omega })$. Now we consider the map

$$\begin{aligned} {\mathcal {F}}: (- \eta , \eta ) \times H_0^1(\Omega \cup \Gamma _{1, \Omega }) \rightarrow H_0^1(\Omega \cup \Gamma _{1, \Omega })^* \end{aligned}$$

defined as

$$\begin{aligned} {\mathcal {F}}(t, v) = - {{\,\textrm{div}\,}}({\widehat{M}}_t \nabla v) - f(v) J_t. \end{aligned}$$

(2.7)

Since $u_\Omega$ is a solution in $\Omega$ and $\xi _0$ is the identity map we have

$$\begin{aligned} {\mathcal {F}}(0, u_\Omega ) = 0. \end{aligned}$$

Notice that ${\mathcal {F}}$ is differentiable with respect to to v, and

$$\begin{aligned} \partial _v {\mathcal {F}}(0, u_\Omega ) = - \Delta - f'(u_\Omega ). \end{aligned}$$

(2.8)

Indeed, for any $h \in H_0^1(\Omega \cup \Gamma _{1, \Omega })$ we have

$$\begin{aligned} \frac{{\mathcal {F}}(t, v + \varepsilon h) - {\mathcal {F}}(t, v)}{\varepsilon }&= \frac{- {{\,\textrm{div}\,}}({\widehat{M}}_t(\nabla v + \varepsilon \nabla h) ) - f(v + \varepsilon h)J_t - (-{{\,\textrm{div}\,}}({\widehat{M}}_t \nabla v) - f(v)J_t)}{\varepsilon } \nonumber \\&= - \frac{{{\,\textrm{div}\,}}(\varepsilon {\widehat{M}}_t \nabla h)}{\varepsilon } - \frac{(f(v + \varepsilon h) - f(v))J_t}{\varepsilon } \nonumber \\&\rightarrow - {{\,\textrm{div}\,}}({\widehat{M}}_t \nabla h) - f'(v)J_t \end{aligned}$$

as $\varepsilon \rightarrow 0$. Hence ${\mathcal {F}}$ is differentiable and evaluating $\partial _v {\mathcal {F}}$ at $(0, u_\Omega )$ we obtain (2.8).

By the nondegeneracy assumption on the solution $u_\Omega$, we infer that (2.8) gives an isomorphism between $H_0^1(\Omega \cup \Gamma _{1, \Omega })$ and $H_0^1(\Omega \cup \Gamma _{1, \Omega })^*$. Then, by the Implicit Function Theorem, there exists an interval $(- \delta , \delta )$ and a neighborhood ${\mathcal {B}}$ of $u_\Omega$ in $H_0^1(\Omega \cup \Gamma _{1, \Omega })$ such that for every $t \in (- \delta , \delta )$ there exists a unique function ${\widehat{u}}_t \in H_0^1(\Omega \cup \Gamma _{1, \Omega })$ in ${\mathcal {B}}$ such that ${\mathcal {F}}(t, {\widehat{u}}_t) = 0$, that is, ${\widehat{u}}_t$ is the unique solution (in ${\mathcal {B}}$) of (2.5). It follows that $u_t = {\widehat{u}}_t \circ \xi _t^{-1}$ is the unique solution of (2.3) in a neighborhood of $u_\Omega \circ \xi _t^{-1}$ in $H_0^1(\Omega _t \cup \Gamma _{1, t})$.

Finally, since the map $t \mapsto {\widehat{u}}_t$ is smooth, so is the map $t \mapsto u_t$. In addition

$$\begin{aligned} {\widetilde{u}} \, {:}{=}\, \left. \frac{d}{dt} \right| _{t = 0} u_t = \left( \left. \frac{d}{dt} \right| _{t = 0}{\widehat{u}}_t\right) - \langle \nabla u_\Omega , V\rangle . \end{aligned}$$

(2.9)

The proof is complete. $\square$

Note that, as for $u_\Omega$, $u_t$ is a classical solution of (2.3) in $\Omega _t$ and on the regular part of $\partial \Omega _t$. By Proposition 2.1 we have that the energy functional

$$\begin{aligned} T(\Omega _t) = J(u_t) = \frac{1}{2} \int _{\Omega _t} |\nabla u_t|^2 \ dx - \int _{\Omega _t} F(u_t) \ dx, \end{aligned}$$

(2.10)

where $F(s) = \int _0^s f(\tau ) \ d\tau$, is well defined for all sufficiently small t. Observe that, since $u_t$ is a solution to (2.3), we have

$$\begin{aligned} \int _{\Omega _t} |\nabla u_t|^2 \ dx = \int _{\Omega _t} f(u_t)u_t \ dx, \end{aligned}$$

so we can also write

$$\begin{aligned} T(\Omega _t) = \frac{1}{2} \int _{\Omega _t} f(u_t)u_t \ dx - \int _{\Omega _t} F(u_t) \ dx. \end{aligned}$$

(2.11)

In the next result we show that T is differentiable with respect to t and compute its derivative at $t = 0$, that is, at the initial domain $\Omega$.

Proposition 2.2

Assume that $u_\Omega$ is a positive nondegenerate solution of (1.1) which belongs to $W^{1, \infty }(\Omega ) \cap W^{2, 2}(\Omega )$. Then

$$\begin{aligned} \left. \frac{d}{dt} \right| _{t = 0} T(\Omega _t) = - \frac{1}{2} \int _{\Gamma _\Omega } |\nabla u_\Omega |^2 \langle V, \nu \rangle \ d\sigma . \end{aligned}$$

(2.12)

Proof

Recall from Proposition 2.1 that $t \mapsto u_t$ is smooth and (2.9) holds. Differentiating the equation $- \Delta u_t = f(u_t)$ with respect to t we obtain

$$\begin{aligned} - \Delta {\widetilde{u}} = f'(u_\Omega ) {\widetilde{u}} \quad \text { in } \Omega . \end{aligned}$$

(2.13)

Now observe that by the hypotheses on $u_\Omega$ we have that

$$\begin{aligned} {\widetilde{u}} + \langle \nabla u_\Omega , V \rangle = \left( \left. \frac{d}{dt} \right| _{t = 0} {\widehat{u}}_t\right) \in H_0^1(\Omega \cup \Gamma _{1,\Omega }), \end{aligned}$$

(2.14)

thus

$$\begin{aligned} {\widetilde{u}} = - \frac{\partial u_\Omega }{\partial \nu } \langle V, \nu \rangle \quad \text { on } \Gamma _\Omega . \end{aligned}$$

(2.15)

Finally, since $\xi _t$ maps $\partial {\mathcal {C}}$ into itself we have that, for all small t and $x \in (\partial \mathcal C\cap \partial \Omega )^{\textrm{reg}}$

$$\begin{aligned} \langle \nabla u_t(\xi _t(x)), \nu (\xi _t(x)) \rangle = 0. \end{aligned}$$

Differentiating this relation with respect to t and evaluating at $t=0$ we obtain

$$\begin{aligned} 0 = \langle \nabla {\widetilde{u}}(x), \nu (x) \rangle + d_x(\langle \nabla u_\Omega , \nu \rangle )[V(x)], \end{aligned}$$

where $d_x(\langle \nabla u_\Omega , \nu \rangle )[V(x) ]$ is the differential of the function $\langle \nabla u_\Omega , \nu \rangle \big |_{(\partial \mathcal C\cap \partial \Omega )^{\textrm{reg}}}$ computed at x, along V(x). Then, since $\langle \nabla u_\Omega , \nu \rangle =0$ on $(\partial {\mathcal {C}}\cap \partial \Omega )^{\textrm{reg}}$, and in view of (2.13), (2.15), we infer that ${\widetilde{u}}$ satisfies

$$\begin{aligned} \left\{ \begin{array}{rcll} - \Delta {\widetilde{u}} &{} = &{} f'(u_\Omega ) {\widetilde{u}} &{} \quad \text { in } \Omega \\ {\widetilde{u}} &{} = &{} \displaystyle - \frac{\partial u_\Omega }{\partial \nu } \langle V, \nu \rangle &{} \quad \text { on } \Gamma _\Omega \\ \displaystyle \frac{\partial {\widetilde{u}}}{\partial \nu } &{} = &{} 0 &{} \quad \text { on }\ \Gamma _{1,\Omega } \end{array} \right. \end{aligned}$$

(2.16)

in the classical sense in the interior of $\Omega$ and on the regular part of $\partial \Omega$.

Recalling (2.11) we can write

$$\begin{aligned} T(\Omega _t) = \int _{\Omega _t} \frac{1}{2} \left( f(u_t)u_t - F(u_t) \right) \ dx. \end{aligned}$$

Since $t\mapsto f(u_t)u_t - F(u_t)$ is differentiable at $t=0$, $\partial \Omega$ is Lipschitz and taking into account that $u_\Omega \in W^{1, \infty }(\Omega ) \cap W^{2, 2}(\Omega )$, then, applying [16, Theorem 5.2.2], we can compute the derivative with respect to t of the functional T obtaining that

$$\begin{aligned} \left. \frac{d}{dt}\right| _{t = 0} T(\Omega _t)&= \frac{1}{2} \int _\Omega (f'(u_\Omega ){\widetilde{u}} u_\Omega + f(u_\Omega ) {\widetilde{u}}) \ dx - \int _\Omega f(u_\Omega ) {\widetilde{u}} \ dx \nonumber \\&\quad + \int _{\partial \Omega } \left( \frac{1}{2}f(u_\Omega )u_\Omega - F(u_\Omega )\right) \langle V, \nu \rangle \ d \sigma \nonumber \\&= \frac{1}{2} \int _\Omega \left( f'(u_\Omega ) {\widetilde{u}} u_\Omega - f(u_\Omega ){\widetilde{u}} \right) \ dx \nonumber \\&= \frac{1}{2} \int _\Omega \left( (- \Delta {\widetilde{u}}) u_\Omega + \Delta u_\Omega {\widetilde{u}} \right) \ dx \nonumber \\&= \frac{1}{2} \int _{\partial \Omega } \left( {\widetilde{u}} \frac{\partial u_\Omega }{\partial \nu } - u_\Omega \frac{\partial {\widetilde{u}}}{\partial \nu } \right) \ d \sigma \nonumber \\&= - \frac{1}{2} \int _{\Gamma _\Omega } |\nabla u_\Omega |^2 \langle V, \nu \rangle \ d \sigma . \end{aligned}$$

(2.17)

The previous applications of the Divergence Theorem are justified by arguing as in [20, Lemma 2.1], where the regularity hypothesis on $u_\Omega$ comes into play. $\square$

Remark 2.3

It is not difficult to see that ${\widetilde{u}}$ is also a weak solution of (2.16). Indeed, let $\varphi \in C_c^\infty (\Omega \cup \Gamma _{1,\Omega })$. Then, for all sufficiently small t, we also have $\varphi \in C_c^\infty (\Omega _t \cup \Gamma _{1, t})$. Hence, since $u_t$ is a weak solution to (2.3), we have

$$\begin{aligned} 0 = \int _{\Omega _t} \nabla u_t \nabla \varphi \ dx - \int _{\Omega _t} f(u_t) \varphi \ dx = \int _{\Omega } \nabla u_t \nabla \varphi \ dx - \int _{\Omega } f(u_t) \varphi \ dx. \end{aligned}$$

(2.18)

Now, as proved in [17, Claim (3.17)], it holds that

$$\begin{aligned} \left. \frac{d}{dt} \right| _{t = 0} \nabla u_t = \nabla \widetilde{u}. \end{aligned}$$

Then, taking the derivative with respect to t in (2.18), evaluating at $t=0$, and since $\varphi$ is arbitrary, we easily conclude.

Let us now consider domains $\Omega \subset {\mathcal {C}}$ of fixed measure $c>0$ and define

$$\begin{aligned} {\mathcal {A}} \, {:}{=}\, \{\Omega \subset {\mathcal {C}} \ : \ \Omega \text { is admissible and } |\Omega | = c\}, \end{aligned}$$

(2.19)

where admissible means that $\Omega \subset {\mathcal {C}}$ is a bounded domain with smooth relative boundary $\Gamma _\Omega \, {:}{=}\, \partial \Omega \cap {\mathcal {C}}$, $\partial \Gamma _\Omega$ is a smooth $(N-2)$-dimensional manifold and $\Gamma _{1, \Omega } \, {:}{=}\, \partial \Omega \setminus {\overline{\Gamma }}_\Omega$ is such that ${\mathcal {H}}^{N - 1}(\Gamma _{1, \Omega }) > 0$. We consider vector fields that induce deformations that preserve the volume. More precisely we take a one-parameter family of diffeomorphisms $\xi _t$, $t\in (-\eta ,\eta )$, associated to a smooth vector field V such that $V(x) \in T_x\partial {\mathcal {C}}^\textrm{reg}$ for all $x \in \partial {\mathcal {C}}^\textrm{reg}$, and satisfying the condition $|\Omega _t| = |\Omega |$, for all $t\in (-\eta ,\eta )$, where $\Omega _t = \xi _t(\Omega )$.

Definition 2.4

We say that the pair $(\Omega , u_\Omega )$ is energy-stationary under a volume constraint if

$$\begin{aligned} \left. \frac{d}{dt}\right| _{t = 0} T(\Omega _t) = 0 \end{aligned}$$

(2.20)

for any vector field tangent to $\partial {\mathcal {C}}$ such that the associated one-parameter family of diffeomorphisms preserves the volume.

Definition 2.5

Let $(\Omega , u_\Omega )$ be an energy-stationary pair under a volume constraint. We say that it is stable if, for any volume-preserving vector field V, the second derivative

$$\begin{aligned} \left. \frac{d^2}{dt^2} \right| _{t = 0} T(\Omega _t) \end{aligned}$$

is positive.

Since the computation of second domain derivatives is quite involved, we do not present a general formula. Explicit expressions are given in Sects. 3 and 4, in the special cases of cones and cylinders.

A characterization of energy-stationary pairs in ${\mathcal {C}}$ is the following:

Proposition 2.6

Let $\Omega \in {\mathcal {A}}$ and assume that $u_\Omega \in W^{1, \infty }(\Omega ) \cap W^{2, 2}(\Omega )$ is a nondegenerate positive solution of (1.1). Then $(\Omega , u_\Omega )$ is energy-stationary under a volume constraint if and only if $u_\Omega$ satisfies the overdetermined condition $|\nabla u_\Omega | = \text {constant}$ on $\Gamma _\Omega$.

Proof

Let $\xi _t$ be an arbitrary admissible one-parameter family of diffeomorphisms and let V be the associated vector field. Since the volume is preserved and $V(x) \in T_x\partial {\mathcal {C}}$ on $\partial {\mathcal {C}}$,

$$\begin{aligned} 0 = \left. \frac{d}{dt} \right| _{t = 0} |\Omega _t| = \int _{\partial \Omega } \langle V, \nu \rangle \ d \sigma = \int _{\Gamma _{\Omega }} \langle V, \nu \rangle \ d \sigma . \end{aligned}$$

(2.21)

If $|\nabla u_\Omega |$ is constant on $\Gamma _{\Omega }$, then $(\Omega , u_\Omega )$ is energy-stationary, in view of (2.12) and (2.21). On the other hand, if $(\Omega , u_\Omega )$ is energy stationary, then

$$\begin{aligned} \int _{\Gamma _{\Omega }} (|\nabla u_\Omega |^2 - a) \langle V, \nu \rangle \ d \sigma = 0 \end{aligned}$$

(2.22)

for every constant a and every admissible vector field V. Assume by contradiction that $|\nabla u_\Omega |$ is not constant on $\Gamma _{\Omega }$. Then there exists a compact set $K \subset \Gamma _{\Omega }$, with nonempty interior part, such that $|\nabla u_\Omega |$ is not constant on K. Take a nonnegative cutoff function $\Theta$ such that $\Theta \equiv 1$ in K, and choose

$$\begin{aligned} a = \frac{\int _{\Gamma _\Omega } \Theta |\nabla u_\Omega |^2 \ d\sigma }{\int _{\Gamma _\Omega } \Theta \ d\sigma }. \end{aligned}$$

(2.23)

Then we can build a deformation from the vector field $V = (|\nabla u_\Omega |^2 - a) \Theta \nu$, and in this case, since $(\Omega , u_\Omega )$ is energy stationary, we would have

$$\begin{aligned} \int _K (|\nabla u_\Omega |^2 - a)^2 \ d \sigma = 0, \end{aligned}$$

(2.24)

which contradicts the fact that $|\nabla u_\Omega |$ is not constant on K. The proof is complete. $\square$

Remark 2.7

It is relevant to observe that all concepts introduced in this section apply to the case when $\Gamma _{1, \Omega }$ is empty, or, equivalently, when ${\mathcal {C}} = {\mathbb {R}}^N$. Thus all the above results hold for Dirichlet problems in domains in the whole space. In this case it is known, by Serrin’s Theorem (see [23]) that if a positive solution for the overdetermined problem

$$\begin{aligned} \left\{ \begin{array}{rcll} - \Delta u &{} = &{} f(u) &{} \quad \text { in } \Omega \\ u &{} = &{} 0 &{} \quad \text { on } \partial \Omega \\ \displaystyle \frac{\partial u}{\partial \nu } &{} = &{} \text {constant} &{} \quad \text { on } \partial \Omega \end{array} \right. \end{aligned}$$

(2.25)

exists, then $\Omega$ is a ball. Therefore, in view of Proposition 2.6, it follows that the only energy-stationary pairs in ${\mathbb {R}}^N$ are $(B, u_B)$, where B is a ball and $u_B$ is a nondegenerate positive solution.

Remark 2.8

We observe that all the results in this section hold true also for non-degenerate sign-changing solutions $u_\Omega$ to (1.1). However, since in the sequel we study the stability in the case of positive solutions, we have considered only this case

3 The Case of the Cone

Let $D \subset \mathbb {S}^{N - 1}$ be a smooth domain on the unit sphere and let $\Sigma _D$ be the cone spanned by D, which is defined as

$$\begin{aligned} \Sigma _D \, {:}{=}\, \{x \in {\mathbb {R}}^N \ : \ x = t q, \ q \in D, \ t > 0\}. \end{aligned}$$

(3.1)

In $\Sigma _D$ we consider admissible domains $\Omega$, in the sense of (2.19), that are strictly star-shaped with respect to the vertex of the cone, which we choose to be the origin 0 in ${\mathbb {R}}^N$. In other words, we consider domains whose relative boundary is the radial graph in $\Sigma _D$ of a function in $C^2({\overline{D}}, {\mathbb {R}})$. Hence for $\varphi \in C^2({\overline{D}}, {\mathbb {R}})$ we set

$$\begin{aligned} \Gamma _\varphi \, {:}{=}\, \{x \in {\mathbb {R}}^N \ : \ x = e^{\varphi (q)}q, \ q \in D\} \end{aligned}$$

(3.2)

and consider the strictly star-shaped domain $\Omega _\varphi$ defined as

$$\begin{aligned} \Omega _\varphi \, {:}{=}\, \{x \in {\mathbb {R}}^N \ : \ x = tq, \ 0< t < e^{\varphi (q)}, \ q \in D\}. \end{aligned}$$

(3.3)

To simplify the notation we set

$$\begin{aligned} \Gamma _{1, \varphi } \, {:}{=}\, \Gamma _{1, \Omega _\varphi } = \partial \Omega _\varphi \setminus {\overline{\Gamma }}_\varphi . \end{aligned}$$

3.1 Energy Functional for Star-Shaped Domains

In $\Omega _\varphi$ we consider the semilinear elliptic problem

$$\begin{aligned} \left\{ \begin{array}{rcll} - \Delta u &{} = &{} f(u) &{} \quad \text { in } \Omega _\varphi \\ u &{} = &{} 0 &{} \quad \text { on } \Gamma _\varphi \\ \displaystyle \frac{\partial u}{\partial \nu } &{} = &{} 0 &{} \quad \text { on } \Gamma _{1, \varphi }\setminus \{0\} \end{array} \right. \end{aligned}$$

(3.4)

and assume throughout this section that a bounded positive nondegenerate solution $u_{\Omega _\varphi }$ exists and belongs to $W^{1, \infty }(\Omega _\varphi ) \cap W^{2, 2}(\Omega _\varphi )$. Then we can apply the results of Sect. 2 and define the energy functional T as in (2.10) for small variations of $\Omega _\varphi$. Since $\Omega _\varphi$ is strictly star-shaped, this property also holds for the domains obtained by small regular deformations. Thus it is convenient to parametrize the domains and their variations by $C^2$ functions defined on ${\overline{D}}$. Hence, for $v \in C^2({\overline{D}}, {\mathbb {R}})$ and $t \in (- \eta , \eta )$, where $\eta > 0$ is a fixed number sufficiently small, we consider the domain variations $\Omega _{\varphi + tv} \subset \Sigma _D$.

Let $\xi :(- \eta , \eta ) \times \overline{\Sigma }_D\setminus \{0\} \rightarrow \overline{\Sigma }_D\setminus \{0\}$ be the map defined by

$$\begin{aligned} \xi (t, x) = e^{t v\left( \frac{x}{|x|} \right) }x. \end{aligned}$$

(3.5)

Then $\xi |_{\Omega _\varphi }(t, \cdot ) : \Omega _\varphi \rightarrow \Omega _{\varphi + tv}$ is a diffeomorphism, whose inverse is

$$\begin{aligned} (\xi |_{\Omega _\varphi })^{-1}(t, x) = e^{-t v \left( \frac{x}{|x|} \right) }x = \xi (-t, x). \end{aligned}$$

(3.6)

By definition, $\xi (t, x) \in \partial \Sigma _D \setminus \{0\}$ for all $(t, x) \in (- \eta , \eta ) \times (\partial \Sigma _D \setminus \{0\})$ and $\xi$ is the flow associated to the vector field

$$\begin{aligned} V(x) = v\left( \frac{x}{|x|} \right) x, \end{aligned}$$

(3.7)

since $\xi (0, x) = x$ and

$$\begin{aligned} \frac{d}{dt} \xi (t, x) = e^{tv\left( \frac{x}{|x|} \right) } v\left( \frac{x}{|x|} \right) x = V(\xi (t, x)). \end{aligned}$$

The energy functional T in (2.10) becomes a functional defined on functions in $C^2({\overline{D}}, {\mathbb {R}})$. More precisely, we define, for every $v \in C^2({\overline{D}}, {\mathbb {R}})$,

$$\begin{aligned} T(\varphi + tv) := T(\Omega _{\varphi + tv}) = J(u_{\varphi + tv}), \end{aligned}$$

(3.8)

for $t \in (- \delta , \delta )$ with $\delta > 0$ small, where

$$\begin{aligned} u_{\varphi + tv} \, {:}{=}\, u_{\Omega _{\varphi + tv}} \end{aligned}$$

is the unique positive solution of (3.4) in the domain $\Omega _{\varphi + tv}$, in a neighborhood of $u_\varphi \circ \xi (t, \cdot )^{-1}$.

We now compute the first derivative of the functional T at $\varphi$ along a direction $v \in C^2({\overline{D}}, {\mathbb {R}})$, i.e. the derivative with respect to t of (3.8) computed at $t=0$.

Lemma 3.1

Let $\varphi \in C^2({\overline{D}}, {\mathbb {R}})$ and assume that $u_\varphi$ is a bounded positive nondegenerate solution to (3.4) and that $u_\varphi$ belongs to $W^{1, \infty }(\Omega _\varphi ) \cap W^{2, 2}(\Omega _\varphi )$. Then for any $v \in C^2({\overline{D}}, \mathbb R)$ it holds that

$$\begin{aligned} T'(\varphi )[v] = - \frac{1}{2} \int _D \left( \frac{\partial u_\varphi }{\partial \nu }(e^\varphi q) \right) ^2 e^{N \varphi } v \ d \sigma \end{aligned}$$

(3.9)

Proof

The result follows from Proposition 2.2. Indeed, since the exterior unit normal to $\Gamma _\varphi$ is given by

$$\begin{aligned} \nu (x)= \frac{\frac{x}{|x|} - \nabla _{\mathbb {S}^{N - 1}} \varphi \left( \frac{x}{|x|} \right) }{\sqrt{1 + \left| \nabla _{\mathbb {S}^{N - 1}} \varphi \left( \frac{x}{|x|} \right) \right| ^2}},\ \quad x\in \Gamma _\varphi , \end{aligned}$$

where $\nabla _{\mathbb {S}^{N - 1}}$ is the gradient in ${\mathbb {S}}^{N - 1}$ (see [17, Sect. 2]), then, from (3.7), it follows that

$$\begin{aligned} \langle V, \nu \rangle = \frac{|x|}{\sqrt{1 + \left| \nabla _{\mathbb {S}^{N - 1}} \varphi \left( \frac{x}{|x|} \right) \right| ^2}} v \left( \frac{x}{|x|} \right) \quad \text {on} \Gamma _\varphi . \end{aligned}$$

Hence, using the parametrization $x = e^{\varphi (q)} q$, for $q \in D$, taking into account that the induced $(N-1)$-dimensional area element on $\Gamma _\varphi$ is given by

$$\begin{aligned} d \sigma _{\Gamma _\varphi } = e^{(N - 1)\varphi }\sqrt{1 + |\nabla _{\mathbb {S}^{N - 1}} \varphi |^2} \ d \sigma , \end{aligned}$$

and since $u_\varphi =0$ on $\Gamma _\varphi$, then, from (2.12), we readily obtain (3.9). $\square$

The next step is to compute the second derivative of T at $\Omega _\varphi$ with respect to directions $v, w \in C^2(\overline{D}, {\mathbb {R}})$

Lemma 3.2

Let $\varphi$ and $u_\varphi$ be as in Lemma 3.1. Then for any $v, w \in C^2({\overline{D}}, {\mathbb {R}})$ it holds

$$\begin{aligned} T''(\varphi )[v, w]&= - \frac{N}{2} \int _D e^{N \varphi } v w \left( \frac{\partial u_\varphi }{\partial \nu }(e^\varphi q) \right) ^2 \ d \sigma \nonumber \\&\quad - \int _D e^{N \varphi } v \frac{\partial u_\varphi }{\partial \nu }(e^\varphi q) \frac{\partial {\widetilde{u}}_w}{\partial \nu }(e^\varphi q) \ d \sigma \nonumber \\&\quad - \int _D e^{N \varphi } v w \frac{\partial u_\varphi }{\partial \nu }(e^\varphi q) (D^2u_\varphi (e^\varphi q) e^\varphi q) \cdot \nu \ d \sigma \nonumber \\&\quad + \int _D e^{N \varphi } v \frac{\partial u_\varphi }{\partial \nu }(e^\varphi q) \frac{\nabla u_\varphi (e^\varphi q) \cdot \nabla _{\mathbb {S}^{N - 1}} w}{\sqrt{1 + |\nabla _{\mathbb {S}^{N - 1}} \varphi |^2}} \ d \sigma \nonumber \\&\quad + \int _D e^{N \varphi } \left( \frac{\partial u_\varphi }{\partial \nu }(e^\varphi q) \right) ^2 \frac{\nabla _{\mathbb {S}^{N - 1}} \varphi \cdot \nabla _{\mathbb {S}^{N - 1}} w}{1 + |\nabla _{\mathbb {S}^{N - 1}} \varphi |^2} \ d \sigma , \end{aligned}$$

(3.10)

where ${\widetilde{u}}_w = \left. \frac{d}{ds}\right| _{s = 0} u_{\varphi + sw}$ satisfies (2.16) with $V(x) = w\left( \frac{x}{|x|}\right) x$.

Proof

The proof is the same as that of [17, Lemma 3.2] and therefore we omit it. $\square$

In view of Definition 2.4, we are interested in studying pairs $(\Omega _\varphi , u_\varphi )$ which are energy-stationary under a volume constraint. Thus we need to consider domains $\Omega _\varphi$ with a fixed volume. We recall that the volume of the domain defined by the radial graph of a function $\varphi \in C^2({\overline{D}}, {\mathbb {R}})$ is given by

$$\begin{aligned} {\mathcal {V}}(\varphi ) \, {:}{=}\, {\mathcal {V}}(\Omega _\varphi ) = |\Omega _\varphi | = \frac{1}{N} \int _D e^{N\varphi } \ d\sigma . \end{aligned}$$

Simple computations yield, for $v, w \in C^2({\overline{D}}, \mathbb R)$:

$$\begin{aligned} {\mathcal {V}}'(\varphi )[v] = \int _D e^{N \varphi } v\, \, d\sigma , \qquad {\mathcal {V}}''(\varphi )[v, w] = N \int _D e^{N\varphi } v w \ d \sigma . \end{aligned}$$

(3.11)

Then, for $c > 0$ we define the manifold

$$\begin{aligned} M \, {:}{=}\, \{\varphi \in C^2({\overline{D}}, \mathbb R) \ : \ {\mathcal {V}}(\varphi ) = c\}, \end{aligned}$$

(3.12)

whose tangent space at any point $\varphi \in M$ is given by

$$\begin{aligned} T_\varphi M = \left\{ v \in C^2({\overline{D}}, {\mathbb {R}}) \ : \ \int _D e^{N \varphi } v \ d\sigma = 0 \right\} . \end{aligned}$$

We restrict the energy functional to the manifold M and denote it by

$$\begin{aligned} I(\varphi ) := T\big |_M(\varphi ). \end{aligned}$$

Clearly, if the pair $(\Omega _\varphi , u_\varphi )$ is energy-stationary under a volume constraint, in the sense of Definition 2.4, then $\varphi \in M$ is a critical point of I. Hence, by the Theorem of Lagrange multipliers, there exists $\mu \in {\mathbb {R}}$ such that

$$\begin{aligned} T'(\varphi ) = \mu \mathcal V'(\varphi ). \end{aligned}$$

(3.13)

Moreover, the following result holds true:

Proposition 3.3

Let $\varphi \in M$ such that $(\Omega _\varphi , u_\varphi )$ is energy-stationary under the volume constraint. Then the Lagrange multiplier $\mu$ is negative and

$$\begin{aligned} \frac{\partial u_\varphi }{\partial \nu } = - \sqrt{- 2 \mu } \quad \text { on } \quad \Gamma _\varphi . \end{aligned}$$

(3.14)

Proof

The proof is the same as in [17, Lemma 4.1] $\square$

For the second derivative of I we have

Lemma 3.4

Let $\varphi \in M$ and let $v, w \in T_\varphi M$. If $(\Omega _\varphi , u_\varphi )$ is energy-stationary under the volume constraint, then

$$\begin{aligned} I''(\varphi )[v, w] = T''(\varphi )[v, w] - \mu {\mathcal {V}}''(\varphi )[v, w]. \end{aligned}$$

(3.15)

Proof

The proof is the same as in [17, Lemma 4.3]. $\square$

3.2 Spherical Sectors and Radial Solutions

Given a cone $\Sigma _D$ we consider the spherical sector $\Omega _D$ obtained by intersecting $\Sigma _D$ with the unit ball $B_1$. Obviously its relative boundary $\Gamma _{\Omega _D}$ is the radial graph obtained by taking $\varphi \equiv 0$ in (3.2), which is in fact the domain D which spans the cone, that is $\Gamma _{\Omega _D} = D$.

In the spherical sector $\Omega _D$ we would like to consider a nondegenerate positive radial solution $u_D \, {:}{=}\, u_{\Omega _D}$ of (3.4), hence we first recall conditions on the nonlinearity f which ensure that a positive radial solution of (3.4) in $\Omega _D$ exists. Observe that such $u_D$ is just the restriction to $\Omega _D$ of a positive radial solution of the Dirichlet problem

$$\begin{aligned} \left\{ \begin{array}{rcll} - \Delta u &{} = &{} f(u) &{} \quad \text { in } B_1\\ u &{} = &{} 0 &{} \quad \text { on } \partial B_1 \end{array} \right. \end{aligned}$$

(3.16)

Proposition 3.5

Let $f : {\mathbb {R}} \rightarrow \mathbb R$ be a locally Lipschitz continuous function. Assume that f satisfies one of the following:

• (i) $|f(s)| \le a|s| + b$ for all $s > 0$, where $b > 0$ and $a \in (0, \mu _1(B_1))$, where $\mu _1(B_1)$ is the first eigenvalue of the operator $- \Delta$ in $H_0^1(B_1)$.

• (ii) $f:[0, + \infty ) \rightarrow [0, + \infty )$ is non-increasing.

• (iii)

  • $|f(s)| < c|s|^p + d$, where $c, d > 0$ and $p \in \left( 1, \frac{N + 2}{N - 2} \right)$ if $N \ge 3$, $p > 1$ if $N = 2$;

  • $f(s) = o(s)$ as $s \rightarrow 0$;

  • There exist $\gamma > 2$, $\kappa > 0$ such that for $|s| > \kappa$ it holds

    $$\begin{aligned} 0< \gamma F(s) < s f(s); \end{aligned}$$

  • $\displaystyle f'(s) > \frac{f(s)}{s}$ for all $s > 0$.

Then a radial positive solution of (3.16) in $B_1$, and hence of (3.4) in $\Omega _D$, exists.

Proof

In cases (i) and (ii), the corresponding functional

$$\begin{aligned} J(u) = \frac{1}{2}\int _{B_1} |\nabla u|^2 \ dx - \int _{B_1} F(u) \ dx \end{aligned}$$

is coercive and weakly lower semicontinuous in the space $H^1_{0, rad}(B_1)$, which is the subspace of $H_0^1(B_1)$ of radial functions, and so it has a minimum which is a solution of (3.16). In the case (iii) standard variational methods, such as minimization on the Nehari manifold or Mountain Pass type theorems give a positive solution of (3.16), which is then radial by the Gidas-Ni-Nirenberg Theorem (see [15]). We refer to [4] and [9] for the details. $\square$

We point out that a radial solution $u_D$ is always a classical solution of (3.16) in $B_1$, and hence in $\Omega _D$. In particular, $u_D$ is bounded and $u_D \in C^2({\overline{B}}_1)$

Now we would like to study the nondegeneracy of a radial solution $u_D$ of (3.4) in $\Omega _D$.

As recalled in Sect. 2.1, we need conditions that ensure that zero is not an eigenvalue of the linearized operator

$$\begin{aligned} L_{u_D} = - \Delta - f'(u_D) \end{aligned}$$

(3.17)

in the space $H_0^1(\Omega _D \cup \Gamma _{1, 0})$, where $\Gamma _{1, 0} = \partial \Omega _D\setminus \overline{\Gamma }_{\Omega _D}$. Obviously, if the linearized operator $L_{u_D}$ admits only positive eigenvalues, then $u_D$ is nondegenerate. This is the case of stable solutions of (3.4), which occur when f satisfies conditions (i) or (ii) in Proposition 3.5, in particular, if f is a constant.

In general, $L_{u_D}$ could have negative eigenvalues, so to detect the nondegeneracy of $u_D$ we have to analyze the spectrum of the linear operator (3.17) in $H_0^1(\Omega _D \cup \Gamma _{1, 0})$. As we will see, the fact that $\Omega _D$ is a spherical sector in the cone $\Sigma _D$ (and not the ball $B_1$) plays a role.

The first remark is that zero is an eigenvalue for $L_{u_D}$ if and only if it is an eigenvalue for the following singular problem:

$$\begin{aligned} \left\{ \begin{array}{rcll} - \Delta \psi - f'(u_D) \psi &{} = &{} \displaystyle \frac{{\widehat{\Lambda }}}{|x|^2} \psi &{} \quad \text { in } \Omega _D \\ \psi &{} = &{} 0 &{} \quad \text { on } D \\ \displaystyle \frac{\partial \psi }{\partial \nu } &{} = &{} 0 &{} \quad \text { on } \Gamma _{1, 0} \setminus \{0\}. \end{array} \right. \end{aligned}$$

(3.18)

Indeed, since $N \ge 3$, problem (3.18) is well-defined in the space $H_0^1(\Omega _D \cup \Gamma _{1, 0})$ due to Hardy’s inequality (see [7, Proposition 2.1], for (3.18), and also [3] for the analogous Dirichlet problem).

Therefore we investigate the eigenvalues of (3.18). The advantage of considering this singular eigenvalue problem is that, since $u_D$ is radial, its eigenfunctions can be obtained by separation of variables, using polar coordinates in ${\mathbb {R}}^N$. To this aim we denote by $\{\lambda _j(D)\}_{j \in {\mathbb {N}}}$, the eigenvalues of the Laplace-Beltrami operator $- \Delta _{\mathbb {S}^{N - 1}}$ on the domain D with Neumann boundary conditions. It is well-known that

$$\begin{aligned} 0 = \lambda _0(D) < \lambda _1(D) \le \lambda _2(D) \le \ldots , \end{aligned}$$

(3.19)

and the only accumulation point is $+ \infty$. Then we consider the following singular eigenvalue problem in the interval (0, 1):

$$\begin{aligned} \left\{ \begin{array}{rcll} - z'' - \frac{N - 1}{r} z' - f'(u_D) z &{} = &{} \displaystyle \frac{{\widehat{\nu }}}{r^2} z &{} \quad \text { in } (0, 1) \\ z(1) &{} = &{} 0 &{} \end{array} \right. \end{aligned}$$

(3.20)

It is shown in [3] (see also [7]) that nonpositive eigenvalues for (3.20) can be defined. They are a finite number and we denote them by ${\widehat{\nu }}_i$, $i = 1, \ldots , k$. It is immediate to check that the eigenvalues ${\widehat{\nu }}_i$ are the eigenvalues of (3.18) which correspond to radial eigenfunctions. In particular, we consider the first eigenvalue ${\widehat{\nu }}_1$ of (3.20), referring to [3] for a variational definition and a study of its main properties.

By using (3.18)-(3.20) we obtain the following result:

Proposition 3.6

The problem (3.18) admits zero as an eigenvalue if and only if there exist $i\in \mathbb {N}^+$ and $j\in {\mathbb {N}}$ such that

$$\begin{aligned} {\widehat{\nu }}_i + \lambda _j(D) = 0. \end{aligned}$$

(3.21)

Proof

The proof follows by [7, Proposition 2.6], where it is proved that the nonpositive eigenvalues of (3.18) are obtained by summing the eigenvalues of the one-dimensional problem (3.20) and the Neumann eigenvalues of $- \Delta _{\mathbb {S}^{N - 1}}$ on D. We refer also to [11] for another approach, which consists in approximating the ball by annuli in order to avoid the singularity at 0. $\square$

From Proposition 3.6 we get the following sufficient condition for a radial solution $u_D$ to be nondegenerate.

Corollary 3.7

A radial solution $u_D$ of (3.4) in $\Omega _D$ (i.e. for $\varphi =0$) is nondegenerate if both the following conditions are satisfied:

• (I) the eigenvalue problem (3.20) does not admit zero as an eigenvalue;

• (II) $\lambda _1(D) > - {\widehat{\nu }}_1$.

Proof

From Condition (I) we have

$$\begin{aligned} {\widehat{\nu }}_i \ne 0 \quad \forall i\in \mathbb {N}^+, \end{aligned}$$

(3.22)

which means that zero is not an eigenvalue of (3.18) with a corresponding radial eigenfunction. This, in turn, is equivalent to saying that zero is not a “radial” eigenvalue of the linearized operator (3.17), i.e., $u_D$ is a radial solution of (3.4) in $\Omega _D$ (or of (3.16) in $B_1$) which is nondegenerate in the subspace $H_{0, rad}^1(\Omega _D \cup \Gamma _{1, 0})$, which is the subspace of $H_{0}^1(\Omega _D \cup \Gamma _{1, 0})$ given by radial functions.

Now, since $\lambda _0(D) = 0$, $\lambda _1(D) > 0$ and since ${\widehat{\nu }}_1$ is the smallest eigenvalue of (3.20), then, from Condition (II) and (3.22) we infer that the sum (3.21) can never be zero. Hence, thanks to Proposition 3.6, we have that zero is not an eigenvalue of (3.18) and so cannot be an eigenvalue for the linearized operator (3.17) in the whole $H_{0}^1(\Omega _D \cup \Gamma _{1, 0})$, i.e. $u_D$ is a non-degenerate solution to (3.4) in $\Omega _D$. $\square$

Remark 3.8

Condition (I) in Corollary 3.7, i.e., the nondegeneracy of $u_D$ in the space $H^1_{0, rad}(\Omega _D \cup \Gamma _{1, 0})$, is satisfied by positive radial solutions of (3.4) corresponding to many kinds of nonlinearities.

It holds if f satisfies conditions (i) or (ii) of Proposition 3.5, because in this case all the eigenvalues of (3.17) and of (3.18) are positive. It then follows that (II) holds as well. More precisely, in the case (i), since $0<a < \mu _1(B_1)$, the first eigenvalue of $L_{u_D}$ is positive, so

$$\begin{aligned} \underbrace{\lambda _0(D)}_{= 0} + {\widehat{\nu }}_1 > 0. \end{aligned}$$

(3.23)

In the case (ii), since $f'(u_D) \le 0$, it follows that ${\widehat{\nu }}_1 > 0$.

Among the nonlinearities satisfying condition (iii) of Proposition 3.5 we could consider $f(u) = u^p$, $1< p < \frac{N + 2}{N - 2}$, $N \ge 3$. Then it is known that the positive radial solution of (3.16) is unique and nondegenerate (see [8, 15]), so (I) holds. It is also well-known that for this nonlinearity it holds ${\widehat{\nu }}_1 < 0$ and ${\widehat{\nu }}_1$ is the only negative eigenvalue of (3.20), because $u_D$ can be obtained by the Mountain Pass Theorem or by minimization on the Nehari manifold and thus it has Morse index one. Then the validity of (II) depends on the cone, since it depends on $\lambda _1(D)$. However, once p is fixed, since ${\widehat{\nu }}_1$ does not depend on the cone, it is obvious that, by varying D, there are many cones for which (II) holds. Moreover, it has been proved in [7] that ${\widehat{\nu }}_1 > - (N - 1)$ for every autonomous nonlinearity, so that whenever $\lambda _1(D) > N - 1$ all radial solutions of (3.4) are nondegenerate.

3.3 Stability of $\mathbf {(\Omega _{\textit{D}}, \textit{u}_{\textit{D}})}$

Let us first observe that if $u_D$ is a positive nondegenerate radial solution of (3.4) for $\varphi = 0$, belonging to $W^{1, \infty }(\Omega _D) \cap W^{2, 2}(\Omega _D)$, then $(\Omega _D, u_D)$ is energy-stationary in the sense of Definition 2.4. Indeed, since $u_D$ is radial, we have that $\frac{\partial u_D}{\partial \nu } = \text {constant}$ on $\Gamma _0 = D$ and thanks to Proposition 2.6 we easily conclude.

To investigate the stability of $(\Omega _D, u_D)$ we analyze the quadratic form corresponding to the second derivative $I''(\varphi )$ at $\varphi = 0$. Fixing the constant c in the definition of M (see (3.12)) as $c = |\Omega _D|$, we have that the tangent space to M at $\varphi = 0$ is given by

$$\begin{aligned} T_0 M = \left\{ v \in C^2({\overline{D}}, {\mathbb {R}}) \ : \ \int _D v \ d\sigma = 0 \right\} . \end{aligned}$$

(3.24)

Writing $u_D(r) = u_D(|x|)$, we denote by $u_D'$ and $u_D''$ the derivatives of $u_D$ with respect to r, so that

$$\begin{aligned} u_D'(1) = \left. \frac{\partial u_D}{\partial \nu } \right| _D, \qquad u_D''(1) = [(D^2u_D \nu ) \cdot \nu ] |_D. \end{aligned}$$

(3.25)

By Hopf’s Lemma we know that $u_D'(1) < 0$ and actually

$$\begin{aligned} u_D'(1) = - \sqrt{- 2 \mu _D}, \end{aligned}$$

(3.26)

where $\mu _D$ denotes the Lagrange multiplier in the case $\varphi = 0$, see (3.13).

For $v \in T_0 M$, we will denote by ${\widetilde{u}}_v$ the solution of

$$\begin{aligned} \left\{ \begin{array}{rcll} - \Delta {\widetilde{u}}_v - f'(u_D) {\widetilde{u}}_v &{} = &{} 0 &{} \quad \text { in } \Omega _D \\ {\widetilde{u}}_v &{} = &{} - u_D'(1) v &{} \quad \text { on } D \\ \displaystyle \frac{\partial {\widetilde{u}}_v}{\partial \nu } &{} = &{} 0 &{} \quad \text { on } \Gamma _{1, 0} \setminus \{0\} \end{array} \right. . \end{aligned}$$

(3.27)

Let us remark that for every $q \in D$ the outer unit normal vector $\nu (q)$ is precisely q, hence (3.27) corresponds to (2.16) in $\Omega _D$.

Note that, since $u_D$ is a nondegenerate radial solution, then the weak solution ${\widetilde{u}}_v$ of (3.27) is unique for every v.

Our next result shows that the quadratic form corresponding to the second derivative of I at $\varphi = 0$ has a simple expression.

Lemma 3.9

For any $v \in T_0M$ it holds

$$\begin{aligned} I''(0)[v, v] = - u_D'(1)\left( \int _D v \frac{\partial {\widetilde{u}}_v}{\partial \nu } \ d \sigma + u_D''(1) \int _D v^2 \ d\sigma \right) , \end{aligned}$$

(3.28)

where ${\widetilde{u}}_v$ is the solution of (3.27).

Proof

From Lemma 3.2, (3.11) and Lemma 3.4, with $w = v$, by simple substitutions and elementary computations we obtain:

$$\begin{aligned} I''(0)[v, v]&= - \frac{N}{2} \int _D (u_D'(1))^2 v^2 \ d\sigma - \int _D u_D'(1) v \frac{\partial {\widetilde{u}}_v}{\partial \nu } \ d\sigma \nonumber \\&\quad - \int _D u_D'(1) v^2 (D^2u_D \nu )\cdot \nu \ d\sigma - N \mu _D \int _D v^2 \ d\sigma . \end{aligned}$$

(3.29)

Since ${\widetilde{u}}_v = -u_D'(1) v$ on D, by (3.25) and (3.26), we deduce that

$$\begin{aligned}&- \frac{N}{2} \int _D (u_D'(1))^2 v^2 \ d\sigma = - \frac{N}{2} \int _D {\widetilde{u}}_v^2 \ d\sigma ; \end{aligned}$$

(3.30)

$$\begin{aligned}&- N \mu _D \int _D v^2 \ d\sigma = \frac{N}{2} \int _D \widetilde{u}_v^2 \ d\sigma . \end{aligned}$$

(3.31)

Then (3.28) follows by substituting (3.30)-(3.31) into (3.29). $\square$

To investigate the stability of $(\Omega _D, u_D)$ as an energy stationary pair for I we need to study the solution $\widetilde{u}_v$ of (3.27), for any $v \in T_0M$ (that is, for functions with mean value zero on D). As we will see, it will be enough to consider only functions v which are eigenfunctions of the Laplace-Beltrami operator $- \Delta _{\mathbb {S}^{N - 1}}$ with Neumann boundary conditions on D. Hence we consider the eigenvalue problem

$$\begin{aligned} \left\{ \begin{array}{rcll} - \Delta \psi &{} = &{} \lambda \psi &{} \quad \text { on } D \\ \displaystyle \frac{\partial \psi }{\partial \nu } &{} = &{} 0 &{} \quad \text { on } \partial D \end{array} \right. \end{aligned}$$

(3.32)

and denote its eigenvalues as in (3.19), counted with multiplicity: $0 = \lambda _0(D) < \lambda _1(D)\le \lambda _2(D) \le \ldots$. The corresponding $L^2$-normalized eigenfunctions are denoted by $\{\psi _j\}_{j \in {\mathbb {N}}}$, with $\int _D \psi _j^2 \ d \sigma = 1$, $\psi _0 = \text {constant}$ and $\int _D \psi _j \ d\sigma = 0$ for $j \ge 1$.

Theorem 3.10

Let $j \ge 1$ and ${\widetilde{u}}_j$ be the unique solution of (3.27) for $v = \psi _j$. Then, writing ${\widetilde{u}}_j={\widetilde{u}}_j(r,q)$, the function

$$\begin{aligned} h_j(r) = \int _D {\widetilde{u}}_j(r, q) \psi _j(q) \ d\sigma , \quad r \in (0, 1) \end{aligned}$$

(3.33)

satisfies

$$\begin{aligned} \left\{ \begin{array}{ll} \displaystyle - h_j'' - \frac{N - 1}{r} h_j' - f'(u_D) h_j = - \frac{\lambda _j(D)}{r^2} h_j &{} \quad \text { in } (0, 1) \\ h_j(1) = -u_D'(1) \end{array} \right. \end{aligned}$$

(3.34)

Proof

Since the proof is the same for all j, we drop the index and the dependence on D and write simply h, $\psi$ and $\lambda$.

It is immediate to check that $h(1) = - u_D'(1)$. Moreover, since we can bring the radial derivative inside the integral on D, for every $r \in (0, 1]$ we have:

$$\begin{aligned} - h''(r) - \frac{N - 1}{r} h'(r)&= \int _D \left( - {\widetilde{u}}_{rr}(r, q) - \frac{N - 1}{r} {\widetilde{u}}_r(r, q) \right) \psi (q) \ d \sigma \nonumber \\&= \int _D \left( - \Delta {\widetilde{u}} + \frac{1}{r^2} \Delta _{\mathbb {S}^{N - 1}} {\widetilde{u}} \right) \psi \ d \sigma \nonumber \\&= \int _D f'(u_D(r)) {\widetilde{u}} \psi \ d\sigma + \frac{1}{r^2} \int _D (\Delta _{\mathbb {S}^{N - 1}} {\widetilde{u}}) \psi \ d\sigma . \end{aligned}$$

(3.35)

Now, on the one hand,

$$\begin{aligned} \int _D f'(u_D(r)) {\widetilde{u}} \psi \ d\sigma = f'(u_D(r)) h(r). \end{aligned}$$

(3.36)

On the other hand, applying Green’s formula, taking into account the Neumann conditions on $\psi$ and ${\widetilde{u}}$, we infer that

$$\begin{aligned} \frac{1}{r^2} \int _D (\Delta _{\mathbb {S}^{N - 1}} {\widetilde{u}}) \psi \ d \sigma&= \frac{1}{r^2} \int _D {\widetilde{u}} \Delta _{\mathbb {S}^{N - 1}} \psi \ d\sigma \nonumber \\&= - \frac{\lambda }{r^2} \int _D {\widetilde{u}} \psi \ d\sigma \nonumber \\&= - \frac{\lambda }{r^2} h(r). \end{aligned}$$

(3.37)

Substituting (3.36) and (3.37) into (3.35) we conclude the proof. $\square$

Remark 3.11

Note that with ${\widetilde{u}}_j$ and $h_j$ as in Theorem 3.10 we have that

$$\begin{aligned} {\widetilde{u}}_j(r, q) = h_j(r) \psi _j(q). \end{aligned}$$

Indeed, the boundary conditions are clearly satisfied by this function, and it holds

$$\begin{aligned} - \Delta (h_j \psi _j)&= - h_j'' \psi _j - \frac{(N - 1)}{r} h_j' \psi _j - \frac{h_j}{r^2} \Delta _{\mathbb {S}^{N - 1}} \psi _j\\&= f'(u_D) h_j \psi _j - \frac{\lambda _j(D)}{r^2} h_j \psi _j + \frac{\lambda _j(D)}{r^2} h_j \psi _j \\&= f'(u_D) h_j \psi _j. \end{aligned}$$

Proposition 3.12

Let $N \ge 3$. For any $j \ge 1$ we have

$$\begin{aligned} \int _0^1 r^{N - 3} h_j^2 \ dr < + \infty \end{aligned}$$

(3.38)

and

$$\begin{aligned} \int _0^1 r^{N - 1} (h_j')^2 \ dr < + \infty . \end{aligned}$$

(3.39)

Moreover, $h_j \in L^\infty (0, \infty )$ and $h_j(0) = 0$.

Proof

Again, for simplicity, we drop the index j. Since ${\widetilde{u}} \in H^1(\Omega _D)$ (see Sect. 2), writing ${\widetilde{u}}={\widetilde{u}}(r,q)$ and recalling that $\psi$ is a $L^2(D)$-normalized solution to (3.32), we get that

$$\begin{aligned} + \infty&> \int _{\Omega _D}|\nabla {\widetilde{u}}|^2 \ dx \nonumber \\&= \int _0^1 r^{N - 1} (h')^2 \int _D \psi ^2 \ d\sigma \ dr + \int _0^1 r^{N - 3} h^2 \int _D |\nabla _{\mathbb {S}^{N - 1}} \psi |^2 \ d\sigma \ dr \nonumber \\&= \int _0^1 r^{N - 1}(h')^2 \ dr + \lambda \int _0^1 r^{N - 3} h^2 \ dr, \end{aligned}$$

which proves (3.38) and (3.39). Once we have these estimates, we can proceed as in [10, Lemma A.9] to get the boundness of h and $h(0) = 0$. $\square$

Proposition 3.13

Let $\lambda _j(D)$, $j \ge 1$ be a nontrivial Neumann eigenvalue of $- \Delta _{\mathbb {S}^{N - 1}}$ on D. Assume that

$$\begin{aligned} - {\widehat{\nu }}_1 < \lambda _j(D), \end{aligned}$$

where ${\widehat{\nu }}_1$ is the smallest eigenvalue of (3.20). Then for the solution $h_j$ of (3.34) it holds that

$$\begin{aligned} h_j > 0 \quad \text { in } \quad (0, 1). \end{aligned}$$

Proof

Let $z_1$ be an $L^2$-normalized first eigenfunction of (3.20). From [3, Sect. 3.1] we know that $z_1$ does not change sign.

Writing the equations satisfied by $h_j$ and $z_1$ in Sturm-Liouville form we have:

$$\begin{aligned}&(r^{N - 1} h_j')' + r^{N - 1}(f'(u_D) - r^{- 2} \lambda _j(D))h_j = 0, \\&(r^{N - 1} z_1')' + r^{N - 1}(f'(u_D) + r^{-2} {\widehat{\nu }}_1)z_1 = 0. \end{aligned}$$

By Proposition 3.12 we know that $h_j(0) = 0$ and $h_j(1) = -u_D'(1)>0$.

Now, assume by contradiction that $h_j$ changes sign in (0, 1). Then there would exist $r_0 \in (0, 1)$ such that $h_j(0) = 0$. Since $- {\widehat{\nu }}_1 < \lambda _j(D)$, then, by the Sturm-Picone Comparison Theorem it would follow that $z_1$ has a zero in $(0, r_0)$. This is a contradiction, because $z_1$ does not change sign. Hence the only possibility is that $h_j$ is strictly positive in (0, 1). $\square$

We are ready to prove our main result for problem (1.1) in the case of the cone, i.e., Theorem 1.1, which is a sharp instability/stability result for the pair $(\Omega _D, u_D)$.

3.4 Proof of Theorem 1.1

Let us fix the domain D which spans the cone, so that we denote $\lambda _1(D)$ simply by $\lambda _1$.

For (i), let ${\widetilde{u}}_1 = h_1 \psi _1$ be the solution of (3.27) with $v = \psi _1$. Then

$$\begin{aligned} I''(0)[\psi _1, \psi _1] = - u_D'(1) (h_1'(1) + u_D''(1)). \end{aligned}$$

(3.40)

Putting (3.34) in Sturm-Liouville form we get

$$\begin{aligned} - (r^{N - 1} h_1')' - r^{N - 1}f'(u_D)h_1 = - r^{N - 3} \lambda _1 h_1. \end{aligned}$$

(3.41)

On the other hand, writing $- \Delta u_D = f(u_D)$ in polar coordinates and differentiating with respect to $r = |x|$ we get

$$\begin{aligned} - (u_D')'' - \frac{N - 1}{r} (u_D')' - f'(u_D) u_D' = - \frac{N - 1}{r^2} u_D', \end{aligned}$$

which in Sturm-Liouville form is

$$\begin{aligned} - (r^{N - 1} u_D'')' - r^{N - 1} f'(u_D)u_D' = - r^{N - 3} (N - 1) u_D'. \end{aligned}$$

(3.42)

Multiplying (3.41) by $u_D'$ and integrating by parts in $({\bar{r}}, 1)$ we get that

$$\begin{aligned} \begin{array}{lll} &{}&{}\displaystyle \int _{{\bar{r}}}^1 r^{N - 1} h_1' u_D'' \ dr - (r^{N - 1} h_1' u_D')\big |_{{\bar{r}}}^1- \int _{{\bar{r}}}^1 r^{N - 1} f'(u_D) h_1 u_D' \ dr\\ &{}=&{}\displaystyle - \lambda _1 \int _{{\bar{r}}}^1 r^{N - 3} h_1 u_D' \ dr. \end{array} \end{aligned}$$

(3.43)

Similarly, multiplying (3.42) by $h_1$ and integrating by parts we deduce that

$$\begin{aligned} \begin{array}{lll} &{}\displaystyle \int _{{\bar{r}}}^1 r^{N - 1} h_1' u_D'' \, dr - (r^{N - 1} h_1 u_D'')\big |_{{\bar{r}}}^1 - \int _{{\bar{r}}}^1 r^{N - 1} f'(u_D) u_D' h_1 \, dr\\ =&{}\displaystyle -(N - 1) \int _{{\bar{r}}}^1 r^{N - 3} u_D' h_1 \, dr. \end{array} \end{aligned}$$

(3.44)

Notice that, in view of Proposition 3.12, the right-hand sides of (3.43), (3.44) remain finite when taking the limit as ${\bar{r}}\rightarrow 0^+$. In addition, we claim that

$$\begin{aligned} \lim _{{\bar{r}} \rightarrow 0^+} r^{N - 1} h_1'({\bar{r}}) u_D'({\bar{r}}) = 0. \end{aligned}$$

(3.45)

Indeed, integrating (3.41) and taking the absolute value we obtain

$$\begin{aligned} \left| \int _{{\bar{r}}}^1 - (r^{N - 1} h_1')' \ dr \right|&= \left| {\bar{r}}^{N - 1} h_1'({\bar{r}}) - h_1'(1)\right| \\&\le \int _{{\bar{r}}} r^{N - 1} |f'(u_D)| h_1\, dr + \int _0^1 r^{N - 3} \lambda _1 h_1 \, dr \\&\le C_1 \end{aligned}$$

for some $C_1 > 0$. Hence

$$\begin{aligned} \limsup _{{\bar{r}} \rightarrow 0^+} {\bar{r}}^{N - 1}|h_1'({\bar{r}})| \le C_2 \end{aligned}$$

(3.46)

for some $C_2>0$, and thus, since $\lim _{{\bar{r}} \rightarrow 0^+} u_D'(\bar{r}) = 0$, (3.45) follows.

Now, subtracting (3.44) from (3.43) and taking the limit as ${\bar{r}} \rightarrow 0^+$, then, thanks to (3.45) and since $h_1(0)=0$, $h_1(1) = - u_D'(1)$, we obtain

$$\begin{aligned} - u_D'(1)(h_1'(1) + u_D''(1)) = (N - 1 - \lambda _1) \int _0^1 r^{N - 3} h_1 u_D' \ dr. \end{aligned}$$

(3.47)

Since $\lambda _1 > -{\widehat{\nu }}_1$, then, by Proposition 3.13, we have that $h_1 > 0$ in (0, 1). On the other hand $u_D' < 0$ in (0, 1) and $\lambda _1 < N - 1$ by assumption. Hence by (3.40) and (3.47) we obtain

$$\begin{aligned} I''(0)[\psi _1, \psi _1] < 0, \end{aligned}$$

which proves (i).

For (ii), we choose an orthonormal basis $(\psi _j)_j$ of $L^2(D)$ made of normalized eigenfunctions of (3.32). Then any $v \in T_0M$ can be written as

$$\begin{aligned} v = \sum _{j = 1}^\infty (v, \psi _j) \psi _j, \end{aligned}$$

where $(\cdot , \cdot )$ denotes the inner product in $L^2(D)$. We assume without loss of generality that $\int _D v^2 \ d\sigma = 1$. Let ${\widetilde{u}}_j$ be the solution of (3.27) with $v = \psi _j$, then we can check that

$$\begin{aligned} {\widetilde{v}} = \sum _{j = 1}^\infty (v, \psi _j) {\widetilde{u}}_j \end{aligned}$$

is the solution of (3.27). As observed in Remark 3.11, ${\widetilde{u}}_j(r,q) = h_j(r) \psi _j(q)$ for every $j \in {\mathbb {N}}$, so

$$\begin{aligned} \frac{\partial {\widetilde{u}}_j}{\partial \nu }(1,q) = h_j'(1) \psi _j(q)\ \quad \text {on { D}. } \end{aligned}$$

By an argument analogous to the one presented in the proof of (i), we have that if $k > j$, then $h_k'(1) {\ge } h_j'(1)$ and in fact $h_k'(1) > h_j'(1)$ if $k>j$ are such that $\lambda _k > \lambda _j$.

Indeed, writing the equations for $h_j, h_k$, multiplying the first one by $h_k$ and the second one by $h_j$, integrating by parts and subtracting we get

$$\begin{aligned} - u_D'(1) (h_k'(1) - h_j'(1)) = (- \lambda _j + \lambda _k) \int _0^1 r^{N - 3}h_ih_j \ge 0. \end{aligned}$$

Exploiting the orthogonality of the basis $(\psi _j)_j$ and exploiting (3.47) we obtain

$$\begin{aligned} I''(0)[v, v]&= - u_D'(1) \left( \int _D \left( \sum _{j = 1}^\infty (v, \psi _j) \psi _j \right) \left( \sum _{k = 1}^\infty (v, \psi _j) h_k'(1) \psi _k\right) \ d\sigma + u_D''(1) \int _D v^2 \ d\sigma \right) \nonumber \\&= - u_D'(1) \left( \left( \sum _{j = 1}^\infty (v, \psi _j)^2 h_j'(1) \right) + u_D''(1)\right) \nonumber \\&\ge - u_D'(1) \left( h_1'(1) \left( \sum _{j = 1}^\infty (v, \psi _j)^2 \right) + u_D''(1) \right) \nonumber \\&= - u_D'(1) (h_1'(1) + u_D''(1)) \nonumber \\&= (N - 1 - \lambda _1) \int _0^1 r^{N - 3} h_1 u_D' \ dr>0, \end{aligned}$$

because $h_1>0$ in (0, 1), $u^\prime _D<0$ in (0, 1) and $\lambda _1>N-1$ by assumption. The proof is complete. $\square$

Remark 3.14

As already pointed out in Remark 2.7, in the case when ${\mathcal {C}} = {\mathbb {R}}^N$, the couples $(B, u_B)$, where B is a ball and $u_B$ is a positive nondegenerate radial solution, are the only energy-stationary pairs. Thus it remains to study the stability of $(B, u_B)$ as critical point of the energy functional T. This can be done by looking at the problem as the case of a cone spanned by the domain $D = \mathbb {S}^{N - 1}$.

As observed in Remark 1.2, the first eigenvalue ${\widehat{\nu }}_1$ of the singular eigenvalue problem (3.20) is always larger than $-(N - 1)$. On the other hand, it is known that the first nontrivial eigenvalue of the Laplace-Beltrami operator on the whole $\mathbb {S}^{N - 1}$ is precisely $N - 1$. Then any radial solution $u_B$ is nondegenerate and we obtain that the pair $(B, u_B)$ is a semistable stationary-point.

4 The Case of the Cylinder

Let $\omega \subset {\mathbb {R}}^{N - 1}$ be a smooth bounded domain and let $\Sigma _\omega$ be the half-cylinder spanned by $\omega$, namely

$$\begin{aligned} \Sigma _\omega \, {:}{=}\, \omega \times (0, + \infty ). \end{aligned}$$

We denote by $x = (x', x_N)$ the points in $\overline{\Sigma }_\omega$, where $x'=(x_1,\ldots ,x_{N-1}) \in {\overline{\omega }}$ and $x_N \ge 0$.

In analogy with the case of the cone, we consider domains whose relative boundaries are the cartesian graphs of functions in $C^2({\overline{\omega }})$. More precisely, for $\varphi \in C^2({\overline{\omega }})$ we set

$$\begin{aligned} \Gamma _\varphi \, {:}{=}\, \{(x', x_N) \in \Sigma _\omega \ : \ x_N = e^{\varphi (x')}\} \end{aligned}$$

and consider domains of the type

$$\begin{aligned} \Omega _\varphi = \{(x', x_N) \in \Sigma _\omega \ : \ x_N < e^{\varphi (x')}\}. \end{aligned}$$

Finally, let

$$\begin{aligned} \Gamma _{1, \varphi } \, {:}{=}\, (\partial \Omega _\varphi \setminus \overline{\Gamma }_\varphi ). \end{aligned}$$

Observe that the outer unit normal vector on $\Gamma _\varphi$ at a point $(x', e^{\varphi (x')})$ is given by

$$\begin{aligned} \nu =\nu _\varphi (x^\prime )= \frac{(- e^{\varphi (x^\prime )} \nabla _{\mathbb {R}^{N-1}}\varphi (x^\prime ), 1)}{\sqrt{1 + |e^{\varphi (x^\prime )} \nabla _{\mathbb {R}^{N-1}} \varphi (x^\prime )|^2}}, \end{aligned}$$

(4.1)

where $\nabla _{\mathbb {R}^{N-1}}$ denotes the gradient with respect to the variables $x_1,\ldots ,x_{N-1}$.

4.1 Energy Functional in Cylindrical Domains

We study the semilinear elliptic problem

$$\begin{aligned} \left\{ \begin{array}{rcll} - \Delta u &{} = &{} f(u) &{} \quad \text { in } \Omega _\varphi \\ u &{} = &{} 0 &{} \quad \text { on } \Gamma _\varphi \\ \displaystyle \frac{\partial u}{\partial \nu } &{} = &{} 0 &{} \quad \text { on } \Gamma _{1, \varphi } \end{array} \right. \end{aligned}$$

(4.2)

and consider bounded positive weak solutions of (4.2) in the Sobolev space $H_0^1(\Omega _\varphi \cup \Gamma _{1,\varphi })$, which is the space of functions in $H^1(\Omega _\varphi )$ whose trace vanishes on $\Gamma _\varphi$.

As before, we assume that a bounded nondegenerate positive solution $u_\varphi$ of (4.2) exists and belongs to $W^{1, \infty }(\Omega _\varphi ) \cap W^{2, 2}(\Omega _\varphi )$, so that we can apply the results of Sect. 2.

We consider variations of the domain $\Omega _\varphi$ in the class of cartesian graphs of the type $\Omega _{\varphi + tv}$, for $v \in C^2({\overline{\omega }})$, which amounts to consider a one-parameter family of diffeomorphisms $\xi :(-\eta ,\eta )\times {\overline{\Sigma }}_\omega \rightarrow {\overline{\Sigma }}_\omega$ of the type

$$\begin{aligned} \xi (t, x) = (x', e^{t v(x')}x_N), \end{aligned}$$

whose inverse, for any fixed $t\in (-\eta , \eta )$, is given by

$$\begin{aligned} \xi (t, x)^{-1} = (x', e^{- t v(x')} x_N) = \xi (- t, x). \end{aligned}$$

This one-parameter family of diffeomorphisms is generated by the vector field

$$\begin{aligned} V(x) = (0^\prime , v(x')x_N), \end{aligned}$$

(4.3)

where $0^\prime :=(0, \ldots , 0)\in \mathbb {R}^{N-1}$. Indeed, $\xi (0, x) = x$ for every $x \in {\overline{\Sigma }}_\omega$,

$$\begin{aligned} \frac{d\xi }{dt}(t, x) = (0^\prime , e^{tv(x')}v(x')x_N) = V(\xi (t, x))\quad \forall (t,x)\in (-\eta ,\eta )\times \Sigma _\omega \end{aligned}$$

and $\xi (t, x)\in \partial \Sigma _\omega$, for all $(t,x)\in (-\eta ,\eta )\times \partial \Sigma _\omega$. We also observe that, in view of (4.1), it holds

$$\begin{aligned} \langle V, \nu \rangle = \left\langle (0^\prime , v e^\varphi ), \frac{(- e^\varphi \nabla _{\mathbb {R}^{N-1}} \varphi , 1)}{\sqrt{1 + |e^\varphi \nabla _{\mathbb {R}^{N-1}} \varphi |^2}} \right\rangle = \frac{v e^\varphi }{\sqrt{1 + |e^\varphi \nabla _{\mathbb {R}^{N-1}} \varphi |^2}} \quad \ \text {on} \Gamma _\varphi . \end{aligned}$$

(4.4)

The energy functional T defined in (2.10) becomes a functional depending only on functions in $C^2({\overline{\omega }})$. More precisely, for every $v \in C^2({\overline{\omega }})$, in view of Proposition 2.1, there exists $\delta > 0$ sufficiently small such that for all $t \in (- \delta , \delta )$

$$\begin{aligned} T(\varphi + tv) = T(\Omega _{\varphi + tv}) = J(u_{\varphi + tv}), \end{aligned}$$

is well defined, where $u_{\varphi + tv} \, {:}{=}\, u_{\Omega _{\varphi + tv}}$ is the unique positive solution of (4.2) in the domain $\Omega _{\varphi + tv}$, in a neighborhood of $u_\varphi \circ \xi _t^{-1}$.

By the results of Sect. 2 we know that the map $t \mapsto u_{\varphi + tv}$ is differentiable at $t = 0$, and the derivative ${\widetilde{u}}$ is a weak solution of

$$\begin{aligned} \left\{ \begin{array}{rcll} - \Delta {\widetilde{u}} &{} = &{} f'(u_\varphi ) {\widetilde{u}} &{} \quad \text { in } \Omega _\varphi \\ {\widetilde{u}} &{} = &{} \displaystyle - \frac{\partial u_\varphi }{\partial \nu } \frac{v e^\varphi }{\sqrt{1 + |e^\varphi \nabla _{\mathbb {R}^{N-1}} \varphi |^2}} &{} \quad \text { on } \Gamma _\varphi \\ \displaystyle \frac{\partial {\widetilde{u}}}{\partial \nu } &{} = &{} 0 &{} \quad \text { on } \Gamma _{1, \varphi } \end{array} \right. \end{aligned}$$

(4.5)

We now compute the first derivative of T at $\Omega _\varphi$, i.e., for $t = 0$, with respect to variations $v \in C^2(\overline{\omega })$.

Lemma 4.1

Let $\varphi \in C^2({\overline{\omega }})$ and assume that $u_\varphi$ is a positive nondegenerate solution of (4.2) which belongs to $W^{1, \infty }(\Omega ) \cap W^{2, 2}(\Omega )$. Then, for any $v \in C^2({\overline{\omega }})$ we have

$$\begin{aligned} T'(\varphi )[v] = - \frac{1}{2} \int _\omega \left( \frac{\partial u_\varphi }{\partial \nu }(x^\prime ,e^\varphi )\right) ^2 v e^\varphi \ dx'. \end{aligned}$$

(4.6)

Proof

The proof is similar to that of Lemma 3.1. It suffices to observe that for the parametrization of $\Gamma _\varphi$ given by $x=(x^\prime ,e^{\varphi (x^\prime )})$, for $x^\prime \in \omega$, the induced $(N-1)$-dimensional area element on $\Gamma _\varphi$ is expressed by

$$\begin{aligned} d \sigma _{\Gamma _\varphi } = \sqrt{1 + |e^\varphi \nabla _{\mathbb {R}^{N-1}} \varphi |^2} \ dx'. \end{aligned}$$

Then the result follows immediately from Proposition 2.2, taking into account (4.4). $\square$

Lemma 4.2

Let $\varphi$ and $u_\varphi$ be as in Lemma 4.1. Then for any $v, w \in C^2(\overline{\omega })$ it holds

$$\begin{aligned} T''(\varphi )[v, w] =&- \frac{1}{2} \int _\omega \left( \frac{\partial u_\varphi }{\partial \nu }(x^\prime ,e^\varphi ) \right) ^2 e^\varphi v w \ dx' \nonumber \\&- \int _\omega \frac{\partial {\widetilde{u}}_w}{\partial \nu }(x^\prime ,e^\varphi ) \frac{\partial u_\varphi }{\partial \nu }(x^\prime ,e^\varphi ) e^\varphi v \ dx' \nonumber \\&- \int _\omega \frac{\partial u_\varphi }{\partial \nu }(x^\prime ,e^\varphi ) [(D^2u_\varphi (x^\prime ,e^\varphi ) (0^\prime ,e^\varphi )) \cdot \nu ] vw \ dx' \nonumber \\&+ \int _\omega \frac{\partial u_\varphi }{\partial \nu }(x^\prime ,e^\varphi )e^{2\varphi } v \frac{\nabla u_\varphi (x^\prime ,e^\varphi ) \cdot (w \nabla _{{\mathbb {R}}^{N - 1}} \varphi + \nabla _{{\mathbb {R}}^{N - 1}} w, 0)}{\sqrt{1 + |e^\varphi \nabla _{{\mathbb {R}}^{N - 1}} \varphi |^2}} \ dx' \nonumber \\&+ \int _\omega \left( \frac{\partial u_\varphi }{\partial \nu }(x^\prime ,e^\varphi ) \right) ^2 e^{3\varphi } v \frac{\nabla _{{\mathbb {R}}^{N - 1}} \varphi \cdot (w\nabla _{\mathbb R^{N - 1}} \varphi + \nabla _{{\mathbb {R}}^{N - 1}} w)}{1 + |e^\varphi \nabla _{{\mathbb {R}}^{N - 1}} \varphi |^2} \ dx', \end{aligned}$$

(4.7)

where ${\widetilde{u}}_w$ is the solution of (4.5), with w in the place of v.

Proof

Let $v, w \in C^2({\overline{\omega }})$. By definition, Lemma 4.1 and using the Leibniz rule, we have:

$$\begin{aligned} T''(\varphi )[v, w]&= \left. \frac{d}{ds} \right| _{s = 0}\left( - \frac{1}{2} \int _\omega \left( \frac{\partial u_{\varphi + sw}}{\partial \nu }(x^\prime ,e^{\varphi + sw})\right) ^2 e^{\varphi + sw} v \ dx' \right) \nonumber \\&= - \int _\omega e^\varphi v \frac{\partial u_\varphi }{\partial \nu } \left. \frac{d}{ds}\right| _{s = 0}\left( \frac{\partial u_{\varphi + sw}}{\partial \nu }(x^\prime ,e^{\varphi + sw}) \right) \ dx' \nonumber \\&\ \ \ - \frac{1}{2} \int _\omega \left( \frac{\partial u_\varphi }{\partial \nu }(x^\prime ,e^{\varphi }) \right) ^2 e^\varphi v w \ dx'. \end{aligned}$$

(4.8)

To conclude it suffices to compute the derivative in the first integral of the right-hand side of (4.8). To this end we observe that

$$\begin{aligned} \left. \frac{d}{ds}\right| _{s = 0} \left( \frac{\partial u_{\varphi + sw}}{\partial \nu }(x^\prime ,e^{\varphi + sw}) \right)&= \left. \frac{d}{ds}\right| _{s = 0} \left( \nabla u_{\varphi + sw}(x^\prime ,e^{\varphi + sw}) \cdot \nu _{\varphi + sw} \right) \nonumber \\&= \left. \frac{d}{ds}\right| _{s = 0} (\nabla u_{\varphi + sw}(x', e^{\varphi + sw})) \cdot \nu _\varphi \nonumber \\&\ \ \ + \nabla u_\varphi (x^\prime , e^\varphi )\cdot \left. \frac{d}{ds}\right| _{s = 0} \nu _{\varphi + sw} \end{aligned}$$

(4.9)

where $\nu _\varphi$ is given by (4.1) and

$$\begin{aligned} \nu _{\varphi + sw} = \frac{(- e^{\varphi + sw} \nabla _{{\mathbb {R}}^{N - 1}} (\varphi + sw), 1)}{\sqrt{1 + |e^{\varphi + sw} \nabla _{{\mathbb {R}}^{N - 1}}(\varphi + sw)|^2}}. \end{aligned}$$

Now, for the first term in the right-hand side of (4.9), thanks to the argument presented in [17, Lemma 3.2], we have

$$\begin{aligned} \frac{d}{ds} (\nabla u_{\varphi + sw}) = \nabla \left( \frac{d}{ds} u_{\varphi + sw} \right) , \end{aligned}$$

and thus we obtain

$$\begin{aligned} \left. \frac{d}{ds}\right| _{s = 0} (\nabla u_{\varphi + sw} (x', e^{\varphi + sw})) = \nabla \widetilde{u}_w(x^\prime ,e^\varphi ) + D^2u_\varphi (x^\prime ,e^\varphi ) (0^\prime ,we^\varphi ). \end{aligned}$$

(4.10)

On the other hand, for the last term in (4.9), we check that

$$\begin{aligned} \left. \frac{d}{ds} \right| _{s = 0} \nu _{\varphi + sw}&= - \frac{e^\varphi }{\sqrt{1 + |e^\varphi \nabla _{{\mathbb {R}}^{N - 1}} \varphi |^2}} (\nabla _{{\mathbb {R}}^{N - 1}} w + w \nabla _{{\mathbb {R}}^{N - 1}} \varphi , 0) \nonumber \\&\quad - \frac{(e^\varphi )^2 (w |\nabla _{{\mathbb {R}}^{N - 1}} \varphi |^2 + \nabla _{{\mathbb {R}}^{N - 1}} \varphi \cdot \nabla _{{\mathbb {R}}^{N - 1}} w)}{1 + |e^\varphi \nabla _{{\mathbb {R}}^{N - 1}} \varphi |^2} \nu _{\varphi } \end{aligned}$$

(4.11)

Finally, substituting (4.9)–(4.11) into (4.8) we obtain (4.7). $\square$

As in Sect. 3, in view of Definition 2.4, we consider a volume constraint. In the case of cartesian graphs, the volume of the domain $\Omega _\varphi$ associated to $\varphi \in C^2({\overline{\omega }})$ is expressed by

$$\begin{aligned} {\mathcal {V}}(\varphi ) = |\Omega _\varphi | = \int _\omega e^\varphi \ dx'. \end{aligned}$$

(4.12)

The functional ${\mathcal {V}}$ is of class $C^2$ and for every $v, w \in C^2({\overline{\omega }})$ it holds

$$\begin{aligned} {\mathcal {V}}'(\varphi ) [v] = \int _\omega e^\varphi v \ dx', \qquad {\mathcal {V}}''(\varphi )[v, w] = \int _\omega e^\varphi v w \ dx'. \end{aligned}$$

(4.13)

For $c > 0$ we define the manifold

$$\begin{aligned} M \, {:}{=}\, \left\{ \varphi \in C^2({\overline{\omega }}) \ : \ \int _\omega e^\varphi \ dx' = c\right\} , \end{aligned}$$

whose tangent space at any point $\varphi \in M$ is given by

$$\begin{aligned} T_\varphi M = \left\{ v \in C^2({\overline{\omega }}) \ : \ \int _\omega e^\varphi v \ dx' = 0 \right\} . \end{aligned}$$

(4.14)

We consider the restricted functional

$$\begin{aligned} I(\varphi ) = T|_M(\varphi ), \quad \varphi \in M. \end{aligned}$$

As before, if $\varphi \in M$ is a critical point for I, then there exists a Lagrange multiplier $\mu \in {\mathbb {R}}$ such that

$$\begin{aligned} T'(\varphi ) = \mu I'(\varphi ). \end{aligned}$$

Results analogous to Proposition 3.3 and Lemma 3.4 hold with the same proofs. In particular, we point out that for an energy stationary pair $(\Omega _\varphi , u_\varphi )$ under a volume constraint the function $u_\varphi$ has constant normal derivative on $\Gamma _\varphi$. For the reader’s convenience, we restate here these results.

Proposition 4.3

Let $\varphi \in M$ and let $(\Omega _\varphi , u_\varphi )$ be energy-stationary under a volume constraint. Then the Lagrange multiplier $\mu$ is negative and

$$\begin{aligned} \frac{\partial u_\varphi }{\partial \nu } = - \sqrt{- 2 \mu } \quad \text { on } \quad \Gamma _\varphi . \end{aligned}$$

Proof

The same as in [17, Lemma 4.1] $\square$

For the second derivative of I we have

Lemma 4.4

Let $\varphi \in M$ and let $v, w \in T_\varphi M$. If $(\Omega _\varphi , u_\varphi )$ is energy-stationary under a volume constraint, then

$$\begin{aligned} I''(\varphi )[v, w] = T''(\varphi )[v, w] - \mu \mathcal V''(\varphi )[v, w]. \end{aligned}$$

(4.15)

Proof

The same as in [17, Lemma 4.3] $\square$

4.2 The Case $\mathbf {\varphi \equiv 0}$ and One-Dimensional Solutions

When $\varphi \equiv 0$ (that is, $\Gamma _\varphi = \Gamma _0$ is the intersection of the cylinder with the plane $x_N = 1$), the domain $\Omega _0$ is just the finite cylinder

$$\begin{aligned} \Omega _\omega :=\omega \times (0, 1). \end{aligned}$$

Then, if f is a locally Lipschitz continuous function, any weak solution of (4.2) is also a classical solution up to the boundary, i.e., it belongs to $C^2({\overline{\Omega }}_\omega )$. This follows by standard regularity theory by considering the boundary conditions and that $\partial \Omega _\omega$ is made by the union of three $(N - 1)$-dimensional manifolds (with boundary) intersecting orthogonally (see also [20, Proposition 6.1]).

In $\Omega _\omega$, for suitable nonlinearities, we can find a solution of (4.2) in $\Omega _\omega$ which depends only on $x_N$ in the following way: first, we can apply some variational method to find a solution u of the ordinary differential equation

$$\begin{aligned} {\left\{ \begin{array}{ll} -u'' = f(u) \quad \text { in } (0, 1) \\ u'(0) = u(1) = 0 \end{array}\right. } \end{aligned}$$

(4.16)

and then set

$$\begin{aligned} u_\omega (x', x_N) := u(x_N), \quad \hbox { }\ (x', x_N) \in \Omega _\omega . \end{aligned}$$

Recall that, in one dimension, there is no critical Sobolev exponent for the embedding into $L^p$. So one example of a suitable nonlinearity is $f(u) = u^p$ with $1< p < \infty$, or those of Proposition 3.5 with the only caution that in (iii), for $N\ge 2$ we can take $1< p < \infty$.

For our purposes we need to consider one-dimensional solutions $u_\omega$ of (4.2) in $\Omega _\omega$ that are nondegenerate, which means that the linearized operator

$$\begin{aligned} L_{u_\omega } = - \Delta - f'(u_\omega ) \end{aligned}$$

does not admit zero as an eigenvalue. In other words, $u_\omega$ is nondegenerate if there are no nontrivial weak solutions $\phi \in H_0^1(\Omega _\omega \cup \Gamma _{1, 0})$ of the problem

$$\begin{aligned} \left\{ \begin{array}{rcll} - \Delta \phi - f'(u_\omega ) \phi &{} = &{} 0 &{} \quad \text { in } \Omega _\omega \\ \phi &{} = &{} 0 &{} \quad \text { on } \Gamma _0 \\ \displaystyle \frac{\partial \phi }{\partial \nu } &{} = &{} 0 &{} \quad \text { on } \Gamma _{1, 0} \end{array} \right. \end{aligned}$$

(4.17)

To analyze the spectrum of $L_{u_\omega }$ it is convenient to consider the following auxiliary one-dimensional eigenvalue problem:

$$\begin{aligned} {\left\{ \begin{array}{ll} - z'' - f'(u_\omega ) z = \alpha z \quad \text { in } (0, 1) \\ z'(0) = z(1) = 0 \end{array}\right. } \end{aligned}$$

(4.18)

We denote the eigenvalues of (4.18) by $\alpha _i$, for $i\in \mathbb {N}$. Clearly, they correspond to the eigenvalues of the linear operator

$$\begin{aligned} {\widehat{L}}_{u_\omega }(z) = - z'' - f'(u_\omega ) z \end{aligned}$$

(4.19)

with the boundary conditions of (4.18).

We also consider the following Neumann eigenvalue problem in the domain $\omega \subset {\mathbb {R}}^{N - 1}$:

$$\begin{aligned} \left\{ \begin{array}{rcll} - \Delta _{\mathbb {R}^{N-1}} \psi &{} = &{} \lambda \psi &{} \quad \text { in } \omega \\ \displaystyle \frac{\partial \psi }{\partial \nu _{\partial \omega }} &{} = &{} 0 &{} \quad \text { on } \partial \omega \end{array} \right. \end{aligned}$$

(4.20)

where $- \Delta _{\mathbb {R}^{N-1}} = - \sum _{i = 1}^{N - 1} \frac{\partial ^2}{\partial x_i^2}$ is the Laplacian in ${\mathbb {R}}^{N - 1}$, i.e. with respect to the variables $x_1,\ldots ,x_{N-1}$. We denote its eigenvalues by

$$\begin{aligned} 0 = \lambda _0(\omega ) < \lambda _1(\omega ) \le \lambda _2 (\omega ) \le \ldots . \end{aligned}$$

(4.21)

It is well-known that $\lambda _j(\omega ) \nearrow + \infty$ as $j \rightarrow \infty$ and that the normalized eigenfunctions form a basis $(\psi _j)_j$ of the tangent space $T_0M$ defined in (4.14) when $\varphi \equiv 0$.

Lemma 4.5

The spectra of $L_{u_\omega }$, ${\widehat{L}}_{u_\omega }$ and $- \Delta _{\mathbb {R}^{N-1}}$ with respect to the above boundary conditions are related by

$$\begin{aligned} \sigma (L_{u_\omega }) = \sigma ({\widehat{L}}_{u_\omega }) + \sigma (- \Delta _{\mathbb {R}^{N-1}}). \end{aligned}$$

(4.22)

Proof

We begin by showing that $\sigma (L_{u_\omega }) \subset \sigma ({\widehat{L}}_{u_\omega }) + \sigma (- \Delta _{\mathbb {R}^{N-1}})$. Let $\tau \in \sigma (L_{u_\omega })$ and let $\phi \in H_0^1(\Omega _\omega \cup \Gamma _{1,0})$ be an associated eigenfunction, that is, $\phi$ is a weak solution of

$$\begin{aligned} \left\{ \begin{array}{rcll} - \Delta \phi - f'(u_\omega ) \phi &{} = &{} \tau \phi &{} \quad \text { in } \Omega _\omega \\ \phi &{} = &{} 0 &{} \quad \text { on } \Gamma _0 \\ \displaystyle \frac{\partial \phi }{\partial \nu } &{} = &{} 0 &{} \quad \text { on } \Gamma _{1, 0} \end{array} \right. \end{aligned}$$

(4.23)

As observed at the beginning of this subsection for the the nonlinear problem (4.2), by the shape of $\Omega _\omega$ and the boundary conditions, since $f \in C^{1, \alpha }({\mathbb {R}})$, by standard elliptic regularity, we have that $\phi$ is a classical solution of (4.23) in $\overline{\Omega }_\omega$.

Let $\lambda$ be an eigenvalue of $- \Delta _{\mathbb {R}^{N-1}}$ with homogeneous Neumann boundary condition on $\omega$ and let $\psi$ be an associated eigenfunction. Define

$$\begin{aligned} z(x_N) := \int _\omega \phi (x', x_N) \psi (x') \ dx'. \end{aligned}$$

(4.24)

Then, differentiating with respect to $x_N$, using Green’s formulas and the boundary conditions we have

$$\begin{aligned} - z''&= \int _\omega - \frac{\partial ^2 \phi }{\partial x_N^2} \psi \ dx' \nonumber \\&= \int _\omega (- \Delta \phi + \Delta _{\mathbb {R}^{N-1}} \phi ) \psi \ dx' \nonumber \\&= \int _\omega f'(u_\omega ) \phi \psi \ dx' + \int _\omega \tau \phi \psi \ dx' + \int _\omega \Delta _{\mathbb {R}^{N-1}} \psi \phi \ dx' \nonumber \\&= f'(u_\omega ) z + \tau z - \lambda z. \end{aligned}$$

Thus $(\tau - \lambda ) \in \sigma ({\widehat{L}}_{u_\omega })$ and hence $\tau = (\tau - \lambda ) + \lambda \in \sigma ({\widehat{L}}_{u_\omega }) + \sigma (- \Delta _{\mathbb {R}^{N-1}})$.

To show the reverse inclusion, let $\alpha \in \sigma (\widehat{L}_{u_\omega })$, $\lambda \in \sigma ( - \Delta _{\mathbb {R}^{N-1}})$ and let $z, \psi$ be, respectively, the associated eigenfunctions. Setting for $x = (x', x_N)\in \Omega _\omega$

$$\begin{aligned} \phi (x', x_N) := z(x_N) \psi (x'), \end{aligned}$$

we note that

$$\begin{aligned} - \Delta \phi&= - z'' \psi - \Delta _{\mathbb {R}^{N-1}} \psi z \nonumber \\&= f'(u_\omega ) z \psi + \alpha z \psi + \lambda z \psi \nonumber \\&= f'(u_\omega ) \phi + (\alpha + \lambda )\phi . \end{aligned}$$

(4.25)

Finally, by construction, we easily check that $\phi$ satisfies the boundary conditions of (4.23). As a consequence, we deduce that

$$\begin{aligned} \alpha + \lambda \in \sigma (L_{u_\omega }) \end{aligned}$$

and this concludes the proof. $\square$

Corollary 4.6

The problem (4.17) admits zero as an eigenvalue if and only if there exist $i \in {\mathbb {N}}^+$ and $j \in {\mathbb {N}}$ such that

$$\begin{aligned} \alpha _i + \lambda _j(\omega ) = 0 \end{aligned}$$

holds.

Proof

It follows immediately from Lemma 4.5. $\square$

Corollary 4.7

A one-dimensional solution of (4.2) is nondegenerate if both the following conditions are satisfied:

• (i) the eigenvalue problem (4.18) in (0, 1) does not admit zero as an eigenvalue;

• (ii) $\lambda _1(\omega ) > - \alpha _1$.

Proof

Analogous to the proof of Corollary 3.7. $\square$

4.3 Stability/Instability of the Pair $\mathbf {(\Omega _\omega , \textit{u}_\omega )}$

In this subsection, we prove a general stability/instability theorem for the pair $(\Omega _\omega , u_\omega )$. We begin with some preliminary results.

Firstly, we recall that when $\varphi \equiv 0$ the tangent space $T_0M$ is given by

$$\begin{aligned} T_0M = \left\{ v \in C^2({\overline{\omega }}) \ : \ \int _\omega v \ dx' = 0 \right\} . \end{aligned}$$

(4.26)

Since $u_\omega$ depends on $x_N$ only, in order to simplify the notations, we denote with a prime the derivative with respect to $x_N$, and thus we write

$$\begin{aligned} u_\omega ^\prime (x_N)=u_\omega ^\prime (x^\prime ,x_N) \, {:}{=}\, \frac{\partial u_\omega }{\partial x_N}(x', x_N). \end{aligned}$$

Then, for $v \in T_0 M$, we have that the function ${\widetilde{u}}$ (see (4.5)), which belongs to $H^1(\Omega _\omega )$, is a weak solution of

$$\begin{aligned} \left\{ \begin{array}{rcll} - \Delta {\widetilde{u}} &{} = &{} f'(u_\omega ) {\widetilde{u}} &{} \quad \text { in } \Omega _\omega \\ {\widetilde{u}} &{} = &{} - u_\omega '(1) v &{} \quad \text { on } \Gamma _0 \\ \displaystyle \frac{\partial {\widetilde{u}}}{\partial \nu } &{} = &{} 0 &{} \quad \text { on } \Gamma _{1, 0} \end{array} \right. \end{aligned}$$

(4.27)

As before, by elliptic regularity we know that ${\widetilde{u}}$ is regular in ${\overline{\Omega }}_\omega$, and thus it is a classical solution. We also note that, by the nondegeneracy of $u_\omega$, there exists a unique solution of (4.27).

Lemma 4.8

Let $\lambda _j > 0$ be any positive eigenvalue for the Neumann problem (4.20) and let $\psi _j$ be any normalized eigenfunction associated to $\lambda _j$. Let ${\widetilde{u}}_j \in H^1(\Omega _\omega )$ be the solution of (4.27) with $v = \psi _j$. Then the function

$$\begin{aligned} h_j(x_N) := \int _\omega {\widetilde{u}}_j(x', x_N) \psi _j(x') \ dx', \quad x_N \in (0, 1] \end{aligned}$$

(4.28)

satisfies

$$\begin{aligned} {\left\{ \begin{array}{ll} - h_j'' - f'(u_\omega )h_j = - \lambda _j h_j \quad \text { in } \quad (0, 1) \\ h_j(1) = - u_\omega '(1) \\ h_j'(0) = 0 \end{array}\right. } \end{aligned}$$

(4.29)

Proof

For simplicity of notation we drop the index j and simply write ${\tilde{u}}$, h, $\psi$ and $\lambda$ instead of ${\tilde{u}}_j$, $h_j, \psi _j$ and $\lambda _j$.

First observe that, as ${\tilde{u}}=- u_\omega '(1)\psi$ on $\Gamma _0$, we have

$$\begin{aligned} h(1) = \int _\omega - u_\omega '(1) \psi ^2 \ dx' = - u_\omega '(1). \end{aligned}$$

Now, differentiating with respect to $x_N$ under the integral sign and using Green’s formula, taking into account the boundary conditions, we have

$$\begin{aligned} - h''&= \int _\omega - \frac{\partial ^2 {\widetilde{u}}}{\partial x_N^2} \psi \ dx' = \int _\omega (- \Delta {\widetilde{u}} + \Delta _{\mathbb {R}^{N-1}}{\widetilde{u}}) \psi \ dx' \nonumber \\&= \int _\omega f'(u_\omega ) {\widetilde{u}} \psi \ dx' + \int _\omega \Delta _{\mathbb {R}^{N-1}} {\widetilde{u}} \psi \ dx'\nonumber \\&= f'(u_\omega ) h + \int _\omega {\widetilde{u}} \Delta _{\mathbb {R}^{N-1}} \psi \ dx' \nonumber \\&= f'(u_\omega ) h - \lambda \int _\omega {\widetilde{u}} \psi \ dx' = f'(u_\omega )h - \lambda h. \end{aligned}$$

Finally, exploiting the Neumann condition for ${\widetilde{u}}$ on $\Gamma _{1, 0}$, we check that $h'(0) = 0$. $\square$

Remark 4.9

Note that for ${\widetilde{u}}_j$, $h_j$ as in Lemma 4.8 we have that

$$\begin{aligned} {\widetilde{u}}_j (x', x_N) = h_j(x_N) \psi _j(x'). \end{aligned}$$

Indeed:

$$\begin{aligned} - \Delta (h_j(x_N) \psi _j(x'))&= - h_j(x_N) \Delta _{\mathbb {R}^{N-1}} \psi _j(x') - h_j''(x_N) \psi _j(x') \nonumber \\&= \lambda _j h_j(x_N) \psi _j(x') + f'(u_\omega )h_j(x_N) \psi _j(x') - \lambda _j h_j(x_N) \psi _j(x') \nonumber \\&= f'(u_\omega ) {\widetilde{u}}_j. \end{aligned}$$

Moreover, by (4.29) and (4.20), the function $h_j \psi _j$ satisfies the boundary conditions in (4.27), so that $h_j \psi _j$ is the unique solution of (4.27) and thus coincides with ${\widetilde{u}}_j$.

Proposition 4.10

Let $j \ge 1$, $\lambda _j$ be a positive Neumann eigenvalue of $- \Delta _{\mathbb {R}^{N-1}}$ in $\omega$, and let $h_j$ be the solution of (4.29). Assume that $-\alpha _1 < \lambda _j$, where $\alpha _1$ is the smallest eigenvalue of (4.18). Then it holds that

$$\begin{aligned} h_j > 0 \quad \text { in } [0, 1]. \end{aligned}$$

Proof

We can reflect $h_j$ by parity with respect to 0 to have a solution of the linear problem

$$\begin{aligned} {\left\{ \begin{array}{ll} - h_j'' - f'(u_\omega ) h_j + \lambda _j h_j = 0 \quad \text { in } (-1, 1) \\ h_j(-1) = h_j(1) = -u_\omega '(1) > 0. \end{array}\right. } \end{aligned}$$

(4.30)

By reflection and (4.18), the first eigenvalue of the linear operator

$$\begin{aligned} z'' - f'(u_\omega )z \quad \hbox { in}\ (0, 1) \end{aligned}$$

with the boundary condition $z(-1) = z(1) = 0$ is exactly $\alpha _1$. Therefore the first eigenvalue of the linear operator

$$\begin{aligned} {\widetilde{L}}_{u_\omega }{g} = - g''- f'(u_\omega )g + \lambda _j g \end{aligned}$$

with zero boundary condition in $(-1, 1)$ is $\beta _1 = \alpha _1 + \lambda _j$.

It is well-known that ${\widetilde{L}}_{u_\omega }$ satisfies the maximum principle whenever $\beta _1 > 0$, i.e., when $\lambda _j > - \alpha _1$. Therefore, by (4.30), the function $h_j$ satisfies $h_j \ge 0$ in $(-1,1)$, and by the strong maximum principle we conclude that $h_j > 0$ in $(-1, 1)$. $\square$

We can now state and prove the main result of this section.

Theorem 4.11

Let $\omega \subset {\mathbb {R}}^{N - 1}$ be a smooth bounded domain. Let $f\in C^{1, \alpha }_{loc}(\mathbb {R})$ such that there exists a positive one-dimensional non-degenerate solution $u_\omega$ of (1.1) in $\Omega _\omega$, and let $h_1$ be the solution to (4.29) with $j=1$. Let $\lambda _1=\lambda _1(\omega )$ be the first non-trivial eigenvalue of $-\Delta _{\mathbb {R}^{N-1}}$ with homogeneous Neumann conditions, let $\alpha _1$ be the first-eigenvalue of (1.10) and let $\rho$ be the number defined by

$$\begin{aligned} \rho \, {:}{=}\, - f(u_\omega (0))h_1(0) -\lambda _1\int _0^1 h_1u_\omega ^\prime \, dx_N . \end{aligned}$$

(4.31)

Assume that $\lambda _1 > - \alpha _1$. Then

• (i) if $\rho <0$, then $(\Omega _\omega , u_\omega )$ is an unstable energy-stationary pair;

• (ii) if $\rho >0$, then $(\Omega _\omega , u_\omega )$ is a stable energy stationary pair.

Proof

We first observe that since $\frac{\partial u_\omega }{\partial \nu }$ is constant on $\Gamma _0$ then, by the analogous of Proposition 3.3 for cylinders, we infer that the pair $(\Omega _\omega , u_\omega )$ is an energy-stationary pair.

Let $w \in T_0M$ and assume without loss of generality that $\int _\omega w^2 \ dx' = 1$. In order to prove (i)-(ii) we first determine a suitable expression for $I''(0)[w, w]$. To this end, for each $j \in {\mathbb {N}}^+$, let ${\widetilde{u}}_j$ be the solution of (4.27) with $v = \psi _j$ and let $h_j$ be the solution of (4.29). Then we can write

$$\begin{aligned} w = \sum _{j = 1}^\infty (w, \psi _j) \psi _j \end{aligned}$$

where $(\cdot , \cdot )$ is the inner product in $L^2(\omega )$. Moreover, we can check that

$$\begin{aligned} {\widetilde{u}} = \sum _{j = 1}^\infty (w, \psi _j) {\widetilde{u}}_j \end{aligned}$$

is the solution of (4.27) corresponding to w. Then, taking $\varphi =0$ in Lemma 4.2, exploiting Lemma 4.4, taking into account that by Proposition 4.3 the Lagrange multiplier $\mu$ is given by

$$\begin{aligned} \mu = - \frac{1}{2}(u_\omega '(1))^2, \end{aligned}$$

by Remark 4.9 and observing that $\nabla u_\omega \perp (\nabla _{{\mathbb {R}}^{N - 1}} w, 0)$, we infer that

$$\begin{aligned} I''(0)[w, w]&= - \frac{1}{2} \int _\omega (u_\omega '(1))^2 w^2 \ dx' \nonumber \\&\quad - \int _\omega u_\omega '(1) \left( \sum _{j = 1}^\infty (w, \psi _j) h_j'(1) \psi _j \right) \left( \sum _{k = 1}^\infty (w, \psi _k) \psi _k\right) \ dx' \nonumber \\&\quad - \int _\omega u_\omega '(1) u_\omega ''(1) w^2 \ dx' + \frac{1}{2} (u_\omega '(1))^2 \int _\omega w^2 \ dx' \nonumber \\&= - u_\omega '(1) \int _\omega \left( \sum _{j = 1}^\infty (w, \psi _j)^2 h_j'(1) \psi _j^2 \ \right) dx' - u_\omega '(1) u_\omega ''(1) \end{aligned}$$

Finally, since $u_\omega$ is a solution to (4.16) we deduce that

$$\begin{aligned} I''(0)[w, w]=- u_\omega '(1) \int _\omega \left( \sum _{j = 1}^\infty (w, \psi _j)^2 h_j'(1) \psi _j^2 \ \right) dx' + u_\omega '(1) f(0). \end{aligned}$$

(4.32)

In particular, choosing $w=\psi _1$ and plugging it into (4.32) we infer that

$$\begin{aligned} I''(0)[\psi _1, \psi _1]=- u_\omega '(1) h_1'(1) + u_\omega '(1) f(0). \end{aligned}$$

(4.33)

Multiplying the equation in (4.29) (with $j=1$) by $u^\prime _\omega$ and integrating by parts we get

$$\begin{aligned} -(h_1^\prime u_\omega ^\prime )\big |_0^1 + \int _0^1h_1^\prime u_\omega ^{\prime \prime }\, dx_N = \int _0^1(f^\prime (u_\omega )-\lambda _1)h_1u_\omega ^\prime \, dx_N. \end{aligned}$$

Exploiting (4.16), integrating by parts and taking into account that $h_1(1)=-u_\omega ^\prime (1)$ we obtain

$$\begin{aligned} \begin{array}{lll} &{}&{}\displaystyle -h_1^\prime (1)u_\omega ^\prime (1) - \int _0^1h_1^\prime f(u_\omega )\, dx_N\\ {} &{}=&{}\displaystyle \int _0^1f^\prime (u_\omega )u_\omega ^\prime h_1\, dx_N -\lambda _1\int _0^1 h_1 u_\omega ^\prime \, dx_N\\ &{}=&{}\displaystyle (f(u_\omega )h_1)\big |_0^1 - \int _0^1 f(u_\omega ) h_1'\, dx_N -\lambda _1\int _0^1 h_1 u_\omega ^\prime \, dx_N\\ &{}=&{}\displaystyle -f(0)u_\omega ^\prime (1)-f(u_\omega (0))h_1(0) - \int _0^1 f(u_\omega ) h_1'\, dx_N -\lambda _1\int _0^1 h_1 u_\omega ^\prime \, dx_N\\ \end{array} \end{aligned}$$

(4.34)

Hence, we deduce that

$$\begin{aligned} -h_1^\prime (1)u_\omega ^\prime (1) =-f(0)u_\omega ^\prime (1)-f(u_\omega (0))h_1(0) -\lambda _1\int _0^1 h_1 u_\omega ^\prime \, dx_N \end{aligned}$$

(4.35)

In the end, from (4.33), (4.35) and recalling (4.31), we obtain

$$\begin{aligned} I''(0)[\psi _1, \psi _1]=-f(u_\omega (0))h_1(0) -\lambda _1\int _0^1 h_1 u_\omega ^\prime \, dx_N=\rho . \end{aligned}$$

Therefore, if $\rho <0$ then $I''(0)[\psi _1, \psi _1]<0$, i.e., $(\Omega _\omega , u_\omega )$ is an unstable energy-stationary pair, and this proves (i).

Let us prove (ii). Let $w \in T_0M$ such that $\int _\omega w^2 \ dx' = 1$. From (4.32) we know that $I''(0)[w, w]=- u_\omega '(1) \int _\omega \left( \sum _{j = 1}^\infty (w, \psi _j)^2 h_j'(1) \psi _j^2 \ \right) dx' + u_\omega '(1) f(0)$. Thanks to the assumption $\lambda _1>-\alpha _1$ the following holds true.

Claim: if $k > j$, then

$$\begin{aligned} h_k'(1) {\ge } h_j'(1), \end{aligned}$$

(4.36)

and actually $h_k'(1) > h_j'(1)$ if $\lambda _k > \lambda _j$.

Indeed, by definition $h_k$, $h_j$ satisfy, respectively, the following:

$$\begin{aligned}&- h_k'' - f'(u_\omega ) h_k = - \lambda _k h_k, \end{aligned}$$

(4.37)

$$\begin{aligned}&- h_j'' - f'(u_\omega ) h_j = - \lambda _j h_j. \end{aligned}$$

(4.38)

Multiplying (4.37) by $h_j$ and integrating on (0, 1) we obtain

$$\begin{aligned} \int _0^1 - h_k'' h_j \, d x_N&= \int _0^1 h_k' h_j' \, d x_N - (h_k' h_j)\big |_0^1 \nonumber \\&= \int _0^1 f'(u_\omega ) h_j h_k \, d x_N - \lambda _k \int _0^1 h_j h_k \, d x_N \end{aligned}$$

(4.39)

Similarly, multiplying (4.38) by $h_k$, integrating on (0, 1) and then subtracting the result from (4.39), we obtain

$$\begin{aligned} - (h_k' h_j - h_j' h_k)(1) = (\lambda _j - \lambda _k) \int _0^1 h_j h_k \, dx_N\ {\le }\ 0, \end{aligned}$$

(4.40)

because $h_j > 0$ and $h_k > 0$ (see Proposition 4.10, which holds true for any $j\in \mathbb {N}^+$ because $\lambda _1>-\alpha _1$). Now, since $h_j(1)=h_k(1)=-u_\omega (1)$, then by (4.40) we deduce that

$$\begin{aligned} u_\omega '(1)(h_k'(1) - h_j'(1))\le 0. \end{aligned}$$

Hence, as $u_\omega '(1)<0$, Claim (4.36) easily follows.

Now, thanks to (4.32) and Claim (4.36), recalling again that $u'_\omega (1) < 0$ and exploiting (4.35) it follows that

$$\begin{aligned} \begin{array}{lll} \displaystyle I''(0)[w, w]&{}\ge &{}\displaystyle - u_\omega '(1)h_1^\prime (1)\int _\omega \left( \sum _{j = 1}^\infty (w, \psi _j)^2\psi _j^2 \ \right) dx' + u_\omega '(1) f(0)\\ &{}=&{}\displaystyle - u_\omega '(1)h_1^\prime (1) + u_\omega '(1) f(0)\\ &{}=&{}\displaystyle -f(u_\omega (0))h_1(0) -\lambda _1\int _0^1 h_1 u_\omega ^\prime \, dx_N=\rho . \end{array} \end{aligned}$$

(4.41)

Hence, if $\rho >0$ we have that $I''(0)[w, w]>0$ for all $w \in T_0M$, i.e., $(\Omega _\omega , u_\omega )$ is a stable energy-stationary pair, and this proves (ii). The proof is complete. $\square$

As a simple corollary of Theorem 4.11 we can now prove the stability/instability result of Theorem 1.4, which concerns the case of the torsional energy, i.e. when $f \equiv 1$.

4.4 Proof of Theorem 1.4

When $f\equiv 1$ the eigenvalue problem (4.18) has only positive eigenvalues and therefore the condition $\lambda _1 > - \alpha _1$ is automatically satisfied. The only solution of

$$\begin{aligned} \left\{ \begin{array}{rcll} - \Delta u &{} = &{} 1 &{} \quad \text { in } \Omega _\omega \\ u &{} = &{} 0 &{} \quad \text { on } \Gamma _0 \\ \displaystyle \frac{\partial u}{\partial \nu } &{} = &{} 0 &{} \quad \text { on } \Gamma _{1, 0} \end{array} \right. \end{aligned}$$

(4.42)

is the one-dimensional positive function given by

$$\begin{aligned} u_\omega (x_N)=\frac{1-x_N^2}{2}. \end{aligned}$$

(4.43)

Clearly, as $u_\omega '(1)=-1$ and $f\equiv 1$, then for any $j\in \mathbb {N}^+$ (4.29) reduces to

$$\begin{aligned} {\left\{ \begin{array}{ll} - h_j'' + \lambda _j h_j = 0 \quad \text { in } \quad (0, 1) \\ h_j(1) = - u_\omega '(1) \\ h_j'(0) = 0 \end{array}\right. } \end{aligned}$$

whose unique solution is given by

$$\begin{aligned} h_j(x_N)=\frac{1}{\cosh (\sqrt{\lambda _j})}\cosh (\sqrt{\lambda _j} x_N). \end{aligned}$$

In particular, taking $j=1$ and exploiting (4.43) we can compute explicitly the number $\rho$ in (4.31), namely

$$\begin{aligned} \rho =-\frac{1}{\cosh (\sqrt{\lambda _1})} +\frac{\lambda _1}{\cosh (\sqrt{\lambda _1})}\int _0^1 \cosh (\sqrt{\lambda _1} x_N) x_N\, dx_N . \end{aligned}$$

Integrating by parts we readily check that

$$\begin{aligned} \int _0^1 \cosh (\sqrt{\lambda _1} x_N) x_N\, dx_N=\frac{\sinh (\sqrt{\lambda _1})}{\sqrt{\lambda _1}} -\frac{\cosh (\sqrt{\lambda _1})}{\lambda _1}+\frac{1}{\lambda _1}, \end{aligned}$$

and thus we obtain

$$\begin{aligned} \rho = \sqrt{\lambda _1} \tanh (\sqrt{\lambda _1})-1. \end{aligned}$$

(4.44)

Let us consider the function $g:[0,+\infty [ \rightarrow \mathbb {R}$, defined by $g(t)= \sqrt{t} \tanh (\sqrt{t})-1$. Clearly $g(0)=-1$ and $g(t)\rightarrow +\infty$ as $t\rightarrow +\infty$ and by monotonicity we infer that g has a unique zero in $]0,+\infty [$. We denote it by $\beta$ and from the previous argument and (4.44) we infer that $\rho <0$ if and only if $\lambda _1<\beta$. Then, by Theorem 4.11-(i) we get that $(\Omega _\omega , u_\omega )$ is an unstable energy-stationary pair, and this proves (i).

Analogously, as $\rho >0$ if and only if $\lambda _1>\beta$, from Theorem 4.11-(ii) we obtain that $(\Omega _\omega , u_\omega )$ is a stable energy-stationary pair. The proof is complete. $\square$

We conclude this section with the proof of Theorem 1.5.

4.5 Proof of Theorem 1.5

Let $w \in T_0M$ such that $\int _\omega w^2 \ dx' = 1$. Since $\lambda _1>-\alpha _1$, we can argue as in the proof of Theorem 4.11-(ii), in particular, from the first two lines of (4.41), taking into account that, by assumption, $f(0)=0$, we have

$$\begin{aligned} I''(0)[w, w]\ge \displaystyle - u_\omega '(1)h_1^\prime (1). \end{aligned}$$

(4.45)

Now, since $h_1'' = (\lambda _1- f'(u_\omega )) h_1$ in (0, 1) and $h_1>0$ in [0, 1] by Proposition 4.10, then, thanks to the assumption $\lambda _1>\sup _{x_N\in (0,1)}|f^\prime (u_\omega (x_N))|$ we infer that $h_1''>0$ in [0, 1]. In particular, as $h_1^\prime (0)=0$ we deduce that

$$\begin{aligned} h_1^\prime (1)>0. \end{aligned}$$

(4.46)

Finally, combining (4.45) and (4.46) we obtain that $I''(0)[w, w]>0$ for all $w \in T_0M$, which means that $(\Omega _\omega , u_\omega )$ is a stable energy-stationary pair. $\square$

Remark 4.12

We notice that, if f is a non-negative monotone increasing function, as in the case of the Lane-Emden nonlinearity (1.3), then by the Gidas-Ni-Nirenberg theorem ( [15]) and by the monotonicity of f we infer that $\sup _{x_N\in (0,1)}|f^\prime (u_\omega (x_N))|=f^\prime (u_\omega (0))$. Thus the stability condition of Theorem 1.5 reduces to

$$\begin{aligned} \lambda _1>f^\prime (u_\omega (0)). \end{aligned}$$

Remark 4.13

In the case of the Lane-Emden nonlinearity $f(u) = u^p$, at least for some integer values of p, it is possible to compute the solution $u_\omega$ numerically, as well as the eigenvalue $\alpha _1$ and the function $h_1$ for different values of $\lambda _1(\omega )$. This allows to compute $\rho$ numerically, so that, plotting the result for $\rho$ as a function of $\lambda _1(\omega )$, we obtain a region of instability for $\lambda _1(\omega )$ close to $- \alpha _1$.