Energy stability for a class of semilinear elliptic problems

In this paper, we consider semilinear elliptic problems in a bounded domain $\Omega$ contained in a given unbounded Lipschitz domain $\mathcal C \subset \mathbb R^N$. Our aim is to study how the energy of a solution behaves with respect to volume-preserving variations of the domain $\Omega$ inside $\mathcal C$. Once a rigorous variational approach to this question is set, we focus on the cases when $\mathcal C$ is a cone or a cylinder and we consider spherical sectors and radial solutions or bounded cylinders and special one-dimensional solutions, respectively. In these cases, we show both stability and instability results, which have connections with related overdetermined problems.

We consider the following semilinear elliptic problem: in Ω u = 0 on Γ Ω ∂u ∂ν = 0 on Γ 1,Ω (1.1) where f : R → R is a locally C 1,α nonlinearity and ν denotes the exterior unit normal vector to ∂Ω. Let u Ω be a positive weak solution of (1.1) in the Sobolev space H 1 0 (Ω ∪ Γ 1,Ω ), which is the space of functions in H 1 (Ω) whose trace vanishes on Γ Ω . By standard variational methods we have that under suitable hypotheses on f such a solution exists and is a critical point of the energy functional where F (s) = s 0 f (τ ) dτ . A classical example of a nonlinearity for which a positive solution exists for any domain Ω in C is the Lane-Emden nonlinearity, namely In this case, u Ω can be obtained, for instance, by minimizing the functional J on the Nehari manifold Given the unbounded region C, an interesting question is to understand how the energy J(u Ω ) behaves with respect to variations of a domain Ω inside C. In particular, one could ask whether the energy J(u Ω ) increases or decreases by deforming Ω into a domain Ω sufficiently close to Ω and with the same measure.
Loosely speaking, one could consider the function Ω → T (Ω) = J(u Ω ) and study it in a suitable "neighborhood" of Ω. Under this aspect, domains Ω which are local minima of T could be particularly interesting. This question could be attacked by differentiating T (Ω) with respect to variations of Ω which leave the volume invariant and studying the stability or instability of its critical points. However, since (1.1) is a nonlinear problem and solutions of (1.1) are not unique in general, it is not clear a priori how to well define the functional T (Ω).
We will show in Section 2 that for nondegenerate solutions u Ω of (1.1) the energy functional T (Ω) is well defined for domains obtained by small deformations of Ω induced by vector fields which leave C invariant.
We remark that the study of the stationary domains of the energy functional T (Ω) with a volume constraint is strictly related to the overdetermined problem obtained from (1.1) by adding the condition that the normal derivative ∂u ∂ν is constant on Γ Ω , see Proposition 2.5. This is well-known for a Dirichlet problem in R N and when T (Ω) is globally defined for all domains Ω ⊂ R N (as in the case of the torsion problem, i.e. f ≡ 1). It has been observed in [18] and [14] in the relative setting of the cone.
The existence or not of domains that are local minimizers of the energy and their shapes obviously depend on the unbounded region C where the domains Ω are contained. In this paper, we consider unbounded cones and cylinders, in which there are some particular domains that, for symmetry or other geometric reasons, could be natural candidates for being local minimizers of the energy.
Let us first describe the case when C is a cone Σ D defined as Σ D := {x ∈ R N : x = tq, q ∈ D, t > 0}, (1.4) where D is a smooth domain on the unit sphere S N −1 .
In Σ D we consider the spherical sector Ω D obtained by intersecting the cone with the unit ball centered at the origin, i.e. Ω D = Σ D ∩ B 1 . In Ω D we can consider a radially symmetric solution u D of problem (1.1), for the nonlinearities f for which they exist. Obviously, u D is a radial solution of the analogous Dirichlet problem in the unit ball B 1 .
In Section 3 we show that, whenever u D is a nondegenerate solution of (1.1), then the pair (Ω D , u D ) is energy-stationary in the sense of Definition 2.4 and investigate its "stability" as a critical point of the energy functional T , which is well defined for small perturbations of Ω D (see Sections 2 and 3).
The main result we get is that the stability of (Ω D , u D ) depends on the first nontrivial Neumann eigenvalue λ 1 (D) of the Laplace-Beltrami operator −∆ S N−1 on the domain D ⊂ S N −1 which spans the cone. In particular, we obtain a precise threshold for stability/instability which is independent of the nonlinearity, and on the radial positive solution considered, whenever multiple radial positive solutions exist. Let us remark that for several nonlinearities the radial positive solution is unique (see [16]). For example, this is the case if f (u) = u p , p > 1.
To state precisely our result we need to introduce the first eigenvalue ν 1 of the following singular eigenvalue problem: This problem arises naturally when studying the spectrum of the linearized operator −∆− f ′ (u D ). We refer to Section 3 for more details. Remark 1.2. The case N = 2 is special and in this case, the overdetermined torsion problem has been completely solved in [17] using that the boundary of any cone in dimension 2 is flat. In the nonlinear case, the condition N ≥ 3 arises from the study of an auxiliary singular problem (see Proposition 3.12). It is important to observe that the singular eigenvalue ν 1 which appears in (i) is larger than −(N − 1) for all autonomous nonlinearities f (u) (see [5,Proposition 3.4]). Thus the condition Let us comment on the meaning of Theorem 1.1. The statement (ii) will be proved by showing that the quadratic form corresponding to the second derivative of the energy functional, with a fixed volume constraint, is positive definite in all directions. This means that the spherical sector locally minimizes the energy among small volume preserving perturbations of Ω D and of the corresponding radial solution u D .
On the contrary, when − ν 1 < λ 1 (D) < N − 1, by (i) we have that the pair (Ω D , u D ) is unstable and therefore Ω D is not a local minimizer of the energy. This means that there exist small volume preserving deformations of the spherical sector Ω D which produce domains Ω t and solutions u t of (1.1) in Ω t whose energy J(u t ) is smaller than the energy J(u D ) of the positive radial solution u D in the spherical sector Ω D .
Moreover, observe that the function f = f (s) could satisfy suitable hypotheses such that problem (1.1) has a unique positive solution u Ω in any domain Ω ⊂ Σ D (or more generally in Ω ⊂ C). This is the case, for example, when f ≡ 1, i.e., (1.1) is a "relative" torsion problem. Then the energy functional T (Ω) = J(u Ω ) is well defined for any domain Ω ⊂ Σ D . Hence we may ask whether a global minimum for T exists, once the volume of Ω is fixed, and is given by the spherical sector Ω D . This question has been addressed in [17], [18] and [14] when f ≡ 1, showing that Ω D is a global minimizer if Σ D is a convex cone ( [18]), as a consequence of an isoperimetric inequality introduced in [15], see also [4,11,19]. Instead, in [14] it is proved that Ω D is not a local minimizer whenever λ 1 (D) < N − 1, which is the same threshold we get in Theorem 1.1 for general nonlinearities.
The other example of an unbounded domain we consider in the present paper is a half-cylinder, defined as Σ ω := ω × (0, +∞) ⊂ R N , (1.6) where ω ⊂ R N −1 is a smooth bounded domain. We denote the points in Σ ω by x = (x ′ , x N ), x ′ ∈ ω. In this case, a geometrically simple domain we consider is the bounded cylinder (1.7) In Ω ω we consider a positive solution which is obtained by trivially extending to Ω ω a positive one-dimensional solution of the problem for a nonlinearity f for which such a solution exists. Before stating the results concerning the stability of the pair (Ω ω , u ω ) we again consider an auxiliary eigenvalue problem (but not singular): The problem (1.10) is considered in Section 4 to study the spectrum of the linearized operator −∆ − f ′ (u ω ). We denote by α 1 the first eigenvalue of (1.10).
We start by stating a sharp stability/instability result for the torsion problem, i.e., taking f ≡ 1 in (1.1).
Note that the number β that gives the threshold for the stability is independent of the dimension N . Its value is obtained by solving numerically the equation √ λ 1 tanh( √ λ 1 ) − 1 = 0 (see (4.44) in the proof of Theorem 1.3).
It is interesting to observe that the instability result of Theorem 1.3 is related to a bifurcation theorem obtained in [10]. Indeed, if we consider the cylinder Σ ω in R 2 , in which case ω is simply an interval in R and Ω ω is a rectangle, a byproduct of Theorem 1.1 of [10] is the existence of a domain Ω ω in Σ ω that is a small deformation of the rectangle Ω ω and in which the overdetermined problem has a solution. By looking at the proof of [10] and relating it to our instability result it is clear that the bifurcation should occur when the eigenvalue λ 1 (ω) crosses the value β provided by Theorem 1.3.
The proof of Theorem 1.3 can be derived from a general condition for the stability of the pair (Ω ω , u ω ) in the nonlinear case, which is obtained in Theorem 4.11. The proof of Theorem 4.11 involves auxiliary functions that appear naturally in the study of derivatives of the energy functional T , see Section 4.
Let us remark that in the case when f ≡ 1 we succeed in obtaining the sharp bound of Theorem 1.3 because the solution given by (1.8) and (1.9) is explicit: and so are the auxiliary functions which are solutions of simple linear ODEs. This allows us to use the condition of Theorem 4.11 to obtain Theorem 1.3. The result of Theorem 1.3 gives a striking difference between the torsional energy problem and the isoperimetric problem in cylinders. Indeed, Proposition 2.1 of [1] shows that the only stationary cartesian graphs for the perimeter functional are the flat ones. Instead, Theorem 1.3 (as well as the result of [10]) indicate that there are domains for which the overdetermined problem relative to (1.1), with f ≡ 1, has a solution and whose relative boundary is a non-flat cartesian graph.
For the semilinear problem, we obtain a stability result for a large class of nonlinearities as soon as the eigenvalue λ 1 (ω) is sufficiently large. Indeed, we have Theorem 1.4. Let Σ ω and Ω ω be as in (1.6) and (1.7), and let u ω be a positive onedimensional solution of (1.1) in Ω ω . Let α 1 be the first eigenvalue of (1.10) and let λ 1 (ω) be as in Theorem 1.3. If the nonlinearity f satisfies f (0) = 0 and then the pair (Ω ω , u ω ) is a stable energy-stationary pair.
The condition (1.11) shows that the stability depends on an interplay between the geometry of the cylinder Σ ω (through the eigenvalue λ 1 (ω)) and the nonlinearity f . On the contrary, numerical evidence shows, for the Lane-Emden nonlinearity (1.3), that, if λ 1 is sufficiently close to −α 1 , instability occurs, see Remark 4.13.
Concerning the eigenvalue α 1 in the bound (1.11), as well as the analogous one, λ 1 (D) > − ν 1 , of Theorem 1.1, we point out that they are used in the proofs of both theorems to deduce the positivity of some auxiliary functions. It is an open problem to understand if they really play a role in the stability/instability result.
We delay further comments on the results and their proofs to the respective sections. The paper is organized as follows. In Section 2 we study problem (1.1) in domains Ω contained in a general unbounded set C. We define the energy functional and its derivative with respect to variations of Ω which leave C invariant and preserve the measure of Ω. This is done by considering nondegenerate solutions of (1.1) in Ω.
In Section 3 we consider the case when C is a cone Σ D . In this setting we take domains which are defined by smooth radial graphs over D, in particular we consider the spherical sector Ω D and a corresponding radial solution u D for which we prove the stability/instability result.
Finally in Section 4 we study the case of the cylinder Σ ω and prove the corresponding stability/instability result for the pair (Ω ω , u ω ) when Ω ω is a bounded cylinder and u ω is as in (1.8) and (1.9).

Semilinear elliptic problems in unbounded sets
In this section we consider problem (1.1) in a bounded Lipschitz domain Ω contained in an unbounded open set C which we assume to be (uniformly) Lipschitz regular.
Starting from a positive nondegenerate solution of (1.1) in Ω we show how to define an energy functional for small variations of Ω which preserve the volume.
We consider a positive weak solution u Ω of (1.1) in the Sobolev space H 1 0 (Ω ∪ Γ 1,Ω ), which is the subspace of H 1 (Ω) of functions whose trace vanishes on Γ Ω . By standard variational methods, such as constrained minimization, Mountain-Pass Theorem etc, it is easy to exhibit many nonlinearities f = f (s) for which such a solution exists. Moreover, with suitable assumptions on the growth of f we also have, by regularity results, that u Ω is a classical solution of (1.1) inside Ω and at any regular point of ∂Ω, and that u Ω is bounded (see also [5,Proposition 3.1]).
We assume that u Ω is nondegenerate, i.e., the linearized operator does not have zero as an eigenvalue in H 1 0 (Ω ∪ Γ 1,Ω ) or, in other words, L u Ω defines an isomorphism between H 1 0 (Ω ∪ Γ 1,Ω ) and its dual space. We consider small deformations of Ω which leave C invariant and would like to show that the nondegeneracy of u Ω induces a local uniqueness result for solutions of (1.1) in the deformed domains. Thus we take a one-parameter family of diffeomorphisms ξ t , for t ∈ (−η, η), η > 0, associated to a smooth vector field V such that V (x) ∈ T x ∂C for every x ∈ ∂C reg , V (x) = 0 for x ∈ ∂C \ ∂C reg , and set Ω t := ξ t (Ω), where T x ∂C denotes the tangent space to ∂C at the point x, and ∂C reg denotes the regular part of ∂C. In particular Ω 0 = Ω and in order to simplify the notations we set Γ t := Γ Ωt , Γ 1,t := Γ 1,Ωt .
(2.2) Proposition 2.1. Let u Ω be a positive nondegenerate solution of (1.1), which belongs to W 1,∞ (Ω) ∩ W 2,2 (Ω). Let V be a smooth vector field and let ξ t be the associated family of diffeomorphisms. Then there exists δ > 0 such that for any t ∈ (−δ, δ) there is a unique solution u t of the problem in a neighborhood of the function u Ω • ξ −1 t in the space H 1 0 (Ω t ∪ Γ 1,t ). Moreover, the map t → u t is differentiable.
Proof. By using the diffeomorphism ξ t we can pass from the space H 1 0 (Ω ∪ Γ 1,Ω ) to the space H 1 0 (Ω t ∪ Γ 1,t ). Indeed, Moreover, u t is a weak solution of (2.3), i.e., In other words, setting M t := M t J t , we have that u t is a solution of in the space H 1 0 (Ω ∪ Γ 1,Ω ). Now we consider the map Since u Ω is a solution in Ω and ξ 0 is the identity map we have Notice that F is differentiable with respect to to v, and as ε → 0. Hence F is differentiable and evaluating ∂ v F at (0, u Ω ) we obtain (2.8).
Note that, as for u Ω , u t is a classical solution of (2.3) in Ω t and on the regular part of ∂Ω t . By Proposition 2.1 we have that the energy functional so we can also write In the next result we show that T is differentiable with respect to t and compute its derivative at t = 0, that is, at the initial domain Ω.
Proposition 2.2. Assume that u Ω is a positive nondegenerate solution of (1.1) which belongs to W 1,∞ (Ω) ∩ W 2,2 (Ω). Then Proof. Recall from Proposition 2.1 that t → u t is smooth and (2.9) holds. Differentiating the equation −∆u t = f (u t ) with respect to t we obtain Now observe that by the hypotheses on u Ω we have that Finally, since ξ t maps ∂C into itself we have that, for all small t and x ∈ (∂C ∩ ∂Ω) reg Differentiating this relation with respect to t and evaluating at t = 0 we obtain . Then, since ∇u Ω , ν = 0 on (∂C ∩ ∂Ω) reg , and in view of (2.13), (2.15), we infer that u satisfies  in the classical sense in the interior of Ω and on the regular part of ∂Ω. Recalling (2.11) we can write Since t → f (u t )u t − F (u t ) is differentiable at t = 0, ∂Ω is Lipschitz and taking into account that u Ω ∈ W 1,∞ (Ω)∩W 2,2 (Ω), then, applying [13, Theorem 5.2.2], we can compute the derivative with respect to t of the functional T obtaining that The previous applications of the Divergence Theorem are justified by arguing as in [17,Lemma 2.1], where the regularity hypothesis on u Ω comes into play.

Remark 2.3.
It is not difficult to see that u is also a weak solution of (2.16). Indeed, let ϕ ∈ C ∞ c (Ω ∪ Γ 1,Ω ). Then, for all sufficiently small t, we also have ϕ ∈ C ∞ c (Ω t ∪ Γ 1,t ). Hence, since u t is a weak solution to (2.3), we have Now, as proved in [14,Claim (3.17)], it holds that Then, taking the derivative with respect to t in (2.18), evaluating at t = 0, and since ϕ is arbitrary, we easily conclude.
Let us now consider domains Ω ⊂ C of fixed measure c > 0 and define where admissible means that Ω ⊂ C is a bounded domain with smooth relative boundary We consider vector fields that induce deformations that preserve the volume. More precisely we take a one-parameter family of diffeomorphisms ξ t , t ∈ (−η, η), associated to a smooth vector field V such that V (x) ∈ T x ∂C reg for all x ∈ ∂C reg , and satisfying the condition Definition 2.4. We say that the pair (Ω, u Ω ) is energy-stationary under a volume con- for any vector field tangent to ∂C such that the associated one-parameter family of diffeomorphisms preserves the volume.
A characterization of energy-stationary pairs in C is the following: Let Ω ∈ A and assume that u Ω ∈ W 1,∞ (Ω)∩W 2,2 (Ω) is a nondegenerate positive solution of (1.1). Then (Ω, u Ω ) is energy-stationary under a volume constraint if and only if u Ω satisfies the overdetermined condition |∇u Ω | = constant on Γ Ω .
Proof. Let ξ t be an arbitrary admissible one-parameter family of diffeomorphisms and let V be the associated vector field. Since the volume is preserved and V (x) ∈ T x ∂C on ∂C, If |∇u Ω | is constant on Γ Ω , then (Ω, u Ω ) is energy-stationary, in view of (2.12) and (2.21).
On the other hand, if (Ω, u Ω ) is energy stationary, then for every constant a and every admissible vector field V . Assume by contradiction that |∇u Ω | is not constant on Γ Ω . Then there exists a compact set K ⊂ Γ Ω , with nonempty interior part, such that |∇u Ω | is not constant on K. Take a nonnegative cutoff function Θ such that Θ ≡ 1 in K, and choose (2.23) Then we can build a deformation from the vector field V = (|∇u Ω | 2 − a)Θν, and in this case, since (Ω, u Ω ) is energy stationary, we would have which contradicts the fact that |∇u Ω | is not constant on K. The proof is complete.
Remark 2.6. It is relevant to observe that all concepts introduced in this section apply to the case when Γ 1,Ω is empty, or, equivalently, when C = R N . Thus all the above results hold for Dirichlet problems in domains in the whole space. In this case it is known, by Serrin's Theorem (see [20]) that if a positive solution for the overdetermined problem exists, then Ω is a ball. Therefore, in view of Proposition 2.5, it follows that the only energy-stationary pairs in R N are (B, u B ), where B is a ball and u B is a nondegenerate positive solution.
Remark 2.7. We observe that all the results in this section hold true also for nondegenerate sign-changing solutions u Ω to (1.1). However, since in the sequel we study the stability in the case of positive solutions, we have considered only this case 3. The case of the cone Let D ⊂ S N −1 be a smooth domain on the unit sphere and let Σ D be the cone spanned by D, which is defined as In Σ D we consider admissible domains Ω, in the sense of (2.19), that are strictly starshaped with respect to the vertex of the cone, which we choose to be the origin 0 in R N . In other words, we consider domains whose relative boundary is the radial graph in Σ D of a function in C 2 (D, R). Hence for ϕ ∈ C 2 (D, R) we set and consider the strictly star-shaped domain Ω ϕ defined as To simplify the notation we set

Energy functional for star-shaped domains.
In Ω ϕ we consider the semilinear elliptic problem and assume throughout this section that a bounded positive nondegenerate solution u Ωϕ exists and belongs to W 1,∞ (Ω ϕ ) ∩ W 2,2 (Ω ϕ ). Then we can apply the results of Section 2 and define the energy functional T as in (2.10) for small variations of Ω ϕ . Since Ω ϕ is strictly star-shaped, this property also holds for the domains obtained by small regular deformations. Thus it is convenient to parametrize the domains and their variations by C 2 functions defined on D. Hence, for v ∈ C 2 (D, R) and t ∈ (−η, η), where η > 0 is a fixed number sufficiently small, we consider the domain variations Ω ϕ+tv ⊂ Σ D .
Then ξ| Ωϕ (t, ·) : Ω ϕ → Ω ϕ+tv is a diffeomorphism, whose inverse is ) and ξ is the flow associated to the vector field The energy functional T in (2.10) becomes a functional defined on functions in C 2 (D, R).
More precisely, we define, for every v ∈ C 2 (D, R), We now compute the first derivative of the functional T at ϕ along a direction v ∈ C 2 (D, R), i.e. the derivative with respect to t of (3.8) computed at t = 0. Lemma 3.1. Let ϕ ∈ C 2 (D, R) and assume that u ϕ is a bounded positive nondegenerate solution to (3.4) and that u ϕ belongs to W 1,∞ (Ω ϕ )∩W 2,2 (Ω ϕ ). Then for any v ∈ C 2 (D, R) it holds that Proof. The result follows from Proposition 2.2. Indeed, since the exterior unit normal to Γ ϕ is given by Hence, using the parametrization x = e ϕ(q) q, for q ∈ D, taking into account that the induced (N − 1)-dimensional area element on Γ ϕ is given by and since u ϕ = 0 on Γ ϕ , then, from (2.12), we readily obtain (3.9).
The next step is to compute the second derivative of T at Ω ϕ with respect to directions v, w ∈ C 2 (D, R) Lemma 3.2. Let ϕ and u ϕ be as in Lemma 3.1. Then for any v, w ∈ C 2 (D, R) it holds The proof is the same as that of [14, Lemma 3.2] and therefore we omit it.
In view of Definition 2.4, we are interested in studying pairs (Ω ϕ , u ϕ ) which are energystationary under a volume constraint. Thus we need to consider domains Ω ϕ with a fixed volume. We recall that the volume of the domain defined by the radial graph of a function ϕ ∈ C 2 (D, R) is given by Simple computations yield, for v, w ∈ C 2 (D, R): Then, for c > 0 we define the manifold whose tangent space at any point ϕ ∈ M is given by We restrict the energy functional to the manifold M and denote it by Clearly, if the pair (Ω ϕ , u ϕ ) is energy-stationary under a volume constraint, in the sense of Definition 2.4, then ϕ ∈ M is a critical point of I. Hence, by the Theorem of Lagrange multipliers, there exists µ ∈ R such that Moreover, the following result holds true: is energy-stationary under the volume constraint. Then the Lagrange multiplier µ is negative and Proof. The proof is the same as in [14,Lemma 4.1] For the second derivative of I we have Lemma 3.4. Let ϕ ∈ M and let v, w ∈ T ϕ M . If (Ω ϕ , u ϕ ) is energy-stationary under the volume constraint, then Proof. The proof is the same as in [14,Lemma 4.3].
3.2. Spherical sectors and radial solutions. Given a cone Σ D we consider the spherical sector Ω D obtained by intersecting Σ D with the unit ball B 1 . Obviously its relative boundary Γ Ω D is the radial graph obtained by taking ϕ ≡ 0 in (3.2), which is in fact the domain D which spans the cone, that is Γ Ω D = D.
In the spherical sector Ω D we would like to consider a nondegenerate positive radial solution u D := u Ω D of (3.4), hence we first recall conditions on the nonlinearity f which ensure that a positive radial solution of (3.4) in Ω D exists. Observe that such u D is just the restriction to Ω D of a positive radial solution of the Dirichlet problem Proposition 3.5. Let f : R → R be a locally Lipschitz continuous function. Assume that f satisfies one of the following: • There exist γ > 2, κ > 0 such that for |s| > κ it holds 0 < γF (s) < sf (s); Then a radial positive solution of (3.16) in B 1 , and hence of (3.4) in Ω D , exists.
Proof. In cases (i) and (ii), the corresponding functional is coercive and weakly lower semicontinuous in the space H 1 0,rad (B 1 ), which is the subspace of H 1 0 (B 1 ) of radial functions, and so it has a minimum which is a solution of (3.16). In the case (iii) standard variational methods, such as minimization on the Nehari manifold or Mountain Pass type theorems give a positive solution of (3.16), which is then radial by the Gidas-Ni-Nirenberg Theorem (see [12]). We refer to [3] and [7] for the details.
We point out that a radial solution u D is always a classical solution of (3.16) in B 1 , and hence in Ω D . In particular, u D is bounded and u D ∈ C 2 (B 1 ) Now we would like to study the nondegeneracy of a radial solution u D of (3.4) in Ω D .
As recalled in Section 2.1, we need conditions that ensure that zero is not an eigenvalue of the linearized operator Obviously, if the linearized operator L u D admits only positive eigenvalues, then u D is nondegenerate. This is the case of stable solutions of (3.4), which occur when f satisfies conditions (i) or (ii) in Proposition 3.5, in particular, if f is a constant.
In general, L u D could have negative eigenvalues, so to detect the nondegeneracy of u D we have to analyze the spectrum of the linear operator (3.17) in H 1 0 (Ω D ∪ Γ 1,0 ). As we will see, the fact that Ω D is a spherical sector in the cone Σ D (and not the ball B 1 ) plays a role.
The first remark is that zero is an eigenvalue for L u D if and only if it is an eigenvalue for the following singular problem: Therefore we investigate the eigenvalues of (3.18). The advantage of considering this singular eigenvalue problem is that, since u D is radial, its eigenfunctions can be obtained by separation of variables, using polar coordinates in R N . To this aim we denote by {λ j (D)} j∈N , the eigenvalues of the Laplace-Beltrami operator −∆ S N−1 on the domain D with Neumann boundary conditions. It is well-known that (3.19) and the only accumulation point is +∞. Then we consider the following singular eigenvalue problem in the interval (0, 1): It is shown in [2] (see also [5]) that nonpositive eigenvalues for (3.20) can be defined. They are a finite number and we denote them by ν i , i = 1, . . . , k. It is immediate to check that the eigenvalues ν i are the eigenvalues of (3.18) which correspond to radial eigenfunctions. In particular, we consider the first eigenvalue ν 1 of (3.20), referring to [2] for a variational definition and a study of its main properties. By using (3.18)-(3.20) we obtain the following result: We refer also to [9] for another approach, which consists in approximating the ball by annuli in order to avoid the singularity at 0.
From Proposition 3.6 we get the following sufficient condition for a radial solution u D to be nondegenerate. Proof. From Condition (I) we have which means that zero is not an eigenvalue of (3.18) with a corresponding radial eigenfunction. This, in turn, is equivalent to saying that zero is not a "radial" eigenvalue of the linearized operator (3.17), i.e., u D is a radial solution of (3.4) in Ω D (or of (3.16) in B 1 ) which is nondegenerate in the subspace H 1 0,rad (Ω D ∪ Γ 1,0 ), which is the subspace of H 1 0 (Ω D ∪ Γ 1,0 ) given by radial functions. Now, since λ 0 (D) = 0, λ 1 (D) > 0 and since ν 1 is the smallest eigenvalue of (3.20), then, from Condition (II) and (3.22) we infer that the sum (3.21) can never be zero. Hence, thanks to Proposition 3.6, we have that zero is not an eigenvalue of (3.18) and so cannot be an eigenvalue for the linearized operator (3.17) in the whole H 1 0 (Ω D ∪ Γ 1,0 ), i.e. u D is a non-degenerate solution to (3.4) in Ω D . It holds if f satisfies conditions (i) or (ii) of Proposition 3.5, because in this case all the eigenvalues of (3.17) and of (3.18) are positive. It then follows that (II) holds as well. More precisely, in the case (i), since 0 < a < µ 1 (B 1 ), the first eigenvalue of L u D is positive, so (3.23) In the case (ii), since f ′ (u D ) ≤ 0, it follows that ν 1 > 0. Among the nonlinearities satisfying condition (iii) of Proposition 3.5 we could consider f (u) = u p , 1 < p < N +2 N −2 , N ≥ 3. Then it is known that the positive radial solution of (3.16) is unique and nondegenerate (see [6,12]), so (I) holds. It is also well-known that for this nonlinearity it holds ν 1 < 0 and ν 1 is the only negative eigenvalue of (3.20), because u D can be obtained by the Mountain Pass Theorem or by minimization on the Nehari manifold and thus it has Morse index one. Then the validity of (II) depends on the cone, since it depends on λ 1 (D). However, once p is fixed, since ν 1 does not depend on the cone, it is obvious that, by varying D, there are many cones for which (II) holds. Moreover, it has been proved in [5] that ν 1 > −(N − 1) for every autonomous nonlinearity, so that whenever λ 1 (D) > N − 1 all radial solutions of (3.4) are nondegenerate.

3.3.
Stability of (Ω D , u D ). Let us first observe that if u D is a positive nondegenerate radial solution of (3.4) for ϕ = 0, belonging to W 1,∞ (Ω D ) ∩ W 2,2 (Ω D ), then (Ω D , u D ) is energy-stationary in the sense of Definition 2.4. Indeed, since u D is radial, we have that ∂u D ∂ν = constant on Γ 0 = D and thanks to Proposition 2.5 we easily conclude.
To investigate the stability of (Ω D , u D ) we analyze the quadratic form corresponding to the second derivative I ′′ (ϕ) at ϕ = 0. Fixing the constant c in the definition of M (see (3.12)) as c = |Ω D |, we have that the tangent space to M at ϕ = 0 is given by (3.24) Writing u D (r) = u D (|x|), we denote by u ′ D and u ′′ D the derivatives of u D with respect to r, so that By Hopf's Lemma we know that u ′ D (1) < 0 and actually u ′ where µ D denotes the Lagrange multiplier in the case ϕ = 0, see (3.13).
For v ∈ T 0 M , we will denote by u v the solution of Let us remark that for every q ∈ D the outer unit normal vector ν(q) is precisely q, hence (3.27) corresponds to (2.16) in Ω D .
Note that, since u D is a nondegenerate radial solution, then the weak solution u v of (3.27) is unique for every v.
Our next result shows that the quadratic form corresponding to the second derivative of I at ϕ = 0 has a simple expression.

28)
where u v is the solution of (3.27).
Proof. From Lemma 3.2, (3.11) and Lemma 3.4, with w = v, by simple substitutions and elementary computations we obtain:  To investigate the stability of (Ω D , u D ) as an energy stationary pair for I we need to study the solution u v of (3.27), for any v ∈ T 0 M (that is, for functions with mean value zero on D). As we will see, it will be enough to consider only functions v which are eigenfunctions of the Laplace-Beltrami operator −∆ S N−1 with Neumann boundary conditions on D. Hence we consider the eigenvalue problem    −∆ψ = λψ on D ∂ψ ∂ν = 0 on ∂D (3.32) and denote its eigenvalues as in (3.19), counted with multiplicity: 0 = λ 0 (D) < λ 1 (D) ≤ λ 2 (D) ≤ . . .. The corresponding L 2 -normalized eigenfunctions are denoted by {ψ j } j∈N , with D ψ 2 j dσ = 1, ψ 0 = constant and D ψ j dσ = 0 for j ≥ 1. Theorem 3.10. Let j ≥ 1 and u j be the unique solution of (3.27) for v = ψ j . Then, writing u j = u j (r, q), the function Proof. Since the proof is the same for all j, we drop the index and the dependence on D and write simply h, ψ and λ.
It is immediate to check that h(1) = −u ′ D (1). Moreover, since we can bring the radial derivative inside the integral on D, for every r ∈ (0, 1] we have: On the other hand, applying Green's formula, taking into account the Neumann conditions on ψ and u, we infer that Remark 3.11. Note that with u j and h j as in Theorem 4.8 we have that Indeed, the boundary conditions are clearly satisfied by this function, and it holds Moreover, h j ∈ L ∞ (0, ∞) and h j (0) = 0.
Proof. Again, for simplicity, we drop the index j. Since u ∈ H 1 (Ω D ) (see Sect. 2), writing u = u(r, q) and recalling that ψ is a L 2 (D)-normalized solution to (3.32), we get that which proves (3.38) and (3.39). Once we have these estimates, we can proceed as in [8,Lemma A.9] to get the boundness of h and h(0) = 0. (3.20). Then for the solution h j of (3.34) it holds that h j > 0 in (0, 1).
Proof. Let z 1 be an L 2 -normalized first eigenfunction of (3.20). From [2, Section 3.1] we know that z 1 does not change sign. Writing the equations satisfied by h j and z 1 in Sturm-Liouville form we have: By Proposition 3.12 we know that h j (0) = 0 and h j (1) = −u ′ D (1) > 0. Now, assume by contradiction that h j changes sign in (0, 1). Then there would exist r 0 ∈ (0, 1) such that h j (0) = 0. Since − ν 1 < λ j (D), then, by the Sturm-Picone Comparison Theorem it would follow that z 1 has a zero in (0, r 0 ). This is a contradiction, because z 1 does not change sign. Hence the only possibility is that h j is strictly positive in (0, 1).
We are ready to prove our main result for problem (1.1) in the case of the cone, i.e., Theorem 1.1, which is a sharp instability/stability result for the pair (Ω D , u D ).
Proof of Theorem 1.1. Let us fix the domain D which spans the cone, so that we denote λ 1 (D) simply by λ 1 .
For (i), let u 1 = h 1 ψ 1 be the solution of (3.27) with v = ψ 1 . Then Putting (3.34) in Sturm-Liouville form we get On the other hand, writing −∆u D = f (u D ) in polar coordinates and differentiating with respect to r = |x| we get Multiplying (3.41) by u ′ D and integrating by parts in (r, 1) we get that Similarly, multiplying (3.42) by h 1 and integrating by parts we deduce that Notice that, in view of Proposition 3.12, the right-hand sides of (3.43), (3.44) remain finite when taking the limit asr → 0 + . In addition, we claim that Indeed, integrating (3.41) and taking the absolute value we obtain for some C 2 > 0, and thus, since limr →0 + u ′ D (r) = 0, (3.45) follows. Now, subtracting (3.44) from (3.43) and taking the limit asr → 0 + , then, thanks to (3.45) and since h 1 (0) = 0, For (ii), we choose an orthonormal basis (ψ j ) j of L 2 (D) made of normalized eigenfunctions of (3.32). Then any v ∈ T 0 M can be written as where (·, ·) denotes the inner product in L 2 (D). We assume without loss of generality that D v 2 dσ = 1. Let u j be the solution of (3.27) with v = ψ j , then we can check that is the solution of (3.27). As observed in Remark 3.11, u j (r, q) = h j (r)ψ j (q) for every By an argument analogous to the one presented in the proof of (i), we have that if Indeed, writing the equations for h j , h k , multiplying the first one by h k and the second one by h j , integrating by parts and subtracting we get Exploiting the orthogonality of the basis (ψ j ) j and exploiting (3.47) we obtain because h 1 > 0 in (0, 1), u ′ D < 0 in (0, 1) and λ 1 > N − 1 by assumption. The proof is complete. Remark 3.14. As already pointed out in Remark 2.6, in the case when C = R N , the couples (B, u B ), where B is a ball and u B is a positive nondegenerate radial solution, are the only energy-stationary pairs. Thus it remains to study the stability of (B, u B ) as critical point of the energy functional T . This can be done by looking at the problem as the case of a cone spanned by the domain D = S N −1 .
As observed in Remark 1.2, the first eigenvalue ν 1 of the singular eigenvalue problem (3.20) is always larger than −(N − 1). On the other hand, it is known that the first nontrivial eigenvalue of the Laplace-Beltrami operator on the whole S N −1 is precisely N − 1. Then any radial solution u B is nondegenerate and we obtain that the pair (B, u B ) is a semistable stationary-point.
In analogy with the case of the cone, we consider domains whose relative boundaries are the cartesian graphs of functions in C 2 (ω). More precisely, for ϕ ∈ C 2 (ω) we set and consider domains of the type Observe that the outer unit normal vector on Γ ϕ at a point (x ′ , e ϕ(x ′ ) ) is given by where ∇ R N−1 denotes the gradient with respect to the variables x 1 , . . . , x N −1 .

Energy functional in cylindrical domains.
We study the semilinear elliptic problem and consider bounded positive weak solutions of (4.2) in the Sobolev space H 1 0 (Ω ϕ ∪ Γ 1,ϕ ), which is the space of functions in H 1 (Ω ϕ ) whose trace vanishes on Γ ϕ .
As before, we assume that a bounded nondegenerate positive solution u ϕ of (4.2) exists and belongs to W 1,∞ (Ω ϕ ) ∩ W 2,2 (Ω ϕ ), so that we can apply the results of Section 2.
We consider variations of the domain Ω ϕ in the class of cartesian graphs of the type Ω ϕ+tv , for v ∈ C 2 (ω), which amounts to consider a one-parameter family of diffeomorphisms ξ : (−η, η) × Σ ω → Σ ω of the type ξ(t, x) = (x ′ , e tv(x ′ ) x N ), whose inverse, for any fixed t ∈ (−η, η), is given by This one-parameter family of diffeomorphisms is generated by the vector field where 0 ′ := (0, . . . , 0) ∈ R N −1 . Indeed, ξ(0, x) = x for every x ∈ Σ ω , and ξ(t, x) ∈ ∂Σ ω , for all (t, x) ∈ (−η, η) × ∂Σ ω . We also observe that, in view of (4.1), it holds The energy functional T defined in (2.10) becomes a functional depending only on functions in C 2 (ω). More precisely, for every v ∈ C 2 (ω), in view of Proposition 2.1, there exists δ > 0 sufficiently small such that for all t ∈ (−δ, δ) is well defined, where u ϕ+tv := u Ω ϕ+tv is the unique positive solution of (4.2) in the domain Ω ϕ+tv , in a neighborhood of u ϕ • ξ −1 t . By the results of Section 2 we know that the map t → u ϕ+tv is differentiable at t = 0, and the derivative u is a weak solution of We now compute the first derivative of T at Ω ϕ , i.e., for t = 0, with respect to variations v ∈ C 2 (ω).
Lemma 4.1. Let ϕ ∈ C 2 (ω) and assume that u ϕ is a positive nondegenerate solution of (4.2) which belongs to W 1,∞ (Ω) ∩ W 2,2 (Ω). Then, for any v ∈ C 2 (ω) we have Proof. The proof is similar to that of Lemma 3.1. It suffices to observe that for the parametrization of Γ ϕ given by x = (x ′ , e ϕ(x ′ ) ), for x ′ ∈ ω, the induced (N −1)-dimensional area element on Γ ϕ is expressed by Then the result follows immediately from Proposition 2.2, taking into account (4.4).
Lemma 4.2. Let ϕ and u ϕ be as in Lemma 4.1. Then for any v, w ∈ C 2 (ω) it holds

7)
where u w is the solution of (4.5), with w in the place of v.
Proof. Let v, w ∈ C 2 (ω). By definition, Lemma 4.1 and using the Leibniz rule, we have: To conclude it suffices to compute the derivative in the first integral of the right-hand side of (4.8). To this end we observe that where ν ϕ is given by (4.1) and Now, for the first term in the right-hand side of (4.9), thanks to the argument presented in [ (∇u ϕ+sw (x ′ , e ϕ+sw )) = ∇ u w (x ′ , e ϕ ) + D 2 u ϕ (x ′ , e ϕ )(0 ′ , we ϕ ). (4.10) On the other hand, for the last term in (4.9), we check that Finally, substituting (4.9)-(4.11) into (4.8) we obtain (4.7).
As in Section 3, in view of Definition 2.4, we consider a volume constraint. In the case of cartesian graphs, the volume of the domain Ω ϕ associated to ϕ ∈ C 2 (ω) is expressed by (4.12) The functional V is of class C 2 and for every v, w ∈ C 2 (ω) it holds For c > 0 we define the manifold M := ϕ ∈ C 2 (ω) : ω e ϕ dx ′ = c , whose tangent space at any point ϕ ∈ M is given by (4.14) We consider the restricted functional As before, if ϕ ∈ M is a critical point for I, then there exists a Lagrange multiplier µ ∈ R such that T ′ (ϕ) = µI ′ (ϕ). Results analogous to Proposition 3.3 and Lemma 3.4 hold with the same proofs. In particular, we point out that for an energy stationary pair (Ω ϕ , u ϕ ) under a volume constraint the function u ϕ has constant normal derivative on Γ ϕ . For the reader's convenience, we restate here these results. Proposition 4.3. Let ϕ ∈ M and let (Ω ϕ , u ϕ ) be energy-stationary under a volume constraint. Then the Lagrange multiplier µ is negative and Proof. The same as in [14,Lemma 4.1] For the second derivative of I we have Proof. The same as in [14,Lemma 4.3] 4.2. The case ϕ ≡ 0 and one-dimensional solutions. When ϕ ≡ 0 (that is, Γ ϕ = Γ 0 is the intersection of the cylinder with the plane x N = 1), the domain Ω 0 is just the finite cylinder Ω ω := ω × (0, 1).
Then, if f is a locally Lipschitz continuous function, any weak solution of (4.2) is also a classical solution up to the boundary, i.e., it belongs to C 2 (Ω ω ). This follows by standard regularity theory by considering the boundary conditions and that ∂Ω ω is made by the union of three (N − 1)-dimensional manifolds (with boundary) intersecting orthogonally (see also [17,Proposition 6.1]).
In Ω ω , for suitable nonlinearities, we can find a solution of (4.2) in Ω ω which depends only on x N in the following way: first, we can apply some variational method to find a solution u of the ordinary differential equation and then set Recall that, in one dimension, there is no critical Sobolev exponent for the embedding into L p . So one example of a suitable nonlinearity is f (u) = u p with 1 < p < ∞, or those of Proposition 3.5 with the only caution that in (iii), for N ≥ 2 we can take 1 < p < ∞. For our purposes we need to consider one-dimensional solutions u ω of (4.2) in Ω ω that are nondegenerate, which means that the linearized operator does not admit zero as an eigenvalue. In other words, u ω is nondegenerate if there are no nontrivial weak To analyze the spectrum of L uω it is convenient to consider the following auxiliary one-dimensional eigenvalue problem: We denote the eigenvalues of (4.18) by α i , for i ∈ N. Clearly, they correspond to the eigenvalues of the linear operator with the boundary conditions of (4.18). We also consider the following Neumann eigenvalue problem in the domain ω ⊂ R N −1 : is the Laplacian in R N −1 , i.e. with respect to the variables x 1 , . . . , x N −1 . We denote its eigenvalues by It is well-known that λ j (ω) ր +∞ as j → ∞ and that the normalized eigenfunctions form a basis (ψ j ) j of the tangent space T 0 M defined in (4.14) when ϕ ≡ 0. Proof. We begin by showing that σ(L uω ) ⊂ σ( L uω ) + σ(−∆ R N−1 ). Let τ ∈ σ(L uω ) and let φ ∈ H 1 0 (Ω ω ∪ Γ 1,0 ) be an associated eigenfunction, that is, φ is a weak solution of As observed at the beginning of this subsection for the the nonlinear problem (4.2), by the shape of Ω ω and the boundary conditions, since f ∈ C 1,α (R), by standard elliptic regularity, we have that φ is a classical solution of (4.23) in Ω ω .
Let λ be an eigenvalue of −∆ R N−1 with homogeneous Neumann boundary condition on ω and let ψ be an associated eigenfunction. Define (4.24) Then, differentiating with respect to x N , using Green's formulas and the boundary conditions we have To show the reverse inclusion, let α ∈ σ( L uω ), λ ∈ σ(−∆ R N−1 ) and let z, ψ be, respectively, the associated eigenfunctions. Setting for Finally, by construction, we easily check that φ satisfies the boundary conditions of (4.23). As a consequence, we deduce that α + λ ∈ σ(L uω ) and this concludes the proof.  (ii) λ 1 (ω) > −α 1 .
Proof. Analogous to the proof of Corollary 3.7.

4.3.
Stability/instability of the pair (Ω ω , u ω ). In this subsection, we prove a general stability/instability theorem for the pair (Ω ω , u ω ). We begin with some preliminary results. Firstly, we recall that when ϕ ≡ 0 the tangent space T 0 M is given by Since u ω depends on x N only, in order to simplify the notations, we denote with a prime the derivative with respect to x N , and thus we write Then, for v ∈ T 0 M , we have that the function u (see (4.5)), which belongs to As before, by elliptic regularity we know that u is regular in Ω ω , and thus it is a classical solution. We also note that, by the nondegeneracy of u ω , there exists a unique solution of (4.27).
Lemma 4.8. Let λ j > 0 be any positive eigenvalue for the Neumann problem (4.20) and let ψ j be any normalized eigenfunction associated to λ j . Let u j ∈ H 1 (Ω ω ) be the solution of (4.27) with v = ψ j . Then the function Proof. For simplicity of notation we drop the index j and simply writeũ, h, ψ and λ instead ofũ j , h j , ψ j and λ j . First observe that, asũ = −u ′ ω (1)ψ on Γ 0 , we have Now, differentiating with respect to x N under the integral sign and using Green's formula, taking into account the boundary conditions, we have Finally, exploiting the Neumann condition for u on Γ 1,0 , we check that h ′ (0) = 0.
Remark 4.9. Note that for u j , h j as in Lemma 4.8 we have that Indeed: Moreover, by (4.29) and (4.20), the function h j ψ j satisfies the boundary conditions in (4.27), so that h j ψ j is the unique solution of (4.27) and thus coincides with u j . Proposition 4.10. Let j ≥ 1, λ j be a positive Neumann eigenvalue of −∆ R N−1 in ω, and let h j be the solution of (4.29). Assume that −α 1 < λ j , where α 1 is the smallest eigenvalue of (4.18). Then it holds that Proof. We can reflect h j by evenness with respect to 0 to have a solution of the linear with the boundary condition z(−1) = z(1) = 0 is exactly α 1 . Therefore the first eigenvalue of the linear operator L uω g = −g ′′ − f ′ (u ω )g + λ j g with zero boundary condition in (−1, 1) is It is well-known that L uω satisfies the maximum principle whenever β 1 > 0, i.e., when λ j > −α 1 . Therefore, by (4.30), the function h j satisfies h j ≥ 0 in (−1, 1), and by the strong maximum principle we conclude that h j > 0 in (−1, 1).
We can now state and prove the main result of this section.
Proof. We first observe that since ∂uω ∂ν is constant on Γ 0 then, by the analogous of Proposition 3.3 for cylinders, we infer that the pair (Ω ω , u ω ) is an energy-stationary pair.
Let w ∈ T 0 M and assume without loss of generality that ω w 2 dx ′ = 1. In order to prove (i)-(ii) we first determine a suitable expression for I ′′ (0)[w, w]. To this end, for each j ∈ N + , let u j be the solution of (4.27) with v = ψ j and let h j be the solution of (4.29). Then we can write where (·, ·) is the inner product in L 2 (ω). Moreover, we can check that is the solution of (4.27) corresponding to w. Then, taking ϕ = 0 in Lemma 4.2, exploiting Lemma 4.4, taking into account that by Proposition 4.3 the Lagrange multiplier µ is given by 1)) 2 , by Remark 4.9 and observing that ∇u ω ⊥ (∇ R N−1 w, 0), we infer that Finally, since u ω is a solution to (4.16) we deduce that In particular, choosing w = ψ 1 and plugging it into (4.32) we infer that Multiplying the equation in (4.29) (with j = 1) by u ′ ω and integrating by parts we get Exploiting (4.16), integrating by parts and taking into account that h 1 (1) = −u ′ ω (1) we obtain Hence, we deduce that In the end, from (4.33), (4.35) and recalling (4.31), we obtain Therefore, if ρ < 0 then I ′′ (0)[ψ 1 , ψ 1 ] < 0, i.e., (Ω ω , u ω ) is an unstable energy-stationary pair, and this proves (i).
Let us prove (ii). Let w ∈ T 0 M such that ω w 2 dx ′ = 1. From (4.32) we know that . Thanks to the assumption λ 1 > −α 1 the following holds true.
We conclude this section with the proof of Theorem 1.4.
Remark 4.13. In the case of the Lane-Emden nonlinearity f (u) = u p , at least for some integer values of p, it is possible to compute the solution u ω numerically, as well as the eigenvalue α 1 and the function h 1 for different values of λ 1 (ω). This allows to compute ρ numerically, so that, plotting the result for ρ as a function of λ 1 (ω), we obtain a region of instability for λ 1 (ω) close to −α 1 .