Foliated Schwarz symmetry of solutions to a cooperative system of equations involving nonlocal operators

In this paper, we prove foliated Schwarz symmetry of solutions to a cooperatively coupled system of equations involving nonlocal operators. Here, the class of nonlocal operators covers in particular the case of the fractional Laplacian. Moreover, we give an explicit example of a nonlocal nonlinear system, in which our result can be applied.


Introduction
In the following, we investigate symmetry properties of solutions of a system of equations in radial domains. More precisely, for m ∈ N we investigate bounded continuous solutions of where Ω ⊂ R N is a bounded radial domain. Moreover, f 1 , . . . , f m ∈ C 1 ([0, ∞) × R m ) are nonlinearities to be specified later and I is a nonlocal operator, which for u ∈ C 2 (R N ) ∩ L ∞ (R N ) is given by Here k : R N → [0, ∞] is a kernel function, which is given by k(z) = k 0 (|z|), z ∈ R N for a monotone decreasing function k 0 : [0, ∞) → [0, ∞] satisfying ∞ 0 min{1, r 2 } k 0 (r) r N−1 dr < ∞ and ∞ 0 k 0 (r) r N−1 dr = ∞.
We note that these assumption on k in particular cover the case, where I = (−∆) s , s ∈ (0, 1) by setting k 0 (r) = c N,s r −N−2s for a normalization constant c N,s > 0 given by The value of c N,s is chosen to make the fractional Laplacian the pseudodifferential operator whose symbol is |ξ | 2s (see e.g. [4,Section 3.1] or [8,Proposition 3.3] for details; the equality of the two values given in (4) is shown in [12,16], see also [14]). For further information on the operator I and the definition of weak solution, which we use in this paper, we refer to Subsection 2.1 below, see also [13,19].
Symmetry properties of solutions to nonlocal nonlinear problems have been studied for one or more equations in the case where I is the fractional Laplacian in [10,11,14,18], while in [19] the question of symmetries to solutions was studied for a general class of nonlocal operators. However, if Ω is not a ball but rather an annulus or of the solutions change sign, then in general it is no longer true that a solution of (1) must be radial even in the case where I is a local operator and m = 1. However, under some suitable assumptions on the equation or the system, some axial symmetry can still be achieved. In the case where m = 1 and I is a local operator this has been studied in [24,25,30], whereas symmetry for systems have been studied in [5,6] (see also there references in there). For the nonlocal case, the axial symmetry of solutions has been studied in [17] for m = 1.
In the following, we are interested in solutions of (1), which are not radial and possibly change sign. For this we consider a particular kind of axial symmetry called foliated Schwarz symmetry, which was defined in [29,Definition 2.4], based on an idea in [26]. We also refer to the general survey -in particular Section 2.3 -in [33].
Let Ω ⊂ R N , N ≥ 2 be a radial domain, p ∈ S N−1 := {x ∈ R N : |x| = 1}. A function u : Ω → R is called foliated Schwarz symmetric with respect to p in Ω, if for every r > 0 with re 1 ∈ D and c ∈ R, the restricted superlevel set {x ∈ r S N−1 : u(x) ≥ c} is equal to r S N−1 or a geodesic ball in the sphere r S N−1 centered at rp. We simply call u foliated Schwarz symmetric, if u has this property for some unit vector p ∈ R N .
We give an equivalent definition in Section 2.2.1 (see also [30,Proposition 3.3]) below, which we use in our proof. Note that if u : R N → R is such that u| Ω is foliated Schwarz symmetric with respect to some p for some radial set Ω ⊂ R N , then u χ Ω is axially symmetric with respect to the axis R · p and nonincreasing in the polar angle θ = arccos( x |x| · p). Our main result on the symmetry properties of solutions of (1) is the following. Theorem 1.1. Let Ω ⊂ R N be a bounded radial domain and m ∈ N. Assume that f i ∈ C 1 ([0, ∞) × R m , R m ), (r, u 1 , . . . , u m ) → f i (r, u 1 , . . . , u m ), i = 1, . . . , m satisfies Let u 1 , . . . , u m : R N → R be continuous bounded functions satisfying (1) in weak sense. If for all x 1 > 0, x ∈ R N−1 , i = 1, . . . , m, and then there is p ∈ S N−1 such that u 1 , . . . , u m are foliated Schwarz symmetric in Ω with respect to p and strictly decreasing in the polar angle.
Clearly, assumption (5) is restricting, which is due to the fact that we did not assume any further connection between the f i and u j . In the following variant of Theorem 1.1 we weaken the assumption on f i but assume positivity of the u i .
Let u 1 , . . . , u m : R N → R be continuous bounded functions satisfying (1) in weak sense. If u i > 0 in Ω for i = 1, . . . , m and (6) holds, then there is p ∈ S N−1 such that u 1 , . . . , u m are foliated Schwarz symmetric in Ω with respect to p and strictly decreasing in the polar angle.
It is not obvious under which circumstances a solution u 1 , . . . , u m can be found such that (6) is satisfied. In the following, we give an explicit example, covering the case of the fractional Laplacian, where Theorem 1.2 can be applied. For this, we recall the first eigenvalue of the operator I in an open subset Ω of R N given by Recall from [21] that λ 1 (Ω) > 0 if Ω is bounded in one direction. The following existence statement is related to the Brézis-Nirenberg problem, which for systems with the fractional Laplacian has been studied in [9]. In the following statement, we consider a more general class of nonlocal operators, which includes the fractional Laplacian and deals with the geometry of the pair of solutions to the system. Theorem 1.4. Let the function k 0 in (3) be strictly decreasing and, for some 0 < s ≤ σ < 1, γ ∈ (0, 1), and c > 0, satisfy Furthermore, let Ω ⊂ R N be a bounded radial domain, a 1 , a 2 ∈ L ∞ (R) with a + i L ∞ (R) < λ 1 (Ω) for i = 1, 2, and let 1 < q < N N−2s . If a 1 = a 2 , then there are two continuous bounded functions u 1 , u 2 : R N → R, u 1 = u 2 , which are positive in Ω and satisfy in weak sense Moreover, u 1 and u 2 are foliated Schwarz symmetric with respect to some p ∈ S N−1 and, if u 1 and u 2 are not radial, then they are strictly decreasing in the polar angle.
The paper is organized as follows. In Section 2 we present our notation and recall known statements on the nonlocal operators we use. Moreover, we introduce the notation for systems and recall the properties and definitions of Foliated Schwarz symmetry. In Section 3 we state and prove variants of maximum principles, which we use in Section 4 to prove Theorem 1.1. The proof of Theorem 1.2 can be found in Section 5.

Notation and Preliminaries
In the following we use N ∈ N to denote the dimension. For A, B ⊂ R N nonempty measurable sets we denote by χ A : R N → R the characteristic function and |A| the Lebesgue measure. The notation B ⊂⊂ A means that B is compact and contained in the interior of A. We denote dist(A, B) := inf a∈A, b∈B |a − b| and as usual dist({x}, A) := dist(x, A) for x ∈ R N . For r > 0 we denote B r (A) := {x ∈ R N : dist(x, A) < ∞} and then B r (x) denotes the ball of radius r for x ∈ R N . Moreover, we fix S N−1 := ∂ B 1 (0) = {x ∈ R N : |x| = 1} to denote the N-dimensional sphere. As usual, for A open, C m (A) (resp. C m (A)) denotes the space of m-times continuously differentiable functions in A (resp. A) and C 0,1 (A) denotes the space of Lipschitz functions. C m c (A) and C 0,1 c (A) denotes respectively those functions in C m (A) or C 0,1 (A), which have compact support in A. In the following, if X(A) is some function space and u ∈ X(A) is a function, we always mean that u : Finally, for a function u : A → R we use u + := u + := max{u, 0} and u − := − min{u, 0} to denote the positive and negative part of u respectively, so that u = u + − u − .

On the operator and associated spaces
Let k : R N → [0, ∞) be a radial and radial decreasing function. That is k(z) = k 0 (|z|), z ∈ R N for a monotone decreasing function k 0 : [0, ∞) → [0, ∞] satisfying (3). We denote formally the bilinear form associated to k by For Ω ⊂ R N open, this bilinear form is well-defined on It follows that D k (Ω) is a Hilbert space with scalar product where ·, · 2 denotes the usual L 2 scalar product. By standard methods (see e.g. [19,20]) it follows that E k is associated to a (nonlocal) operator I, which on C 2 (R N ) ∩ L ∞ (R N ) is represented by (2) and it holds We note that the embedding D k (Ω) → L 2 (Ω) is locally compact in the sense that and moreover (see [ It hence follows that in this case E k is a scalar product and the induced norm is equivalent to ·, · 2 . In particular, if Ω is bounded, then D k (Ω) → L 2 (Ω) is compact and λ 1 (Ω) corresponds to the first eigenvalue of I.
In the following, we understand solutions in the weak sense, that is, given f ∈ L 2 (R N ), we say that u ∈ D k (Ω) is a solution of if for all ϕ ∈ D k (Ω) we have In particular, u 1 , u 2 are called weak solution of (1), if for i = 1, 2, we have u i ∈ D k (Ω) and for all ϕ ∈ D k (Ω), whenever the right-hand side is well defined.
Finally, in our analysis, we use the rotating plane method and linearize the system of equations. Our symmetry results then follow from an application of different maximum principles for supersolutions. For this, we extend the definition of D k (Ω). Let Ω ⊂ R N open and denote

Clearly by definition we have for
The following Lemma collects all information on V k (Ω) needed in this paper. 1. E k is well-defined on V k (Ω) × D k (Ω) and Moreover, if k 0 is strictly decreasing, then equality holds in these inequalities if and only if u = u + or u = u − a. e. in R N Furthermore, if Ω is in addition bounded and u ∈ V k (Ω), then 4. u χ R N \Ω ≡ 0, then u ∈ D k (Ω).
The additional assertion in Lemma 2.1.3 follows immediately from the proof in [19]. Based on Lemma 2.1 we say u ∈ V k (Ω) satisfies for some f ∈ L 2 (Ω) in weak sense if u ≥ 0 on R N \ Ω and for all v ∈ D k (Ω), v ≥ 0 we have We also call u in this case a supersolution of (13). Similarly, we call u a subsolution of (13) if −u satisfies in weak sense (14).

Notation for the reflection of a hyperplane
Proof. 1. follows immediately from the definition of the function space since k(z e ) = k(z) for all z ∈ R N , e ∈ S N−1 . 2. and 3. follow from [17, Lemma 3.2] with (11) noting that we have Let e ∈ S N−1 and W := W e := U −U e . Then W e ∈ D k (Ω) is a solution of the linear problem which satisfies in addition W = −W e . Here, C(x) = (c i j (x)) 1≤i, j≤m where c i j ∈ L ∞ (Ω), i, j = 1, . . . , m is given by Proof. Let e ∈ S N−1 , W = (w 1 , . . . , w m ) as in the statement and fix i ∈ {1, . . . , m}. Then clearly W = −W e and by Lemma 2.2 we have W ∈ D k (Ω) and we have in weak sense in Ω where we have used the mean value theorem.

Foliated Schwarz symmetry
Denote by H the set of open half spaces in R N . Give u : Assume that there is e 0 ∈ M such that the following is true: For all two dimensional subspaces V ⊂ R N with e 0 ∈ V there are e + , e − ∈ M ∩ V , e + = e − , which are in the same connected component of M ∩V and satisfy u = u e + and u = u e − for every u ∈ P. Then there is p ∈ S N−1 such that for every connected component D on Ω the functions u χ D for u ∈ P are foliated Schwarz symmetric with respect to p.

Proposition 2.4 is essential in our proofs and we apply it to the family
The assumption of the proposition is verified with the rotating plane method based on the notation of Subsection 2.2. We note that the polarization of a function in D k (Ω) remains in D k (Ω) -we include a statement of this fact for the reader's convenience in Lemma 5.7 below.

Linear problems of systems and the maximum principle
In the following we collect maximum principles needed for our proofs for linear systems of equations. Here, the problems are stated in a half space and the definition of supersolution is adjusted to the oddness of the solution with respect to a hyperplane as presented in Lemma 17. In the following, let as above H be the set of half spaces in R N and fix H ∈ H and D ⊂ H, an open bounded set. We denote the reflection at ∂ H by σ H . Moreover, let c i j ∈ L ∞ (D), i, j = 1, . . . , m be given and denote C(x) := (c i j (x)) 1≤i, j≤m . The following maximum principles are for functions We also say, that U satisfies in weak sense We call the linear system (19) weakly coupled (in D), if Moreover, we call the linear system (19) fully coupled, if it is weakly coupled and for all i, j there is a compact set K ⊂ D with |K| > 0 and essinf K c i j > 0. Proof. For m = 1 see [19,Proposition 3.5]. The general case follows similarly. Indeed, let c ∞ > 0 be given and by (12) we may fix δ > 0 such that as a suitable test function and we have with Lemma 2.2 and the weak coupling assumption Hence Proposition 3.2 (Strong maximum principle for systems). Let H ∈ H , D ⊂ H be a domain, and let c i j ∈ L ∞ (D), i, j = 1, . . . , m be strongly coupled. Then for any function U ∈ V k (D) satisfying in weak sense (19) Proof. For m = 1 see [19,Proposition 3.6]. For m ∈ N arbitrary, we first note that for any i = 1, . . . , m we have in weak sense Since there is a compact set K ⊂ D with |K| > 0, essinf K c i j > 0, and also essinf K u i > 0, and (19). Clearly, this however is a contradiction and hence u j ≡ 0 in D is impossible. Thus u j > 0 in D by [19,Proposition 3.6] and since j was arbitrary the statement of the Proposition follows.
Remark 3.3. We note that the connectedness of D in Proposition 3.2 is not needed, if k 0 is strictly decreasing and hence k > 0 in R N .
Remark 3.4. We emphasize that the conclusion of Proposition 3.1 and 3.2 also follow if U ∈ V k (D) satisfies IU ≥ C(x)U in D and U ≥ 0 on R N \ D. The proof in this case is similar, but simpler.

Proof of the symmetry result
Using the notation of the previous sections, Theorem 1.1 can be stated equivalently as Let U ∈ D k (Ω) be a bounded continuous solution of (16), and assume that there then there is p ∈ S N−1 such that U is foliated Schwarz symmetric with respect to p and strictly decreasing in the polar angle.
Proof. Denote W e := U −U e for e ∈ S N−1 and note that W satisfies in weak sense where By assumption (21) it follows that the system (24) is weakly coupled and, moreover, if (22) holds or if U > 0 in Ω and (23) holds, then the system is also strongly coupled.
Step 1: We claim that Note that W e 0 0 in Ω e 0 and for Hence it by Proposition 3.2 that W e 0 > 0 (since W e 0 ≡ 0 is impossible by assumption), that is (25) holds.
Step 2: Next, by continuity of U and e → σ e , there is for any δ > 0 an ε > 0 such that for any We claim that there is ε > 0 such that To see (26), we use Proposition 3.
Let ε > 0 be given by the above remark and fix e ∈ S N−1 with |e − e 0 | < ε and K ⊂ Ω e ∩ Ω e 0 with W e 0 in K. Finally, let A := Ω e \ K. As before, let W e = (w 1 , . . . , w m ) and note that now Since by the assumptions on F (and U) this system is weakly coupled, Proposition 3.1 implies W e ≥ 0 in H e . Hence (26) holds.
Note that then there must be i ∈ {1, . . . , m} such that w e + ,i ≡ 0 in Ω e + , because otherwise a similar argumentation as in Step 1 and Step 2 allows to continue rotating the hyperplanes which is a contradiction to the definition of e + . Similarly, there must be i ∈ {1, . . . , m} such that w e − ,i ≡ 0 in Ω e − . Let i ∈ {1, . . . , m} such that w e + ,i ≡ 0 in Ω e + in Ω e + . And assume there is j ∈ {1, . . . , m} such that w e + , j ≡ 0 in Ω e + . Then as in Step 1 it follows that this is impossible.
Thus W e + ≡ 0 and similarly, also W e − ≡ 0. Hence (27) holds. (27) it follows that there is p ∈ S N−1 such that U is foliated Schwarz symmetry, since e + and e − are clearly in the same two dimensional component of e's in which U ≥ U • σ e and e + = e − . The fact that U is strictly decreasing in the polar angle now follows, from Step 2 and with Proposition 3.2, we actually have that W e ϕ > 0 in Ω e ϕ for ϕ ∈ (ϕ − , ϕ + ). This finishes the proof.
Hence, the system (28) is weakly coupled as long as the product u 1 u 2 is non-negative in Ω. A similar system is considered in [23] (see also [5, (4.1)]) with a local operator in place of I, and with the bounded set Ω replaced by the whole space R N . An existence proof of a pair of nonnegative, radially symmetric solutions u 1 , u 2 ≥ 0 satisfying u 1 + u 2 ≡ 0 in R N is given there.
In the present paper, to keep the argument as transparent as possible, the nonlinearities in (28) are simpler than those in [23]. However, the parameter ω occurring there is replaced by the function a 2 (x). In order to prove Theorem 1.4, we begin with an existence statement.
The existence proof is based on the mountain-pass theorem (see, for instance, [1,Theorem 8.2] or [31, Chapter III, Theorem 6.1, p. 109]). More precisely, we consider the functional where we have used the notation , for shortness. Note that it is easy to see that D k (Ω) is continuously embedded into H s 0 (Ω) = {u ∈ H s (R N ) : u = 0 on R N \ Ω} and hence by the assumption q < N N−2s , it follows by the Sobolev embedding that D k (Ω) is compactly embedded into L 2q (Ω) (see [8,Theorem 6.7]). Hence the product uv belongs to L q (Ω), and the functional J(u, v) is well defined on D k (Ω). The differential J at (u, v) is the linear operator L given by where (ϕ, ψ) ranges in D k (Ω). Hence the critical points of J are the weak solutions of (28). To apply the mountain pass theorem, we collect in the next section several properties of the Nehari Manifold N . For general applications of the mountain pass theorem to nonlocal operators, see also [27,28].

The Nehari manifold
In the sequel we refer to the Nehari manifold N associated to the functional J. First define the functional G(u, v) = 1 2 (u, v) 2 − uv q L q (Ω) , and then let where the last equality is readily obtained by letting (ϕ, ψ) = (u, v) in (32). In order to prove the existence of polarized solutions of system (28), we need Lemma 5.2.
1. The Nehari manifold N is a C 1 -manifold of codimension one in D k (Ω).
2. If (u, v) belongs to N , then the direction of (u, v) is non-tangential to N .

The manifold N keeps far from the origin in the sense that there exists r
Proof. Choose a point (u 0 , v 0 ) ∈ N + , and observe that the product u 0 v 0 cannot vanish identically (that would be in contrast with (33)). In a neighborhood of (u 0 , v 0 ), the Nehari manifold is the set of zeros of the functional G(u, v), whose differential is the linear functional G (u, v) given by To prove Claim 1 we show that the image of (ϕ, ψ) through G (u, v) does not vanish for every (ϕ, ψ). This is achieved by letting (ϕ, ψ) = (u, v) and taking into account that G(u, v) = 0, which yields This implies that N is a C 1 -manifold of codimension 1, and the direction of (u, v) is nontangential, thus proving Claims 1 and 2 at once. To prove the last claim, observe that by the Poincaré inequality and the Sobolev embedding we have where C 0 ,C,C 1 > 0 are constants. Hence we may write uv q L q (Ω) ≤ C 2 (u, v) 2q , and therefore the inequality: holds provided that (u, v) < r 0 with a conveniently small r 0 > 0. The last claim follows, and the proof is complete.
Proof of Theorem 5.1. Let us check that the functional (31) satisfies the assumptions of the mountain-pass theorem.
Step 2: The functional J is unbounded from below. To see this, fix a pair (u, v) ∈ D k (Ω) satisfying uv q L q (Ω) > 0 in Ω. Since for every t ≥ 0 we have we see that J(tu,tv) → −∞ as t → ∞, hence J is unbounded from below, as claimed.
Step 3: The last condition needed to apply the mountain-pass theorem is the Palais-Smale compactness condition. More precisely, assume that a sequence of pairs (u i , v i ) ∈ D k (Ω) satisfies J(u i , v i ) → c ∈ (0, ∞) as i → ∞ in the Euclidean topology of the real line, as well as J (u i , v i ) → 0 as i → ∞ in the strong topology of the dual space (D k (Ω)) . Then we have to prove the existence of a strongly convergent subsequence in D k (Ω). To this purpose, observe that the differential J at the point (u i , v i ) is the linear functional L i (ϕ, ψ) given by where (ϕ, ψ) ranges in D k (Ω) ⊂ (L 2q (Ω)) 2 . In the special case when (ϕ, ψ) Let us combine the equality above with the assumption that J Now we are ready to prove the existence of a strongly convergent subsequence. As usual, the proof is divided into two parts.
Part i: The sequence (u i , v i ) is bounded. Indeed, if we assume (u i , v i ) → ∞ for i → ∞, then we reach a contradiction by the following argument. Taking (39) into account, we have which contradicts the assumption J(u i , v i ) → c < ∞ for i → ∞. Hence the sequence (u i , v i ) must be bounded, as claimed.
Part ii: Once we know that the sequence (u i , v i ) is bounded in D k (Ω), the proof of the existence of a strongly converging subsequence is standard: see [ To be more precise, by the weak compactness theorem in Hilbert spaces there exists a subsequence, still denoted by (u i , v i ), weakly convergent to some (u, v) ∈ D k (Ω). Furthermore, since q < N N−2s , the set D k (Ω) ⊂ H s 0 (Ω) is compactly embedded in the Lebesgue space L 2q (Ω), hence we may assume that when i → ∞ the sequences (u i ), (v i ) converge to u, v, respectively, strongly in L 2q (Ω), and therefore u i v i q L q (Ω) → uv q L q (Ω) . This and (39), taking the boundedness of the sequence (u i , v i ) into account, imply Consider the functional L(ϕ, ψ) in (32). Taking (36) into account, and since for every (ϕ, ψ) ∈ D k (Ω): thus, we have proved the weak- * convergence L i * L. But since L i → 0 strongly by assumption, we must have L = 0. In particular, (u, v) ∈ N . By comparing (33) with (40) we deduce lim Finally, by recalling that the weak convergence in a Hilbert space together with the convergence of the norms to the norm of the limiting function implies the strong convergence, we conclude that (u i , v i ) → (u, v) strongly in D k (Ω), which completes the proof of the Palais-Smale compactness condition.
At this point the mountain-pass theorem implies the existence of a critical point (u, v) = (0, 0) of the functional J, which is therefore a weak solution (u 1 , u 2 ) = (u, v) of the system (28). By the mountain-pass theorem we also know that the two identities u 1 ≡ 0 and u 2 ≡ 0 cannot hold at once, but we may, in principle, have u 2 ≡ 0. However, if u 2 vanishes identically, then system (28) implies Iu 1 = a 1 u 1 in Ω, u 1 = 0 in R N \ Ω, hence u 1 should also vanish identically by unique solvability and the maximum principle, a contradiction. A similar argument shows that (28), then by comparing the two equations -recall a 1 = a 2 -we obtain u 1 ≡ 0, which has been just excluded. Hence u 1 , u 2 are distinct functions, and the proof is complete.

Positivity
Let us now turn to show that the solutions u, v obtained so far do not change sign. To this aim we need to define the set of paths Γ = { γ ∈ C 0 ([0, 1], D k (Ω)) | γ(0) = 0, J(γ(1)) < 0 } and the two infima for a system of local equations. Let us verify that c ≤ c N . Take (u, v) ∈ N and observe that uv q L q (Ω) > 0, otherwise we would reach a contradiction with (33). Then (35) applies, and the path γ(t) = (tu,tv), t ∈ [0, ∞), starts from the origin and satisfies lim t→∞ J(γ(t)) = −∞. Of course, we may find a reparametrization such that J(γ(1)) < 0, but we prefer to avoid unnecessary technicalities. Taking (33) into account, a straightforward computation shows that the realvalued function f (t) = J(γ(t)) of the real variable t > 0 (whose graph is outlined in Figure 1) satisfies f (1) = 0. Furthermore, f attains its maximum (which is positive) at t = 1 and hence c ≤ max Proof. The argument is based on the combination of three inequalities: 1. Since (u, v) is a critical point of the functional J, we have G(u, v) = 0 (see (33)), hence the function for all t > 0, with equality if and only if t = 1 (the graph of f is outlined in Figure 1).
Proof. Consider the non-negative functions u 1 = |u| and u 2 = |v|, where (u, v) is the weak solution whose existence follows from Theorem 5.1. In view of Proposition 5.4, we must have either u 1 = u or u 1 = −u, and either u 2 = v or u 2 = −v. Therefore the pair (u 1 , u 2 ) satisfies (28). But then (u 1 , u 2 ) also satisfies the system of uncoupled inequalities By the strong maximum principle (see Proposition 3.2 and Remark 3.4) we have that for each j = 1, 2 either u j > 0 in Ω or u j ≡ 0 in R N , and the conclusion follows from Theorem 5.1.
Remark 5.6. (i) Since the weak solution (u 1 , u 2 ) whose existence is asserted by Corollary 5.5 minimizes the functional J over the Nehari manifold N , we say that (u 1 , u 2 ) is a ground state.

Polarized solutions
The main result in this paragraph states that if Ω is symmetric, then system (28) admits a solution made up of two polarized functions. Before proceeding further, observe that in our notation we may write u σ H (H) (x) = u H (σ H (x)). Let us describe the effect of polarization on the functionals J and G: The lemma also holds with G in place of J.
. Furthermore, by Cavalieri principle and since due to the symmetry of a 1 , a 2 we also have Ω a 1 u 2 dx = This and Proposition A.3 prove (41). Now suppose that (41) holds with equality. We may write where the last inequality follows from Proposition A.3. Since the right-hand side cannot be negative by Proposition A.2, it must vanish. But then Proposition A.2 implies that either u = u H or u = u σ H (H) , and either v = v H or v = v σ H (H) , as claimed, and Proposition A.3 implies that (47) holds. The argument obviously applies to the functional G as well. Assume further that k 0 satisfies (29) for some s ∈ (0, 1), and 1 < q < N N−2s . Then system (28) has a weak solution (u 1 , u 2 ) satisfying u 1 , u 2 > 0 in Ω, which satisfy either u j = (u j ) H for both j = 1, 2, or u j = (u j ) σ H (H) for both j = 1, 2. Furthermore, if u j is symmetric with respect to ∂ H for some j ∈ { 1, 2 }, then u 3− j is also symmetric. Proof.
Step 1: Construction of a polarized solution. Denote by v j = (u j ) H the polarization of u j for j = 1, 2. By Lemma 5.7, we find G(v 1 , v 2 ) ≤ G(u 1 , u 2 ) = 0, hence the real-valued function , whose graph has the shape depicted in Figure 1, satisfies because the function f (t) = J(tu 1 ,tu 2 ) attains its maximum at t = 1, hence Furthermore, recall that the value c = J(u 1 , u 2 ) is the minimum of J constrained to N by Lemma 5.3: this and (42) imply t 0 = 1 and J(u 1 , u 2 ) = J(v 1 , v 2 ). Hence the pair (v 1 , v 2 ), which is made up of polarized functions, positive in Ω, is also a minimizer of the functional J constrained to N , and therefore the intrinsic gradient, also called the tangential gradient, of the functional J on the manifold N vanishes there.
Let us prove that the normal component of the gradient vanishes as well. By Lemma 5.2 we know that the direction of (u, v) is non-tangential to N . Furthermore, by (33), the differential J at any (u, v) ∈ N vanishes in the direction of (u, v), hence the normal component of the gradient also vanishes, as claimed. But then J (v 1 , v 2 ) = 0, and therefore the pair (v 1 , v 2 ) is a weak solution of system (28).
Step 2: Comparison between solutions. Since J(u 1 , u 2 ) = J(v 1 , v 2 ), by Lemma 5.7 we have that either u 1 = v 1 or u 1 = (u 1 ) σ H (H) , and either u 2 = v 2 or u 2 = (u 2 ) σ H (H) . To prove the theorem we have to exclude two cases: the case when u 1 = v 1 and u 2 = v 2 , and the case when u 1 = v 1 and u 2 = v 2 . We examine the first case in detail, the second one being analogous. Suppose, by contradiction, that u 1 = v 1 and u 2 = v 2 . Then u 1 = (u 1 ) σ H (H) and there exists a set X 1 ⊂ Ω ∩ H having positive measure and such that u 1 (x) < u 1 (σ H (x)) = v 1 (x) for every x ∈ X 1 . We may assume that u 1 (x) = u 1 (σ H (x)) = v 1 (x) in (Ω ∩ H) \ X 1 . Recall that the pairs (u 1 , u 2 ) and (v 1 , v 2 ) = (v 1 , u 2 ) are both critical points of the functional J. Now the condition J (u 1 , u 2 ) = J (v 1 , u 2 ) = 0 comes into play: we have for every (ϕ, ψ) ∈ D k (Ω) (cf. (36)), and similarly Letting ϕ = 0 and ψ = (u 1 − v 1 ) − , and subtracting the second equality from the first one, we This contradiction shows that it is impossible to have u 1 = v 1 and u 2 = v 2 . The case when u 1 = v 1 and u 2 = v 2 is excluded similarly. Hence we must have either (u 1 , , as claimed. To complete the proof, suppose that u j is symmetric with respect to ∂ H for some j ∈ { 1, 2 }. For instance, suppose that u 2 is symmetric, the other case being analogous. Then u 2 = v 2 , and the preceding argument shows that u 1 = v 1 . Now we replace the half-space H with σ H (H), and we apply the same reasoning again, thus proving that u 1 = (u 1 ) σ H (H) , hence u 1 is symmetric. The proof is complete.
Remark 5. 10. We note that if u = u H , then either u is symmetric with respect to the reflection at ∂ H or there is x ∈ H such that u(x) > u(σ H (x)). Moreover, there exist non radial functions u : Ω → R, defined in a radial set Ω, polarized with respect to every half-space H. A twodimensional example is given by Ω = B 1 (0) ⊂ R 2 and u(x 1 , x 2 ) = x 1 (1 − |x| 2 ).

Proof of the existence of solutions with axial symmetry
In the following, we finish the proof of Theorem 1.4. For this we assume k 0 satisfies (9) with c > 0 and 0 < s ≤ σ < 1. Let Ω be an open, bounded, radial domain in R N , N ≥ 2, and let 1 < q < N N−2s . Moreover, we let a 1 , a 2 ∈ L ∞ (R) with a 1 = a 2 and a + i L ∞ (R) < λ 1 (Ω).
Proof of Theorem 1.4 completed. Note that by Theorem 5.1 and Corollary 5.5 it follows that there are u 1 , u 2 ∈ D k (Ω), u 1 , u 2 > 0 satisfying (10) with u 1 = u 2 . Moreover, by Theorem 5.9 and the radiality of Ω and a 1 , a 2 , it follows that for every half-space H with 0 ∈ ∂ H we have either Hence, if either u 1 or u 2 is not radial, it follows that after a rotation -and a renumbering if necessary-the assumption (6) is satisfied. Since clearly the right-hand sides of (10) satisfy (7) the statement of Theorem 1.4 follows from Theorem 1.2 once we have shown that u 1 and u 2 are bounded and continuous in Ω.
The boundedness of the solution pair follows indeed by a standard iteration argument using the Sobolev embedding theorem. We give the details of this argument in the appendix (see Lemma B.1 and Corollary B.3). Having the boundedness of u 1 and u 2 , the continuity of u 1 and u 2 in Ω follow e.g. from [22]. Thus (43) holds and the statement of Theorem 1.4 follows from Theorem 1.2 as mentioned before.
A On the polarization of a function in the nonlocal setting Recall the polarization of a function u with respect to an open half space defined in (18). Moreover, we use the notation of Section 5.
In the next proposition we show that polarization reduces the energy, with special care to the equality case (see also [2,Theorem 2] and [32,Proposition 8]). In the proof we will need the following (somehow surprising) identity: Lemma A.1 (Functional identity and inequality). Let H be a half-space in R N , and let u : For every x 1 , Proof. Define ξ j = 1 2 u(x j ) + u(σ H (x j )) and η j = 1 2 u(x j ) − u(σ H (x j )) , j = 1, 2, so that where we have used the assumption that x 1 , x 2 ∈ H. With this notation, we may write f (x 1 , x 2 ) = 2 |η 1 η 2 | − 2 η 1 η 2 and g(x 1 , x 2 ) = 2 η 1 η 2 − 2 |η 1 η 2 |, while inequality (44) reduces to η 1 η 2 ≥ 0. The lemma follows. Proof. We start by giving a convenient expression of E k (u, u). Since the integral is additive with respect to the domain of integration, we can split The last integral, by the change of variables x = σ H (x 1 ) and y = σ H (z), satisfies and therefore we may write Let us repeat the argument once more: we split Now in the first integral we write x 2 in place of z, and in the last integral we let x 2 = σ H (z), thus obtaining By a similar procedure we also obtain To go further, observe that u 2 (x j ) + u 2 (σ H (x j )) = u 2 H (x j ) + u 2 H (σ H (x j )) for j = 1, 2. Hence where f and g are as in Lemma A.1. Since f = −g, we may write When the pair (x 1 , x 2 ) ranges in the domain of integration H × H, the distance from x 1 to x 2 cannot be larger than the distance from x 1 to σ H (x 2 ) (see (45)). Since k 0 is monotone decreasing by assumption, it follows that k( To manage the special case when k 0 is strictly decreasing, we need the equality where d j ≥ 0 denotes the distance from x j to ∂ H, j = 1, 2. Equality (45) is established as follows. Let π j ∈ ∂ H be the projection of x j onto ∂ H, j = 1, 2. Then by the Pythagorean theorem (see Figure 2) we have (45) follows. Such an equality shows that k( In the first case, (44) implies that u(x 2 ) − u(σ H (x 2 )) ≥ 0 a.e. in H, hence u = u H . In the second case, (44) implies that u = u σ H (H) . The proof is complete.
The proposition above, which deals with the energy functional, is used in combination with the following, which deals with the L q -norm of the product of two given functions. Contrary to what one may expect, it turns out that polarization increases the norm:

B On the boundedness of solutions
In the following, N ≥ 2, and we assume that k 0 satisfies (9) for some s, γ, σ ∈ (0, 1) and c > 0. Moreover, Ω is an open bounded set in R N with Lipschitz boundary, E k (u, v) and D k (Ω) are defined as in Subsection 2.1. For a related result with the fractional Laplacian, see also [9,Lemma 2.3].
Remark B.2. Since u 1 , u 2 , ϕ ∈ D k (Ω), it follows immediately by the Sobolev embedding theorem that u 1 , u 2 , ϕ ∈ L m (Ω) for every m ∈ [1, 2 s ]. Since q ≤ 2 s 2 the integral in (49) converges. The strict inequality q < 2 s 2 is needed in the Moser iteration (see below). Proof of Lemma B.1. We follow the idea of Moser's iteration presented in [3] to show the claim.
Step 1: Preliminaries. Let g ∈ W 1,1 loc (R) be nondecreasing and define Then we have (see also [3,Lemma A.2]) for a, b ∈ R using Hölder's inequality Hence, if g : R → R is a nondecreasing Lipschitz function, that is, we have for some L g > 0, |g(a) − g(b)| ≤ L g |a − b| for all a, b ∈ R, it follows that E k (g(v), g(v)) ≤ L 2 g E k (v, v) and In particular, we see that both g(v) and G(v) belong to D k (Ω).
Step 2: A convenient Lipschitz function. To apply Moser's iteration, define for L > 0, r ≥ 2 the Lipschitz function Then g (t) = 0 = G(t) for t < 0, where G is defined as in (50), and for t > 0 we have By the definition of g, it follows that if v ≥ 0 then g(v) ≥ 0.
Step 3: Energy estimate from above. We perform a suitable truncation of the kernel k and the solutions u i : our purpose is to get rid of the linear term A |u i | in (49), thus proving (52). Let k δ := χ B δ (0) k and j δ := k − k δ .
Note that k δ satisfies the same assumptions as k and, in particular, we have D k δ (Ω) = D k (Ω) ⊂ L 2 s (Ω) for all δ > 0. Moreover, by our assumptions on k 0 , we have j δ ∈ L 1 (R N ) ∩ L 2 (R N ) for all δ > 0 and J δ := j δ L 1 (R N ) → ∞ for δ → 0, hence we can fix some δ > 0 such that With the Cauchy-Schwarz inequality and v in = (u i − n) + for n ∈ N 0 we have, since k δ = k − j δ and taking (49) into account, where we have used that u i ≥ n in the set { g(v in ) > 0 }. Since J δ > A and u i ∈ L 2 (Ω), we can fix from now on some n ∈ N large such that (A − J δ ) n + u i L 2 (R N ) j δ L 2 (R N ) ≤ 0, and therefore E k δ (u i , g(v in )) ≤ Ω |u 3−i (x)| q |u i (x)| q−1 g(v in (x)) dx, i = 1, 2.
Without loss of generality we take M ≥ 1 2 , so that Mr m+1 ≥ Mr 0 ≥ 1, and therefore we may write a m+1 ≤ max 1, (Mr m+1 ) Using (58), it is readily seen that the infinite product in the right-hand side converges to a (finite) limit, hence we have v in ∞ = lim m→∞ v in α m < ∞.
Since the argument above also holds with w in = (−u i − n) + in place of v in we conclude that u 1 , u 2 ∈ L ∞ (Ω).