A simple variational approach to weakly coupled competitive elliptic systems

Abstract

The main purpose of this paper is to exhibit a simple variational setting for finding fully nontrivial solutions to the weakly coupled elliptic system (1.1). We show that such solutions correspond to critical points of a $\mathcal {C}^1$-functional $\Psi :\mathcal {U}\rightarrow \mathbb {R}$ defined in an open subset $\mathcal {U}$ of the product $\mathcal {T}:=S_1\times \cdots \times S_M$ of unit spheres $S_i$ in an appropriate Sobolev space. We use our abstract setting to extend and complement some known results for the system (1.1).

1 Introduction

We study the weakly coupled elliptic system

$$\begin{aligned} {\left\{ \begin{array}{ll} -\Delta u_i + \kappa _iu_i = \mu _i|u_i|^{p-2}u_i + \sum \limits _{j\ne i}\lambda _{ij}\beta _{ij}|u_j|^{\alpha _{ij}}|u_i|^{\beta _{ij}-2}u_i, \\ u_i\in H,\qquad i,j=1,\ldots ,M, \end{array}\right. } \end{aligned}$$

(1.1)

where $\Omega$ is a domain in $\mathbb {R}^N$, $N\ge 3$, $\mu _i>0$, $\lambda _{ij}=\lambda _{ji}<0$, $\alpha _{ij},\beta _{ij}>1$, $\alpha _{ij}=\beta _{ji}$, and $\alpha _{ij}+\beta _{ij}=p\in (2,2^*]$. As usual, $2^*:=\frac{2N}{N-2}$ is the critical Sobolev exponent. The space H is, either $H^1_0(\Omega )$, or $D^{1,2}_0(\Omega )$, and the operators $-\Delta + \kappa _i$ are assumed to be well defined and coercive in H.

The cubic system (1.1) in $\mathbb {R}^3$ with $\alpha _{ij}=\beta _{ij}=2$ arises as a model in many physical phenomena, for example, in the study of standing waves for a mixture of Bose–Einstein condensates of M-hyperfine states which overlap in space. The sign of $\mu _i$ reflects the interaction of the particles within each single state, whereas that of $\lambda _{ij}$ reflects the interaction between particles in two different states. The interaction is attractive if the sign is positive, and it is repulsive if the sign is negative. The system is called competitive if, as we are assuming here, all of the $\lambda _{ij}$’s are negative.

A solution $u_i$ to the equation

$$\begin{aligned} -\Delta u + \kappa _iu = \mu _i|u|^{p-2}u,\qquad u\in H, \end{aligned}$$

gives rise to a solution of the system (1.1) whose i-th component is $u_i$ and all other components are trivial, i.e., $u_j=0$ if $j\ne i$. A solution with at least one trivial and one nontrivial component is called semitrivial. We are interested in finding solutions all of whose components are nontrivial. These are called fully nontrivial solutions. A fully nontrivial solution is said to be positive if every component $u_i$ is nonnegative.

The main purpose of this paper is to exhibit a simple variational setting for finding fully nontrivial solutions to the system (1.1). Our approach is inspired by the ideas introduced by Szulkin and Weth in [19, 20].

We will show that the fully nontrivial solutions to (1.1) correspond to the critical points of a $\mathcal {C}^1$-functional $\Psi :\mathcal {U}\rightarrow \mathbb {R}$ defined in an open subset $\mathcal {U}$ of the product $\mathcal {T}:=S_1\times \cdots \times S_M$ of unit spheres $S_i$ in H. The functional $\Psi$ tends to infinity at the boundary of $\mathcal {U}$ in $\mathcal {T}$, thus allowing the application of the usual descending gradient flow techniques to obtain existence and multiplicity of critical points.

This variational setting can be easily extended to systems whose coefficients $\kappa _i,\mu _i,\lambda _{ij}$ are functions defined in $\Omega$ and satisfying suitable assumptions. It may also be extended, with some care, to systems having more general nonlinearities. We chose to treat only the constant coefficient system (1.1) in order to make the ideas more transparent.

Our abstract results (Theorems 3.3 and 3.4) apply to many interesting types of systems. Here we consider the following three.

Firstly, we consider the subcritical system

$$\begin{aligned} {\left\{ \begin{array}{ll} -\Delta u_i + \kappa _iu_i = \mu _i|u_i|^{p-2}u_i + \sum \limits _{j\ne i} \lambda _{ij}\beta _{ij}|u_j|^{\alpha _{ij}}|u_i|^{\beta _{ij}-2}u_i, \\ u_i\in H^1_0(\Omega ),\qquad i,j=1,\ldots ,M, \end{array}\right. } \end{aligned}$$

(1.2)

with $\kappa _i,\mu _i>0$, $\lambda _{ij}=\lambda _{ji}<0$, $\alpha _{ij},\beta _{ij}>1$, $\alpha _{ij}=\beta _{ji}$, and $\alpha _{ij}+\beta _{ij}=p\in (2,2^*)$, in an exterior domain $\Omega$ of $\mathbb {R}^N$ (i.e., $\mathbb {R}^N\smallsetminus \Omega$ is bounded, possibly empty), $N\ge 3$.

We assume that $\Omega$ is invariant under the action of a closed subgroup G of the group O(N) of linear isometries of $\mathbb {R}^N$, and look for G-invariant solutions, i.e., solutions whose components are G-invariant.

Let $Gx:=\{gx:g\in G\}$ denote the G-orbit of $x\in \mathbb {R}^N$. We prove the following result.

Theorem 1.1

If $\dim (Gx)>0$ for every $x\in \mathbb {R}^N\smallsetminus \{0\}$ and $\Omega$ is a G-invariant exterior domain in $\mathbb {R}^N$, then the system (1.2) has an unbounded sequence of G-invariant fully nontrivial solutions. One of them is positive and has least energy among all G-invariant fully nontrivial solutions.

There is an extensive literature on subcritical systems in bounded domains and in the whole of $\mathbb {R}^3$. We refer to [17] for a detailed account. Theorem 1.1 seems to be the first existence result for the system (1.2) in an exterior domain. A cubic system of two equations with variable coefficients in an expanding exterior domain was recently considered in [10].

Our second application concerns the critical system

$$\begin{aligned} {\left\{ \begin{array}{ll} -\Delta u_i = \mu _i|u_i|^{2^*-2}u_i + \sum \limits _{j\ne i} \lambda _{ij}\beta _{ij}|u_j|^{\alpha _{ij}}|u_i|^{\beta _{ij}-2}u_i, \\ u_i\in D^{1,2}(\mathbb {R}^N),\qquad i,j=1,\ldots ,M, \end{array}\right. } \end{aligned}$$

(1.3)

where $N\ge 3$, $\mu _i>0$, $\lambda _{ij}=\lambda _{ji}<0$, $\alpha _{ij},\beta _{ij}>1$, $\alpha _{ij}=\beta _{ji}$, and $\alpha _{ij}+\beta _{ij}=2^*$.

We look for solutions which are invariant under the conformal action of the group $\Gamma :=O(m)\times O(n)$ on $\mathbb {R}^N$, with $m+n=N+1$ and $n,m\ge 2$, which is induced by the isometric action of $\Gamma$ on the standard N-dimensional sphere, by means of the stereographic projection. We prove the following result.

Theorem 1.2

The system (1.3) has an unbounded sequence of $\Gamma$-invariant fully nontrivial solutions. One of them is positive and has least energy among all $\Gamma$-invariant fully nontrivial solutions.

Theorem 1.2 extends some earlier results obtained in [5, 6] for a system of two equations; see also [9]. Existence and multiplicity results for the purely critical system in a bounded domain may be found in [5, 13, 14]. Supercritical systems were recently considered in [4].

Finally, we consider the critical system

$$\begin{aligned} {\left\{ \begin{array}{ll} -\Delta u_i + \kappa _i u_i = \mu _i|u_i|^{2^*-2}u_i + \sum \limits _{j\ne i} \lambda _{ij}\beta _{ij}|u_j|^{\alpha _{ij}}|u_i|^{\beta _{ij}-2}u_i, \\ u_i\in D^{1,2}_0(\Omega ),\qquad i,j=1,\ldots ,M, \end{array}\right. } \end{aligned}$$

(1.4)

where $\Omega$ is a bounded domain with $\mathcal {C}^2$-boundary in $\mathbb {R}^N$, $N\ge 4$, $\kappa _i\in (-\lambda _1(\Omega ),0)$, $\mu _i>0$, $\lambda _{ij}=\lambda _{ji}<0$, $\alpha _{ij},\beta _{ij}>1$, $\alpha _{ij}=\beta _{ji}$, and $\alpha _{ij}+\beta _{ij}=2^*$. As usual, $\lambda _1(\Omega )$ denotes the first Dirichlet eigenvalue of $-\Delta$ in $\Omega$.

We prove the following result.

Theorem 1.3

Let $N\ge 4$. Assume that $\min \{\alpha _{ij},\beta _{ij}\}\ge \frac{4}{3}$ if $N=5$ and that $\alpha _{ij}=\beta _{ij}=2$ if $N=4$, for all $i,j=1,\ldots ,M$. Then, the system (1.4) has a positive least energy fully nontrivial solution.

Note that there is no condition on $\alpha _{ij},\beta _{ij}$, other than $\alpha _{ij},\beta _{ij}>1$ and $\alpha _{ij}+\beta _{ij}=2^*$, if $N\ge 6$.

Theorem 1.3 extends some earlier results obtained in [2, 3] for a system of two equations. Multiple positive solutions were constructed in [15] when $N=4$, and the existence of infinitely many sign-changing solutions was established in [11] when $N\ge 7$ and $\alpha _{ij}=\beta _{ij}=\frac{2^*}{2}$; see also [12].

Our variational approach is based on some elementary properties of a certain function in M variables, which are established in Sect. 2. In Sect. 3 we introduce our variational setting and we derive some abstract results concerning the existence and multiplicity of fully nontrivial solutions to the system (1.1). Section 4 is devoted to the proof of Theorems 1.1, 1.2 and 1.3.

2 On a function in M variables

Let $J:(0,\infty )^M\rightarrow \mathbb {R}$ be the function given by

$$\begin{aligned} J(s):= \sum _{i=1}^Ma_is_i^2 - \sum _{i=1}^Mb_is_i^p + \sum _{i\ne j}d_{ij}s_j^{\alpha _{ij}}s_i^{\beta _{ij}}, \end{aligned}$$

where $s=(s_1,\ldots ,s_M)$, $a_i,b_i>0$, $d_{ij}\ge 0$, $d_{ij}=d_{ji}$, $\alpha _{ij},\beta _{ij}>1$, $\alpha _{ij}+\beta _{ij}=p>2$, and $\alpha _{ji}=\beta _{ij}$. Then, for $i=1,\ldots , M$,

$$\begin{aligned} \partial _i J(s)= & {} 2a_is_i-pb_is_i^{p-1} + \sum _{j\ne i}d_{ij}\beta _{ij}s_j^{\alpha _{ij}}s_i^{\beta _{ij}-1} + \sum _{j\ne i}d_{ji}\alpha _{ji}s_i^{\alpha _{ji}-1}s_j^{\beta _{ji}} \nonumber \\= & {} 2a_is_i-pb_is_i^{p-1} + 2\sum _{j\ne i}d_{ij}\beta _{ij}s_j^{\alpha _{ij}}s_i^{\beta _{ij}-1}. \end{aligned}$$

(2.1)

Lemma 2.1

If $pb_i > 2\sum _{j\ne i}d_{ij}\beta _{ij}$ for all $i=1,\ldots ,M$, then there exist $0<r<R<\infty$ such that

$$\begin{aligned} \max _{s\in (0,\infty )^M}J(s) = \max _{s\in [r,R]^M}J(s). \end{aligned}$$

(2.2)

In particular, J attains its maximum on $(0,\infty )^M$.

Proof

Fix $R>r>0$ such that, for all $i=1,\ldots ,M$,

$$\begin{aligned} 2a_{i}t - \left( pb_i - 2\sum _{j\ne i}d_{ij}\beta _{ij}\right) t^{p-1} < 0 \qquad \text {if } t\in [R,\infty ) \end{aligned}$$

and

$$\begin{aligned} 2a_{i}t-pb_{i}t^{p-1}>0\qquad \text {if }t\in (0,r]. \end{aligned}$$

Let $s=(s_1,\ldots ,s_M)\in (0,\infty )^M$. If $s_i\ge R \text { and } s_i=\max \{s_1,\ldots ,s_M\}$, we have that

$$\begin{aligned} \partial _iJ(s) \le 2a_{i}s_i - \left( pb_i - 2\sum _{j\ne i}d_{ij}\beta _{ij}\right) s_i^{p-1} < 0, \end{aligned}$$

(2.3)

whereas, if $s_i\le r$, then

$$\begin{aligned} \partial _i J(s) \ge 2a_{i}s_i-pb_{i}s_i^{p-1}>0. \end{aligned}$$

(2.4)

Therefore (2.2) holds true. $\square$

Lemma 2.2

If J has a critical point in $(0,\infty )^M$, then it is unique and it is a global maximum of J in $(0,\infty )^M$.

Proof

Assume first that $(1,\ldots ,1)$ is a critical point of J. Then, from (2.1) we get that

$$\begin{aligned} 0<2a_{i} = pb_i - 2\sum _{j\ne i}d_{ij}\beta _{ij} \qquad \text {for all } i=1,\ldots ,M. \end{aligned}$$

(2.5)

If $s=(s_1,\ldots ,s_M)$ is a critical point of J in $(0,\infty )^M$, then, for each $i=1,\ldots ,M$, (2.1) and (2.5) yield

$$\begin{aligned} 2a_i(s_i - s_i^{p-1}) = 2\sum _{j\ne i}d_{ij}\beta _{ij}(s_i^{p-1} - s_j^{\alpha _{ij}}s_i^{\beta _{ij}-1}). \end{aligned}$$

(2.6)

Arguing by contradiction, assume that $s\ne (1,\ldots ,1)$. We consider two cases. Suppose first that $s_i>1$ for some i. We may assume without loss of generality that $s_i\ge s_j$ for all j. Then, the left-hand side in (2.6) is negative whereas the right-hand side is $\ge 0$. This is a contradiction. Now suppose that $s_i<1$ for some i. Again, we may assume that $s_i\le s_j$ for all j. Now the left-hand side in (2.6) is positive while the right-hand side is not, a contradiction again. Hence $(1,\ldots ,1)$ is the only critical point of J in $(0,\infty )^M$. The inequalities (2.5) allow us to apply Lemma 2.1 to conclude that $(1,\ldots ,1)$ is a global maximum.

Now, if $s^0=(s_1^0,\ldots ,s_M^0)$ is a critical point of J in $(0,\infty )^M$, then $(1,\ldots ,1)$ is a critical point of

$$\begin{aligned} {\bar{J}}(s):= \sum _{i=1}^M {\bar{a}}_is_i^2 - \sum _{i=1}^M {\bar{b}}_is_i^p + \sum _{i\ne j} {\bar{d}}_{ij}s_j^{\alpha _{ij}}s_i^{\beta _{ij}}, \end{aligned}$$

where ${\bar{a}}_i:=a_i s_i^0$, ${\bar{b}}_i :=b_i (s_i^0)^{p-1}$ and ${\bar{d}}_{ij} :=d_{ij} (s_j^0)^{\alpha _{ij}}(s_i^0)^{\beta _{ij}-1}$, and the conclusion follows from the special case considered above. $\square$

Lemma 2.3

Assume that J has a critical point $s^0$ in $(0,\infty )^M$. Then, for each $\varepsilon >0$, there exists $\delta >0$ such that, if $\widetilde{d}_{ij}\ge 0$ for all $i\ne j$ and

$$\begin{aligned} \sum _{i=1}^M(|{\widetilde{a}}_i-a_i|+|{\widetilde{b}}_i-b_i|) + \sum _{i\ne j}|{\widetilde{d}}_{ij}-d_{ij}| < \delta , \end{aligned}$$

(2.7)

then the function

$$\begin{aligned} {\widetilde{J}}(s) := \sum _{i=1}^M {\widetilde{a}}_is_i^2 - \sum _{i=1}^M \widetilde{b}_is_i^p + \sum _{i\ne j} \widetilde{d}_{ij}s_j^{\alpha _{ij}}s_i^{\beta _{ij}} \end{aligned}$$

has a unique critical point ${\widetilde{s}}^0$ in $(0,\infty )^M$ which is a global maximum and satisfies $|{\widetilde{s}}^0 - s^0|<\varepsilon$.

Proof

As in the proof of Lemma 2.2, we may assume without loss of generality that $s^0=(1,\ldots ,1)$. Then, (2.5) holds true. So, choosing $\delta >0$ small enough, we have that ${\widetilde{a}}_i,\, {\widetilde{b}}_i > 0$ and $p{\widetilde{b}}_i - 2\sum _{j\ne i}{\widetilde{d}}_{ij}\beta _{ij} >0$ if (2.7) is satisfied. Thus, by Lemma 2.1, ${\widetilde{J}}$ has a global maximum ${\widetilde{s}}^0$ in $(0,\infty )^M$ and, by Lemma 2.2, it is the only critical point of ${\widetilde{J}}$ in $(0,\infty )^M$.

Taking smaller $\delta ,r>0$ and a larger $R>r$ if necessary, we have that $\widetilde{J}$ satisfies the same inequalities and, therefore, ${\widetilde{s}}^0\in (r,R)^M$. Since $(1,\ldots ,1)$ is a strict maximum, it is easy to see that $|{\widetilde{s}}^0 - (1,\ldots ,1)|<\varepsilon$, possibly after choosing a still smaller $\delta$. $\square$

3 The variational setting

The results of this section also apply to the case $N=1$ or 2 and $p\in (2,\infty )$.

Let H be either $H^1_0(\Omega )$ or $D^{1,2}_0(\Omega )$ and, for $v,w\in H$, set

$$\begin{aligned} \langle v,w\rangle _i:=\int _\Omega (\nabla v\cdot \nabla w+\kappa _ivw) \qquad \text {and}\qquad \Vert v\Vert _i:=\left( \int _\Omega (|\nabla v|^2+\kappa _iv^2)\right) ^{1/2}. \end{aligned}$$

Since, by assumption, the operators $-\Delta + \kappa _i$ are well defined and coercive in H, we have that $\Vert \cdot \Vert _i$ is a norm in H, equivalent to the standard one.

Let $\mathcal {H}:=H^M$ with the norm

$$\begin{aligned} \Vert (u_1,\ldots ,u_M)\Vert :=\left( \sum _{i=1}^M\Vert u_i\Vert _i^2\right) ^{1/2}, \end{aligned}$$

and let $\mathcal {J}:\mathcal {H}\rightarrow \mathbb {R}$ be given by

$$\begin{aligned} \mathcal {J}(u_1,\ldots ,u_M) := \frac{1}{2}\sum _{i=1}^M\Vert u_i\Vert _i^2 - \frac{1}{p}\sum _{i=1}^M\int _\Omega \mu _i|u_i|^p - \frac{1}{2}\sum _{j\ne i}\int _\Omega \lambda _{ij}|u_j|^{\alpha _{ij}}|u_i|^{\beta _{ij}}. \end{aligned}$$

This function is of class $\mathcal {C}^1$ and, since $\lambda _{ij}=\lambda _{ji}$ and $\beta _{ij}=\alpha _{ji}$,

$$\begin{aligned}&\partial _i\mathcal {J}(u_1,\ldots ,u_M)v=\langle u_i,v\rangle _i - \int _\Omega \mu _i|u_i|^{p-2}u_iv \\&\qquad - \frac{1}{2}\sum _{j\ne i}\int _\Omega \lambda _{ij}\beta _{ij}|u_j|^{\alpha _{ij}}|u_i|^{\beta _{ij}-2}u_iv - \frac{1}{2}\sum _{j\ne i}\int _\Omega \lambda _{ji}\alpha _{ji}|u_i|^{\alpha _{ji}-2}u_iv|u_j|^{\beta _{ji}}\\&\quad = \langle u_i,v\rangle _i - \int _\Omega \mu _i|u_i|^{p-2}u_iv - \sum _{j\ne i}\int _\Omega \lambda _{ij}\beta _{ij}|u_j|^{\alpha _{ij}}|u_i|^{\beta _{ij}-2}u_iv, \end{aligned}$$

for each $v\in H$, $i=1,\ldots ,M$. So the critical points of $\mathcal {J}$ are the solutions to the system (1.1). The fully nontrivial ones belong to the set

$$\begin{aligned} \mathcal {N} := \{(u_1,\ldots ,u_M)\in \mathcal {H}:u_i\ne 0, \;\partial _i\mathcal {J}(u_1,\ldots ,u_M)u_i=0, \; \forall i=1,\ldots ,M\}. \end{aligned}$$

This Nehari-type set was introduced in [7], and has been used in many works. Note that

$$\begin{aligned} \mathcal {J}(u) = \frac{p-2}{2p}\sum _{i=1}^M\Vert u_i\Vert _i^2\qquad \text {if }u=(u_1,\ldots ,u_M)\in \mathcal {N}. \end{aligned}$$

(3.1)

Given $u=(u_1,\ldots ,u_M)\in \mathcal {H}$ and $s=(s_1,\ldots ,s_M)\in (0,\infty )^M$, we write

$$\begin{aligned} su:=(s_1u_1,\ldots ,s_Mu_M), \end{aligned}$$

and we define $J_u:(0,\infty )^M\rightarrow \mathbb {R}$ by

$$\begin{aligned} J_u(s):=\mathcal {J}(su) = \sum _{i=1}^Ma_{u,i}s_i^2 - \sum _{i=1}^Mb_{u,i}s_i^p + \sum _{i\ne j}d_{u,ij}s_j^{\alpha _{ij}}s_i^{\beta _{ij}}, \end{aligned}$$

where

$$\begin{aligned} a_{u,i}:=\frac{1}{2}\Vert u_i\Vert _i^2,\qquad b_{u,i}:=\frac{1}{p}\int _\Omega \mu _i |u_i|^p,\qquad d_{u,ij}:=-\frac{1}{2}\int _\Omega \lambda _{ij}|u_j|^{\alpha _{ij}}|u_i|^{\beta _{ij}}. \end{aligned}$$

If $u_i\ne 0$ for all $i=1,\ldots ,M$, then, as

$$\begin{aligned} s_i\,\partial _i J_{u}(s)=\partial _i\mathcal {J}(su)[s_iu_i],\qquad i=1,\ldots ,M, \end{aligned}$$

we have that s is a critical point of $J_u$ iff $su\in \mathcal {N}$. Define

$$\begin{aligned} \widetilde{\mathcal {U}}&:= \{u\in \mathcal {H}: su\in \mathcal {N} \text { for some }s\in (0,\infty )^M\} \\&=\{u\in (H\smallsetminus \{0\})^M: J_u \text { has a critical point in }(0,\infty )^M\}. \end{aligned}$$

By Lemma 2.2, if $u\in (H\smallsetminus \{0\})^M$ and $J_u$ has a critical point in $(0,\infty )^M$, then this critical point is unique and it is a global maximum of $J_u$. We denote it by $s_u=(s_{u,1},\ldots ,s_{u,M})$, and we define $\widetilde{\mathfrak {m}}:\widetilde{\mathcal {U}}\rightarrow \mathcal {N}$ by

$$\begin{aligned} \widetilde{\mathfrak {m}}(u):=s_uu. \end{aligned}$$

Then,

$$\begin{aligned} \mathcal {J}(\widetilde{\mathfrak {m}}(u))=\max _{s\in (0,\infty )^M}\mathcal {J}(su). \end{aligned}$$

(3.2)

Let $S_i:=\{v\in H:\Vert v\Vert _i=1\}$, $\mathcal {T}:=S_1\times \cdots \times S_M$, $\mathcal {U}:=\widetilde{\mathcal {U}}\cap \mathcal {T}$, and let $\mathfrak {m}:\mathcal {U}\rightarrow \mathcal {N}$ be the restriction of $\widetilde{\mathfrak {m}}$ to $\mathcal {U}$. We write $\partial \mathcal {U}$ for the boundary of $\mathcal {U}$ in $\mathcal {T}$.

Proposition 3.1

(a):

If $u=(u_1,\ldots ,u_M)\in \mathcal {T}$ is such that $u_i$ and $u_j$ have disjoint supports for every $i\ne j$, then $u\in \mathcal {U}$. Hence $\mathcal {U}\ne \emptyset$. Moreover, $\mathcal {U}$ is an open subset of $\mathcal {T}$.

(b):

$\mathcal {U}\ne \mathcal {T}$ if $-\lambda _{ij}\ge \max \{\frac{\mu _i}{\beta _{ij}},\,\frac{\mu _j}{\beta _{ji}}\}$ for some $i\ne j$.

(c):

$\widetilde{\mathfrak {m}}:\widetilde{\mathcal {U}}\rightarrow \mathcal {N}$ is continuous, and $\mathfrak {m}:\mathcal {U}\rightarrow \mathcal {N}$ is a homeomorphism.

(d):

There exists $d_0>0$ such that $\min _{i=1,\ldots ,M}\Vert u_i\Vert _i\ge d_0$ if $(u_1,\ldots ,u_M)\in \mathcal {N}$. Thus, $\mathcal {N}$ is a closed subset of $\mathcal {H}$.

(e):

If $(u_n)$ is a sequence in $\mathcal {U}$ such that $u_n\rightarrow u\in \partial \mathcal {U}$, then $\Vert \mathfrak {m}(u_n)\Vert \rightarrow \infty$.

Proof

(a) :  Let $u=(u_1,\ldots ,u_M)\in \mathcal {T}$ be such that $u_i$ and $u_j$ have disjoint supports if $i\ne j$. Then, $d_{u,ij}=0$ for every $i\ne j$, and, setting $s_i:=(\mu _i\int _\Omega |u_i|^p)^{-1/(p-2)}$, we have that $(s_1u_1,\ldots ,s_Mu_M)\in \mathcal {N}$. This proves that $u\in \mathcal {U}$. Moreover, as $a_{u,i},b_{u,i},d_{u,ij}$ are continuous functions of u, Lemma 2.3 implies that $\mathcal {U}$ is open.

(b) :  We assume without loss of generality that $i=1$ and $j=2$. Let $v,v_3,\ldots ,v_M\in H$ be nontrivial functions. Assume there exist $t_1,t_2>0$ such that $(t_1v,t_2v,v_3,\ldots ,v_M)\in \mathcal {N}$. Then, as $\alpha _{ij}+\beta _{ij}=p$ and $\lambda _{ij}<0$ for all i, j, we have that

$$\begin{aligned} 0< t_1^2\Vert v\Vert ^2&\le \mu _1t_1^p\int _\Omega |v|^p+\lambda _{12}\beta _{12}t_2^{\alpha _{12}} t_1^{\beta _{12}}\int _\Omega |v|^p \\&= t_1^{\beta _{12}}\int _\Omega |v|^p\,\left( \mu _1 t_1^{\alpha _{12}}+\lambda _{12}\beta _{12} t_2^{\alpha _{12}}\right) , \\ 0 < t_2^2\Vert v\Vert ^2&\le \mu _2t_2^p\int _\Omega |v|^p+\lambda _{21}\beta _{21}t_1^{\alpha _{21}} t_2^{\beta _{21}}\int _\Omega |v|^p \\&= t_2^{\beta _{21}}\int _\Omega |v|^p\,\left( \mu _2 t_2^{\alpha _{21}}+\lambda _{21}\beta _{21} t_1^{\alpha _{21}}\right) . \end{aligned}$$

Since $\lambda _{12}=\lambda _{21}$ and the right-hand sides above must be positive, we get that

$$\begin{aligned} \frac{t_1^{\alpha _{12}}}{t_2^{\alpha _{12}}}>-\lambda _{12}\frac{\beta _{12}}{\mu _1}\qquad \text {and}\qquad \frac{t_2^{\alpha _{21}}}{t_1^{\alpha _{21}}}>-\lambda _{12}\frac{\beta _{21}}{\mu _2}, \end{aligned}$$

which is impossible if $-\lambda _{12}\ge \max \{\frac{\mu _1}{\beta _{12}},\,\frac{\mu _2}{\beta _{21}}\}$. So, if this last inequality holds true, then

$$\begin{aligned} \left( \frac{v}{\Vert v\Vert _1},\frac{v}{\Vert v\Vert _2},\frac{v_3}{\Vert v_3\Vert _3},\ldots ,\frac{v_M}{\Vert v_M\Vert _M}\right) \in \mathcal {T}\smallsetminus \mathcal {U}. \end{aligned}$$

(3.3)

(c) :  If $(u_n)$ is a sequence in $\widetilde{\mathcal {U}}$ and $u_n\rightarrow u\in \widetilde{\mathcal {U}}$, then, for each $i,j=1,\ldots ,M$ with $i\ne j$, we have that $a_{u_n,i}\rightarrow a_{u,i}$,  $b_{u_n,i}\rightarrow b_{u,i}$ and $d_{u_{n},ij}\rightarrow d_{u,ij}$. So, from Lemma 2.3 we get that $s_{u_{n},i}\rightarrow s_{u,i}$. Hence, $\widetilde{\mathfrak {m}}:\widetilde{\mathcal {U}}\rightarrow \mathcal {N}$ is continuous.

The inverse of $\mathfrak {m}:\mathcal {U}\rightarrow \mathcal {N}$ is given by

$$\begin{aligned} \mathfrak {m}^{-1}(u_1,\ldots ,u_M) = \left( \frac{u_1}{\Vert u_1\Vert _1},\ldots ,\frac{u_M}{\Vert u_M\Vert _M}\right) , \end{aligned}$$

which is, obviously, continuous.

(d) :  If $(u_1,\ldots ,u_M)\in \mathcal {N}$ then, as $\lambda _{ij}<0$ for every $i\ne j$, we have that $\Vert u_i\Vert _i^2\le \mu _i\int _\Omega |u_i|^p$ for all $i=1,\ldots ,M$. The statement now follows from Sobolev’s inequality.

(e) :  Let $(u_n)$ be a sequence in $\mathcal {U}$ such that $u_n\rightarrow u\in \partial \mathcal {U}$. If the sequence $(s_{u_{n},i})$ were bounded for every $i=1,\ldots ,M$, then, after passing to a subsequence, $s_{u_{n},i}\rightarrow s_i$. Since $\mathcal {N}$ is closed, we would have that $(s_1u_1,\ldots ,s_Mu_M)\in \mathcal {N}$ and, therefore, $u\in \mathcal {U}$. This is impossible because $u\in \partial \mathcal {U}$ and $\mathcal {U}$ is open in $\mathcal {T}$. $\square$

A fully nontrivial solution u to (1.1) will be called synchronized if $u_i=t_iv$ and $u_j=t_jv$ for some $i\ne j$ and $t_i,t_j\in \mathbb {R}$.

Proposition 3.2

There exists $\Lambda _0<0$ such that if $\lambda _{ij} < \Lambda _0$ for all i, j, then the system (1.1) has no fully nontrivial synchronized solutions.

Proof

Choose $\Lambda _0$ such that $-\Lambda _0\ge \max \{\frac{\mu _i}{\beta _{ij}},\,\frac{\mu _j}{\beta _{ji}}\}$ for all $i\ne j$. Then (3.3) holds true and so u cannot be a solution to (1.1). $\square$

$\mathcal {T}$ is a smooth Hilbert submanifold of $\mathcal {H}$. The tangent space to $\mathcal {T}$ at a point $u=(u_1,\ldots ,u_M)\in \mathcal {T}$ is the space

$$\begin{aligned} \mathrm {T}_u(\mathcal {T}):=\{(v_1,\ldots ,v_M)\in \mathcal {H}:\langle u_i,v_i\rangle _i=0\text { for all }i=1,\ldots ,M\}. \end{aligned}$$

Let $\widetilde{\Psi }:\widetilde{\mathcal {U}}\rightarrow \mathbb {R}$ be given by $\widetilde{\Psi }(u) := \mathcal {J}(\widetilde{\mathfrak {m}}(u))$, and let $\Psi$ be the restriction of $\widetilde{\Psi }$ to $\mathcal {U}$. Then,

$$\begin{aligned} \Psi (u) =\frac{p-2}{2p}\sum _{i=1}^M\Vert s_{u,i}u_i\Vert _i^2=\frac{p-2}{2p}\sum _{i=1}^M s_{u,i}^2\quad \text {for every }u\in \mathcal {U}. \end{aligned}$$

(3.4)

If $u\in \mathcal {U}$ and the derivative $\Psi '(u)$ of $\Psi$ at u exists, then

$$\begin{aligned} \Vert \Psi '(u)\Vert _*:=\sup \limits _{\begin{array}{c} v\in \mathrm {T}_u(\mathcal {T}) \\ v\ne 0 \end{array}}\frac{|\Psi '(u)v|}{\Vert v\Vert }, \end{aligned}$$

i.e., $\Vert \cdot \Vert _*$ is the norm in the cotangent space $\mathrm {T}_u^*(\mathcal {T})$ to $\mathcal {T}$ at u. A sequence $(u_n)$ in $\mathcal {U}$ is called a $(PS)_c$-sequence for $\Psi$ if $\Psi (u_n)\rightarrow c$ and $\Vert \Psi '(u_n)\Vert _*\rightarrow 0$, and $\Psi$ is said to satisfy the $(PS)_c$-condition if every such sequence has a convergent subsequence.

As usual, a $(PS)_c$-sequence for $\mathcal {J}$ is a sequence $(u_n)$ in $\mathcal {H}$ such that $\mathcal {J}(u_n)\rightarrow c$ and $\Vert \mathcal {J}'(u_n)\Vert _{\mathcal {H}^{-1}}\rightarrow 0$, and $\mathcal {J}$ satisfies the $(PS)_c$-condition if any such sequence has a convergent subsequence.

Theorem 3.3

(i):

$\Psi \in \mathcal {C}^1(\mathcal {U},\mathbb {R})$ and

$$\begin{aligned} \Psi '(u)v = \mathcal {J}'(\mathfrak {m}(u))[s_uv] \quad \text {for all } u\in \mathcal {U} \text { and } v\in \mathrm {T}_u(\mathcal {T}). \end{aligned}$$

(ii):

If $(u_n)$ is a $(PS)_c$-sequence for $\Psi$, then $(\mathfrak {m}(u_n))$ is a $(PS)_c$-sequence for $\mathcal {J}$. Conversely, if $(u_n)$ is a $(PS)_c$-sequence for $\mathcal {J}$ and $u_n\in \mathcal {N}$ for all $n\in \mathbb {N}$, then $(\mathfrak {m}^{-1}(u_n))$ is a $(PS)_c$-sequence for $\Psi$.

(iii):

u is a critical point of $\Psi$ if and only if $\mathfrak {m}(u)$ is a fully nontrivial critical point of $\mathcal {J}$.

(iv):

If $(u_n)$ is a sequence in $\mathcal {U}$ such that $u_n\rightarrow u\in \partial \mathcal {U}$, then $\Psi (u_n) \rightarrow \infty$.

(v):

$\Psi$ is even, i.e., $\Psi (u)=\Psi (-u)$ for every $u\in \mathcal {U}$.

Proof

We adapt the arguments of Proposition 9 and Corollary 10 in [20].

(i) :  Let $u\in \widetilde{\mathcal {U}}$ and $v\in \mathcal {H}$. As $s_u$ is the maximum of $J_u$, using the mean value theorem we obtain

$$\begin{aligned}&{\widetilde{\Psi }}(u+tv)-{\widetilde{\Psi }}(u) = \mathcal {J}(s_{u+tv}(u+tv)) - \mathcal {J}(s_uu) \\&\qquad \le \mathcal {J}(s_{u+tv}(u+tv)) - \mathcal {J}(s_{u+tv}u) = \mathcal {J}'(s_{u+tv}(u+\tau _1 tv))\,[ts_{u+tv}v], \end{aligned}$$

for |t| small enough and some $\tau _1\in (0,1)$. Similarly,

$$\begin{aligned} {\widetilde{\Psi }}(u+tv)-{\widetilde{\Psi }}(u) \ge \mathcal {J}(s_{u}(u+tv)) - \mathcal {J}(s_{u}u) = \mathcal {J}'(s_{u}(u+\tau _2 tv))\,[ts_{u}v], \end{aligned}$$

for some $\tau _2\in (0,1)$. From the continuity of $s_u$ and these two inequalities we obtain

$$\begin{aligned} \lim _{t\rightarrow 0} \frac{{\widetilde{\Psi }}(u+tv)-{\widetilde{\Psi }}(u)}{t} = \mathcal {J}'(s_uu)[s_uv] = \mathcal {J}'(\mathfrak {{\widetilde{m}}}(u))[s_uv]. \end{aligned}$$

The right-hand side is linear in v and continuous in v and u. Therefore ${\widetilde{\Psi }}$ is of class $\mathcal {C}^1$. If $u\in \mathcal {U}$ and $v\in \mathrm {T}_u(\mathcal {T})$, then $\mathfrak {\widetilde{m}}(u)=\mathfrak {m}(u)$, and the statement is proved.

(ii) :  Note that $\mathcal {H} = \mathrm {T}_u(\mathcal {T}) \oplus (\mathbb {R}u_1,\ldots ,\mathbb {R}u_M)$ for each $u\in \mathcal {U}$. Since $\mathfrak {m}(u)\in \mathcal {N}$, we have that $\mathcal {J}'(\mathfrak {m}(u))w=0$ if $w\in (\mathbb {R}u_1,\ldots ,\mathbb {R}u_M)$. So, from (i) we get

$$\begin{aligned} C_0(\min _i\{s_{u,i}\})\Vert \mathcal {J}'(\mathfrak {m}(u))\Vert _{\mathcal {H}^{-1}}&\le \Vert \Psi '(u)\Vert _* = \sup _{\begin{array}{c} v\in \mathrm {T}_u(\mathcal {T}) \\ v\ne 0 \end{array}}\frac{|\mathcal {J}'(\mathfrak {m}(u))[s_uv]|}{\Vert v\Vert } \\&\le (\max _i\{s_{u,i}\})\Vert \mathcal {J}'(\mathfrak {m}(u))\Vert _{\mathcal {H}^{-1}}. \end{aligned}$$

If $(\Psi (u_n))$ converges, then $(s_{u_n})$ is bounded in $\mathbb {R}^M$ by (3.4). Moreover, by Proposition 3.1(d), this sequence is bounded away from 0. Therefore, $(\mathfrak {m}(u_n))$ is a $(PS)_c$-sequence for $\mathcal {J}$ iff $(u_n)$ is a $(PS)_c$-sequence for $\Psi$, as claimed.

(iii) :  As $\mathcal {J}'(\mathfrak {m}(u))w=0$ if $w\in (\mathbb {R}u_1,\ldots ,\mathbb {R}u_M)$, it follows from (i) that $\Psi '(u)=0$ if and only if $\mathcal {J}'(\mathfrak {m}(u))=0$.

(iv) :  This statement follows from Proposition 3.1(e) and (3.1).

(v) :  Since $-u\in \mathcal {N}$ iff $u\in \mathcal {N}$, we have that $s_u=s_{-u}$. So, as $\mathcal {J}$ is even, $\Psi (-u)=\mathcal {J}(s_{-u}(-u))=\mathcal {J}(s_{u}u)=\Psi (u)$. $\square$

Let Z be a subset of $\mathcal {T}$ such that $-u\in Z$ iff $u\in Z$. If $Z\ne \emptyset$, the genus of Z is the smallest integer $k \ge 1$ such that there exists an odd continuous function $Z \rightarrow \mathbb {S}^{k-1}$ into the unit sphere $\mathbb {S}^{k-1}$ in $\mathbb {R}^k$. We denote it by $\mathrm {genus}(Z)$. If no such k exists, we define $\mathrm {genus}(Z) := \infty$. We set $\mathrm {genus}(\emptyset ):=0$.

As usual, we write

$$\begin{aligned} \Psi ^{\le a}:=\{u\in \mathcal {U}:\Psi (u)\le a\},\qquad K_c:=\{u\in \mathcal {U}:\Psi (u)=c,\;\Vert \Psi '(u)\Vert _*=0\}. \end{aligned}$$

The previous theorem yields the following one.

Theorem 3.4

(a):

If $\inf _\mathcal {N}\mathcal {J}$ is attained by $\mathcal {J}$ at some $u=(u_1,\ldots ,u_M)\in \mathcal {N}$, then u and $|u|:=(|u_1|,\ldots ,|u_M|)$ are fully nontrivial solutions of (1.1).

(b):

If $\Psi :\mathcal {U}\rightarrow \mathbb {R}$ satisfies the $(PS)_c$-condition for every $c\le a$, then the system (1.1) has, either an infinite (in fact, uncountable) set of fully nontrivial solutions with the same norm, or it has at least $\mathrm {genus}(\Psi ^{\le a})$ fully nontrivial solutions with pairwise different norms.

(c):

If $\Psi :\mathcal {U}\rightarrow \mathbb {R}$ satisfies the $(PS)_c$-condition for every $c\in \mathbb {R}$ and $\mathrm {genus}(\mathcal {U})=\infty$, then the system (1.1) has an unbounded sequence of fully nontrivial solutions.

Proof

Theorem 3.3(iii) states that u is a critical point of $\Psi$ iff $\mathfrak {m}(u)$ is a fully nontrivial critical point of $\mathcal {J}$. Note that $\Psi (u)=\frac{p-2}{2p}\Vert \mathfrak {m}(u)\Vert ^2$, by (3.1).

If $\inf _\mathcal {N}\mathcal {J}=\mathcal {J}(u)$ and $u\in \mathcal {N}$, then $\mathfrak {m}^{-1}(u)\in \mathcal {U}$ and $\Psi (\mathfrak {m}^{-1}(u))=\inf _\mathcal {U}\Psi$. So u is a fully nontrivial critical point of $\mathcal {J}$. As $|u|\in \mathcal {N}$ and $\mathcal {J}(|u|)=\mathcal {J}(u)$ the same is true for |u|. This proves (a).

Theorem 3.3(iv) implies that $\mathcal {U}$ is positively invariant under the negative pseudogradient flow of $\Psi$, so the usual deformation lemma holds true for $\Psi$; see, e.g., [18, Section II.3] or [21, Section 5.3]. Set

$$\begin{aligned} c_j:=\inf \{c\in \mathbb {R}:\mathrm {genus}(\Psi ^{\le c})\ge j\}. \end{aligned}$$

Standard arguments show that, under the assumptions of (b), $c_j$ is a critical value of $\Psi$ for every $j=1,\ldots ,\mathrm {genus}(\Psi ^{\le a})$. Moreover, if some of these values coincide, say $c:=c_j=\cdots =c_{j+k}$, then $\mathrm {genus}(K_c)\ge k+1\ge 2$. Hence, $K_c$ is an infinite set; see, e.g., [18, Lemma II.5.6]. On the other hand, under the assumptions of (c), $c_j$ is a critical value for every $j\in \mathbb {N}$, and a well known argument (see, e.g., [16, Proposition 9.33]) shows that $c_j\rightarrow \infty$ as $j\rightarrow \infty$. This completes the proof. $\square$

4 Some applications

4.1 Subcritical systems in exterior domains

Consider the subcritical system (1.2) in an exterior domain $\Omega$. First, we show that this system cannot be solved by minimization. Set

$$\begin{aligned} S_{p,i}:=\inf \limits _{\begin{array}{c} w\in H^1(\mathbb {R}^N) \\ w\ne 0 \end{array}}\frac{\Vert w\Vert _i^2}{|w|_{p,i}^2}, \end{aligned}$$

where   $\Vert w\Vert _i^2:=\int _{\mathbb {R}^N}(|\nabla w|^2+\kappa _iw^2)$   and   $|w|_{p,i}^p:=\int _{\mathbb {R}^N}\mu _i|w|^p.$

Proposition 4.1

We have that

$$\begin{aligned} \inf _{u\in \mathcal {N}}\mathcal {J}(u)=\frac{p-2}{2p}\sum _{i=1}^M S_{p,i}^\frac{p}{p-2} \end{aligned}$$

(4.1)

and this infimum is not attained by $\mathcal {J}$ on $\mathcal {N}$.

Proof

We consider $H^1_0(\Omega )$ to be a subspace of $H^1(\mathbb {R}^N)$, via trivial extension.

If $(u_1,\ldots ,u_M)\in \mathcal {N}$ then, as $\lambda _{ij}<0$ for every $i\ne j$, we have that $\Vert u_i\Vert _i^2\le |u_i|_{p,i}^p$ for all $i=1,\ldots ,M$. Hence,

$$\begin{aligned} S_{p,i}\le \frac{\Vert u_i\Vert _i^2}{|u_i|_{p,i}^2}\le (\Vert u_i\Vert _i^2)^\frac{p-2}{p}. \end{aligned}$$

It follows from (3.1) that $\mathcal {J}(u)\ge \frac{p-2}{2p}\sum _{i=1}^M S_{p,i}^\frac{p}{p-2}$.

To prove the opposite inequality, set $B_r(x):=\{y\in \mathbb {R}^N:|y-x|<r\}$, and let $w_{i,R}$ be a least energy solution to the problem

$$\begin{aligned} -\Delta w + \kappa _iw = \mu _i|w|^{p-2}w,\qquad w\in H_0^1(B_R(0)). \end{aligned}$$

It is easy to verify that $\lim _{R\rightarrow \infty }\Vert w_{i,R}\Vert _i^2=S_{p,i}^\frac{p}{p-2}$. Fix $\xi _{i,R}\in \Omega$, $i=1\ldots ,m$, such that $B_R(\xi _{i,R})\subset \Omega$ and $B_R(\xi _{i,R})\cap B_R(\xi _{j,R})=\emptyset$ if $i\ne j$, and set $u_R:=(u_{1,R},\ldots ,u_{M,R})$ with $u_{i,R}(x):=w_{i,R}(x-\xi _{i,R})$. Then, $u_R\in \mathcal {N}$ and

$$\begin{aligned} \lim _{R\rightarrow \infty }\mathcal {J}(u_R)= \frac{p-2}{2p}\sum _{i=1}^M S_{p,i}^\frac{p}{p-2}. \end{aligned}$$

This completes the proof of (4.1).

To show that the infimum is not attained, we argue by contradiction. Assume that $u =(u_1,\ldots ,u_M)\in \mathcal {N}$ and $\mathcal {J}(u)=\frac{p-2}{2p}\sum _{i=1}^M S_{p,i}^\frac{p}{p-2}$. We may assume that $u_i\ge 0$ for all $i=1,\ldots ,M$. We fix i and consider two cases. If $\int _\Omega u_j^{\alpha _{ij}}u_i^{\beta _{ij}}\ne 0$ for some $j\ne i$, then $\Vert u_i\Vert _i^2<|u_i|_{p,i}^p$ and, hence, $S_{p,i}^{p/(p-2)} < \Vert u_i\Vert _i^2$. This implies that $\mathcal {J}(u)>\frac{p-2}{2p}\sum _{i=1}^M S_{p,i}^{p/(p-2)}$, contradicting our assumption. On the other hand, if $\int _\Omega u_j^{\alpha _{ij}}u_i^{\beta _{ij}}= 0$ for all $j\ne i$, then $\Vert u_i\Vert _i^2=|u_i|_{p,i}^p=S_{p,i}^{p/(p-2)}$. Hence, $u_i$ is a nontrivial solution to the problem

$$\begin{aligned} -\Delta w + \kappa _iw = \mu _i|w|^{p-2}w,\qquad w\in H_0^1(\mathbb {R}^N). \end{aligned}$$

Moreover, $\int _\Omega u_j^{\alpha _{ij}}u_i^{\beta _{ij}}= 0$ also implies that $u_j^{\alpha _{ij}}u_i^{\beta _{ij}}= 0$ a.e. in $\Omega$. As $u_j\not \equiv 0$ for all j, we have that $u_i=0$ in some subset of positive measure of $\mathbb {R}^N$. This contradicts the maximum principle. $\square$

To obtain multiple solutions to the system (1.2) we introduce some symmetries.

Let G be a closed subgroup of O(N) and $Gx:=\{gx:g\in G\}$. Set $\mathbb {S}^{N-1}:=\{x\in \mathbb {R}^N:|x|=1\}$. We start with the following lemma.

Lemma 4.2

If $\dim (Gx)>0$ for every $x\in \mathbb {R}^N\smallsetminus \{0\}$, then, for each $k\in \mathbb {N}$, there exists $d_k>0$ such that, for every $x\in \mathbb {S}^{N-1}$, there exist $g_1,\ldots ,g_k\in G$ with

$$\begin{aligned} \min _{i\ne j}|g_ix-g_jx|\ge d_k. \end{aligned}$$

Proof

Arguing by contradiction, assume that for some $k\in \mathbb {N}$ and every $n\in \mathbb {N}$ there exists $x_n\in \mathbb {S}^{N-1}$ such that

$$\begin{aligned} \min _{i\ne j}|g_ix_n-g_jx_n|<\frac{1}{n}\qquad \text {for any }k\text { elements }\,g_1,\ldots ,g_k\in G. \end{aligned}$$

After passing to a subsequence, we have that $x_n\rightarrow x$ in $\mathbb {S}^{N-1}$. Since $\dim (Gx)>0$, there exist $\bar{g}_1,\ldots ,\bar{g}_k\in G$ such that $\bar{g}_ix\ne \bar{g}_jx$ if $i\ne j$. Fix $i\ne j$ such that, after passing to a subsequence, $|\bar{g}_ix_n-\bar{g}_jx_n|=\min _{i\ne j}|\bar{g}_ix_n-\bar{g}_jx_n|$ for every $n\in \mathbb {N}$. Then,

$$\begin{aligned} 0<\min _{i\ne j}|\bar{g}_ix-\bar{g}_jx|\le |\bar{g}_ix-\bar{g}_jx|=\lim _{n\rightarrow \infty }|\bar{g}_ix_n-\bar{g}_jx_n|=0. \end{aligned}$$

This is a contradiction. $\square$

We assume that $\Omega$ is G-invariant and define

$$\begin{aligned} H^1_0(\Omega )^G:=\{v\in H^1_0(\Omega ):v\text { is }G\text {-invariant}\}\qquad \text {and}\qquad \mathcal {H}^G:=(H^1_0(\Omega )^G)^M. \end{aligned}$$

Recall that $\Omega$ is called G-invariant if $Gx\subset \Omega$ for all $x\in \Omega$, and a function $v:\Omega \rightarrow \mathbb {R}$ is G-invariant if it is constant on Gx for every $x\in \Omega$. An M-tuple $(v_1,\ldots ,v_M)$ will be called G-invariant if each component $v_i$ is G-invariant.

Lemma 4.3

Assume that $\dim (Gx)>0$ for every $x\in \mathbb {R}^N\smallsetminus \{0\}$ and let $\Omega$ be a G-invariant exterior domain. Then, the embedding $H^1_0(\Omega )^G\hookrightarrow L^p(\Omega )$ is compact for every $p\in (2,2^*)$.

Proof

Let $(w_n)$ be a bounded sequence in $H_{0}^{1}(\Omega )^{G}$. Then, after passing to a subsequence, $w_n\rightharpoonup w$ weakly in $H_{0}^{1}(\Omega )^{G}$. Set $v_n:=w_n-w$. A subsequence of $(v_n)$ satisfies $v_n\rightharpoonup 0$ weakly in $H_{0}^{1}(\Omega )^{G}$, $v_n\rightarrow 0$ in $L_{loc}^{2}(\Omega )$ and $v_n(x)\rightarrow 0$ a.e. in $\Omega$. We claim that

$$\begin{aligned} \sup _{x\in \mathbb {R}^{N}}\int _{B_1(x)}v_{n}^{2}\rightarrow 0\qquad \text {as }\;n\rightarrow \infty . \end{aligned}$$

(4.2)

To prove this claim, let $\varepsilon >0$, and let $C>0$ be such that $\Vert v_n\Vert ^2\le C$ for all $n\in \mathbb {N}$, where $\Vert \cdot \Vert$ is the standard norm in $H_{0}^{1}(\Omega )$. We choose $k\in \mathbb {N}$ such that $C<\varepsilon k$ and $d_k>0$ as in Lemma 4.2, and we fix $R_k>2/d_k$. We consider two cases.

Assume first that $|x|\ge R_k$. By Lemma 4.2, there exist $g_1,\ldots ,g_k\in G$ such that

$$\begin{aligned} |g_ix-g_jx|\ge |x|d_k\quad \text {for all } i\ne j. \end{aligned}$$

Since $|x|\ge R_k$, we have that $|g_ix-g_jx|> 2$. Hence, $B_1(g_ix)\cap B_1(g_jx)=\emptyset$ if $i\ne j$ and, as $v_n$ is G-invariant, we obtain

$$\begin{aligned} k\int _{B_1(x)}v_{n}^{2}=\sum \limits _{i=1}^k\int _{B_1(g_ix)}v_n^2\le \int _{\Omega }v_n^2\le \Vert v_n\Vert ^2\le C\qquad \text {for all }n\in \mathbb {N}. \end{aligned}$$

Therefore,

$$\begin{aligned} \int _{B_1(x)}v_n^2<\varepsilon \qquad \text {for all }n\in \mathbb {N}\text { and all }|x|\ge R_k. \end{aligned}$$

(4.3)

Now assume that $|x|\le R_k$. Then, since $v_n\rightarrow 0$ strongly in $L^2(B_{R_k+1}(0))$, there exists $n_0\in \mathbb {N}$ such that

$$\begin{aligned} \int _{B_1(x)}v_n^2\le \int _{B_{R_k+1}(0)}v_n^2<\varepsilon \qquad \text {for all }n\ge n_{0}. \end{aligned}$$

(4.4)

Inequalities (4.3) and (4.4) yield (4.2). Applying Lions’ lemma [21, Lemma 1.21] we conclude that $v_n\rightarrow 0$ strongly in $L^p(\Omega )$ for any $p\in (2,2^*)$. $\square$

Lemma 4.4

Assume that $\dim (Gx)>0$ for every $x\in \mathbb {R}^N\smallsetminus \{0\}$ and let $\Omega$ be a G-invariant exterior domain. Then, the functional $\mathcal {J}$ satisfies the Palais-Smale condition in $\mathcal {H}^G$, i.e., every sequence $(u_n)$ in $\mathcal {H}^G$ such that $\mathcal {J}(u_n)\rightarrow c$ and $\mathcal {J}'(u_n)\rightarrow 0$ in $(\mathcal {H}^G)'$, contains a convergent subsequence.

Proof

Since

$$\begin{aligned} \frac{p-2}{p}\Vert u_n\Vert ^2 = \mathcal {J}(u_n)-\mathcal {J}'(u_n)u_n \le c_1+c_2\Vert u_n\Vert , \end{aligned}$$

$(u_n)$ is bounded. The rest of the proof follows from Lemma 4.3 by standard arguments. $\square$

Lemma 4.5

Let $\mathcal {U}^G:=\mathcal {U}\cap \mathcal {H}^G$. Then, $\mathrm {genus}(\mathcal {U}^G)=\infty$.

Proof

Given $k\ge 1$, for each $j=1,\ldots ,k$, $i=1,\ldots ,M$, we choose $u_{j,i}\in H_0^1(\Omega )^G$ such that $\Vert u_{j,i}\Vert _i=1$ and $\mathrm {supp}(u_{j,i})\cap \mathrm {supp}(u_{j',i'})=\emptyset$ if $(j,i)\ne (j',i')$.

Let $\{ e_j : 1 \le j \le k\}$ be the canonical basis of $\mathbb {R}^k$, and Q be the set

$$\begin{aligned} Q := \left\{ \sum _{j=1}^k r_j \hat{e}_j : \hat{e}_j \in \{\pm e_j\}, \, r_j \in [0,1], \, \sum _{j=1}^k r_j = 1 \right\} . \end{aligned}$$

Note that Q is homeomorphic to the unit sphere $\mathbb {S}^{k-1}$ in $\mathbb {R}^k$ by an odd homeomorphism.

For each $i=1,\ldots ,M$, define $\sigma _i:Q\rightarrow H_0^1(\Omega )^G$ by setting $\sigma _i(e_j) := u_{j,i}$, $\sigma _i(-e_j):= -u_{j,i}$, and

$$\begin{aligned} \sigma _i\left( \sum _{j=1}^k r_j \hat{e}_j \right) := \frac{\sum _{j=1}^k r_j \sigma _i(\hat{e}_j)}{\Vert \sum _{j=1}^k r_j \sigma _i(\hat{e}_j)\Vert _i}. \end{aligned}$$

Note that, since $u_{j,i}$ and $u_{j',i'}$ have disjoint supports if $(j,i)\ne (j',i')$, these maps are well defined and $\mathrm {supp}(\sigma _i(z))\cap \mathrm {supp}(\sigma _{i'}(z))=\emptyset$ if $i\ne i'$ for every $z\in Q$. So, by Proposition 3.1(a), the map $\sigma : Q \rightarrow \mathcal {U}^G$ given by $\sigma (z):=(\sigma _1(z),\ldots ,\sigma _M(z))$ is well defined. As each $\sigma _i$ is continuous and odd, so is $\sigma$. Hence, $\mathrm {genus}(\mathcal {U}^G) \ge \mathrm {genus}(Q)=k$. $\square$

Proof of Theorem 1.1

The functional $\mathcal {J}$ is G-invariant, so, by the principle of symmetric criticality, the critical points of the restriction of $\mathcal {J}$ to $\mathcal {H}^G$ are the G-invariant critical points of $\mathcal {J}$; see, e.g., [21, Theorem 1.28].

It is readily seen that the results of Sect. 3 are also true for $H:=H_0^1(\Omega )^G$. Theorem 3.3(ii) and Lemma 4.4 imply that $\Psi$ satisfies the $(PS)_c$-condition for every $c\in \mathbb {R}$. This, together with Lemma  4.5 and Theorem 3.4, yields Theorem 1.1. $\square$

4.2 Entire solutions to critical systems

Next, we consider the Yamabe system (1.3).

As usual, we denote

$$\begin{aligned} S:=\inf \limits _{\begin{array}{c} w\in D^{1,2}(\mathbb {R}^N) \\ w\ne 0 \end{array}}\frac{\Vert w\Vert ^2}{|w|_{2^*}^2}, \end{aligned}$$

where   $\Vert w\Vert ^2:=\int _{\mathbb {R}^N}|\nabla w|^2$   and   $|w|_{2^*}^{2^*}:=\int _{\mathbb {R}^N}|w|^{2^*}$. The next result says that the system (1.3) cannot be solved by minimization.

Proposition 4.6

We have that

$$\begin{aligned} \inf _{u\in \mathcal {N}}\mathcal {J}(u)=\frac{1}{N}\sum _{i=1}^M \mu _i^{-\frac{N-2}{2}}S^\frac{N}{2} \end{aligned}$$

and this infimum is not attained by $\mathcal {J}$ on $\mathcal {N}$.

Proof

Following the argument given in [6, Proposition 2.2] for $M=2$ one can easily prove this statement. $\square$

To obtain multiple solutions to the system (1.3) we consider a conformal action on $\mathbb {R}^N$, as in [6, 8].

Let $\Gamma =O(m)\times O(n)$ with $m+n=N+1$ and $m,n\ge 2$ act on $\mathbb {R}^{N+1}\equiv \mathbb {R}^m\times \mathbb {R}^n$ in the obvious way. Then, $\Gamma$ acts isometrically on the unit sphere $\mathbb {S}^N:=\{x\in \mathbb {R}^{N+1}:|x|=1\}$. The stereographic projection $\sigma :\mathbb {S}^N\rightarrow \mathbb {R}^N\cup \{\infty \}$, which maps the north pole $(0,\ldots ,0,1)$ to $\infty$, induces a conformal action of $\Gamma$ on $\mathbb {R}^N$, given by

$$\begin{aligned} (\gamma ,x)\mapsto \widetilde{\gamma }x, \qquad \text {where }\;\widetilde{\gamma }:=\sigma \circ \gamma ^{-1}\circ \sigma ^{-1}:\mathbb {R}^N\rightarrow \mathbb {R}^N. \end{aligned}$$

(4.5)

Note that the map $\widetilde{\gamma }$ is well defined except at a single point.

The group $\Gamma$ acts on the Sobolev space $D^{1,2}(\mathbb {R}^N)$ by linear isometries as follows:

$$\begin{aligned} \gamma w:=|\det \widetilde{\gamma }'|^{1/2^*}w\circ \widetilde{\gamma },\qquad \text {for any }\gamma \in \Gamma \;\text { and }\;w\in D^{1,2}(\mathbb {R}^N); \end{aligned}$$

see [6, Section 3]. We shall say that w is $\Gamma$-invariant if $\gamma w=w$ for all $\gamma \in \Gamma$, and that $(u_1,\ldots ,u_M)$ is $\Gamma$-invariant if each $u_i$ is $\Gamma$-invariant. We set

$$\begin{aligned} D^{1,2}(\mathbb {R}^N)^\Gamma :=\{w\in D^{1,2}(\mathbb {R}^N):w\text { is }\Gamma \text {-invariant}\},\qquad \mathcal {H}^\Gamma :=(D^{1,2}(\mathbb {R}^N)^\Gamma )^M. \end{aligned}$$

One has the following results.

Lemma 4.7

The embedding $D^{1,2}(\mathbb {R}^N)^\Gamma \hookrightarrow L^{2^*}(\mathbb {R}^N)$ is compact.

Proof

This follows from Proposition 3.3 and Example 3.4(1) in [6]. $\square$

Lemma 4.8

The functional $\mathcal {J}$ satisfies the Palais-Smale condition in $\mathcal {H}^\Gamma$.

Proof

The proof follows from Lemma 4.7 by standard arguments (boundedness of Palais-Smale sequences is proved as in Lemma 4.4). $\square$

Lemma 4.9

Let $\mathcal {U}^\Gamma :=\mathcal {U}\cap \mathcal {H}^\Gamma$. Then, $\mathrm {genus}(\mathcal {U}^\Gamma )=\infty$.

Proof

The proof is the same as that of Lemma 4.5. $\square$

Proof of Theorem 1.2

The functional $\mathcal {J}$ is $\Gamma$-invariant; see [6, Section 3]. Thus, the critical points of the restriction of $\mathcal {J}$ to $\mathcal {H}^\Gamma$ are the $\Gamma$-invariant critical points of $\mathcal {J}$.

The results of Sect. 3 hold true for $H=D^{1,2}(\mathbb {R}^N)^\Gamma$. Theorem 3.3(ii) and Lemma 4.8 imply that $\Psi$ satisfies the $(PS)_c$-condition for every $c\in \mathbb {R}$. This, together with Lemma 4.9 and Theorem 3.4, yields Theorem 1.2.

4.3 Brezis–Nirenberg systems

Finally, we consider the Brezis-Nirenberg type system (1.4).

For each $I\subset \{1,\ldots ,M\}$, let $(\mathscr {S}_I)$ be the system of $M-|I|$ equations obtained by replacing $\kappa _i, \mu _i, \lambda _{ij},\lambda _{ji}$ with 0 if $i\in I$, where |I| is the cardinality of I, i.e.,

$$\begin{aligned} (\mathscr {S}_I)\qquad {\left\{ \begin{array}{ll} -\Delta u_i + \kappa _i u = \mu _i|u_i|^{2^*-2}u_i + \sum \limits _{j\ne i} \lambda _{ij}\beta _{ij}|u_j|^{\alpha _{ij}}|u_i|^{\beta _{ij}-2}u_i, \\ u_i\in D^{1,2}_0(\Omega ),\qquad i,j\in \{1,\ldots ,M\}\smallsetminus I. \end{array}\right. } \end{aligned}$$

The fully nontrivial solutions of $(\mathscr {S}_I)$ correspond to the solutions $(u_1,\ldots ,u_M)$ of (1.4) which satisfy $u_i=0$ iff $i\in I$. We set

$$\begin{aligned} c_I:=\inf \{\mathcal {J}(u):u=(u_1,\ldots ,u_M)\text { solves }(1.4)\text { and }u_i=0\text { iff }i\in I\}. \end{aligned}$$

Lemma 4.10

If

$$\begin{aligned} c_0:=\inf _{u\in \mathcal {N}}\mathcal {J}(u)<\min \left\{ c_I + \frac{1}{N}\sum _{i\in I} \mu _i^{-\frac{N-2}{2}}S^\frac{N}{2}:\emptyset \ne I\subset \{1,\ldots ,M\}\right\} , \end{aligned}$$

then this infimum is attained by $\mathcal {J}$ on $\mathcal {N}$.

Proof

Note that $\inf _{v\in \mathcal {N}}\mathcal {J}(v)=\inf _{v\in \mathcal {U}}\Psi (v)$. So, by Ekeland’s variational principle [21, Theorem 8.5] and Theorem 3.3, there exists a sequence $(u_n)$ in $\mathcal {N}$ such that $\mathcal {J}(u_n)\rightarrow c_0$ and $\mathcal {J}'(u_n)\rightarrow 0$. It follows from (3.1) that $(u_n)$ is bounded in $\mathcal {H}:=(D^{1,2}_0(\Omega ))^M$. So, after passing to a subsequence, $u_n\rightharpoonup u$ weakly in $\mathcal {H}$, $u_n\rightarrow u$ strongly in $L^2(\Omega )$ and $u_n\rightarrow u$ a.e. in $\Omega$. A standard argument shows that u is a solution to the system (1.4). We claim that u is fully nontrivial.

Arguing by contradiction, assume that some components of u are trivial. Let $I:=\{i\in \{1,\ldots ,M\}:u_i=0\}$. Then, for each $i\in I$, we have that $u_{n,i}\rightarrow 0$ strongly in $L^2(\Omega )$. As $u_n\in \mathcal {N}$ and $\lambda _{ij}<0$, we get that

$$\begin{aligned} \Vert u_{n,i}\Vert ^2 + o(1) = \Vert u_{n,i}\Vert ^2_i \le \mu _i|u_{n,i}|_{2^*}^{2^*}, \end{aligned}$$

where

$$\begin{aligned} \Vert w\Vert ^2:=\int _{\Omega }|\nabla w|^2,\qquad |w|_p^p:=\int _{\Omega }|w|^p,\qquad \Vert w\Vert ^2_i:=\Vert w\Vert ^2+\kappa _i|w|_2^2. \end{aligned}$$

Hence,

$$\begin{aligned} S\le \frac{\Vert u_{n,i}\Vert ^2}{|u_{n,i}|_{2^*}^2}=\frac{\Vert u_{n,i}\Vert _i^2}{|u_{n,i}|_{2^*}^2}+o(1)\le \mu _i^\frac{N-2}{N}(\Vert u_{n,i}\Vert _i^2)^\frac{2}{N} + o(1), \end{aligned}$$

i.e.,  $\mu _i^{-\frac{N-2}{2}}S^\frac{N}{2}\le \Vert u_{n,i}\Vert _i^2 + o(1)$  for every $i\in I$. As u solves (1.4), we obtain

$$\begin{aligned} c_0&=\lim _{n\rightarrow \infty }\mathcal {J}(u_n)=\lim _{n\rightarrow \infty }\frac{1}{N}\left( \sum _{i\not \in I}\Vert u_{n,i}\Vert _i^2 + \sum _{i\in I}\Vert u_{n,i}\Vert _i^2\right) \\&\ge \liminf _{n\rightarrow \infty }\frac{1}{N}\sum _{i\not \in I}\Vert u_{n,i}\Vert _i^2 + \frac{1}{N}\sum _{i\in I}\mu _i^{-\frac{N-2}{2}}S^\frac{N}{2}\\&\ge \frac{1}{N}\sum _{i\not \in I}\Vert u_i\Vert _i^2 + \frac{1}{N}\sum _{i\in I}\mu _i^{-\frac{N-2}{2}}S^\frac{N}{2} = \mathcal {J}(u)+ \frac{1}{N}\sum _{i\in I}\mu _i^{-\frac{N-2}{2}}S^\frac{N}{2}\\&\ge c_I + \frac{1}{N}\sum _{i\in I}\mu _i^{-\frac{N-2}{2}}S^\frac{N}{2}. \end{aligned}$$

This contradicts our assumption.

Therefore, u is fully nontrivial. This implies that $u\in \mathcal {N}$, and (3.1) yields

$$\begin{aligned} c_0\le \mathcal {J}(u) \le \liminf _{n\rightarrow \infty }\mathcal {J}(u_n)=c_0. \end{aligned}$$

Hence, $\mathcal {J}(u)=c_0$, as claimed. $\square$

Lemma 4.11

Let $N\ge 4$. Assume that $\min \{\alpha _{ij},\beta _{ij}\}\ge \frac{4}{3}$ if $N=5$ and $\alpha _{ij}=\beta _{ij}=2$ if $N=4$, for all $i,j=1,\ldots ,M$. Then

$$\begin{aligned} \inf _{u\in \mathcal {N}}\mathcal {J}(u)<\min \left\{ c_I + \frac{1}{N}\sum _{i\in I} \mu _i^{-\frac{N-2}{2}}S^\frac{N}{2}:\emptyset \ne I\subset \{1,\ldots ,M\}\right\} . \end{aligned}$$

(4.6)

Proof

We prove this statement by induction on M.

If $M=1$ the system reduces to the single equation

$$\begin{aligned} -\Delta u + \kappa _1 u = \mu _1|u|^{2^*-2}u,\qquad u\in D_0^{1,2}(\Omega ), \end{aligned}$$

and the statement was proved by Brezis and Nirenberg in [1].

Assume that the statement is true for every system $(\mathscr {S}_I)$ with $|I|\ge 1$ (i.e., for every system of $M-1$ equations). Then, the right-hand side of (4.6) reduces to

$$\begin{aligned} \min \left\{ c_I + \frac{1}{N}\sum _{i\in I} \mu _i^{-\frac{N-2}{2}}S^\frac{N}{2}:|I|=1\right\} . \end{aligned}$$

Without loss of generality, we may assume that $I=\{M\}$. By Lemma 4.10 and our induction hypothesis, there exists a positive, least energy, fully nontrivial solution $(u_1,\ldots ,u_{M-1})$ to the system $(\mathscr {S}_I)$. Fix $\xi \in \Omega$ and $\varrho \in (0,1)$ such that $B_\varrho (\xi )\subset \Omega$, and a cut-off function $\chi \in \mathcal {C}^\infty _c(B_\varrho (\xi ))$ such that $0\le \chi \le 1$ and $\chi \equiv 1$ in $B_{\varrho /2}(\xi )$. Set

$$\begin{aligned} w_\varepsilon (x):=\chi (x)\mu _M^\frac{2-N}{4} U_{\varepsilon ,\xi }, \end{aligned}$$

where

$$\begin{aligned} U_{\varepsilon ,\xi }:=a_N\left( \frac{\varepsilon }{\varepsilon ^2+|x-\xi |^2}\right) ^\frac{N-2}{2},\qquad \text {with }\;a_N=(N(N-2))^\frac{N-2}{4}. \end{aligned}$$

It is shown in [1] that

$$\begin{aligned} \Vert w_\varepsilon \Vert ^2&=\mu _M^\frac{2-N}{2}S^\frac{N}{2}+O(\varepsilon ^{N-2}),\;\qquad |w_\varepsilon |^{2^*}_{2^*}=\mu _M^{-\frac{N}{2}}S^\frac{N}{2}+O(\varepsilon ^N), \end{aligned}$$

(4.7)

$$\begin{aligned} |w_\varepsilon |_2^2&\ge {\left\{ \begin{array}{ll} d_0\varepsilon ^2+O(\varepsilon ^{N-2})&{}\quad \text {if } N\ge 5,\\ d_0\varepsilon ^2|\ln \varepsilon |+O(\varepsilon ^2)&{}\quad \text {if } N=4, \end{array}\right. } \end{aligned}$$

(4.8)

for some $d_0>0$; see also [21, Lemma 1.46]. Inspecting the proofs in [1, 21] one sees that $d_0$ may be chosen independently of $\varrho$. Moreover, if $\alpha ,\beta >1$ and $\alpha +\beta =2^*$, we have that

$$\begin{aligned} |w_\varepsilon |_\beta ^\beta \le c_1\int _{B_\varrho (0)}\left( \frac{\varepsilon }{\varepsilon ^2+|x|^2}\right) ^{\frac{N-2}{2}\beta }\mathrm {d}x\le {\left\{ \begin{array}{ll} d_1\varepsilon ^{\frac{N-2}{2}\beta } &{}\;\text {if }\beta \ne \frac{2^*}{2},\\ d_1\varepsilon ^\frac{N}{2}(1+|\ln \varepsilon |) &{}\;\text {if }\beta =\frac{2^*}{2}, \end{array}\right. } \end{aligned}$$

(4.9)

for some $c_1, d_1>0$ and $\varepsilon$ small enough. By a regularity result in [18, Appendix B], $u_i\in \mathcal {C}^0({\overline{\Omega }})$ and we get that

$$\begin{aligned} \int _\Omega |w_\varepsilon |^\alpha |u_i|^\beta \le \left( \max _{x\in {\bar{\Omega }}}|u_i(x)|^\beta \right) \int _\Omega |w_\varepsilon |^\alpha \rightarrow 0\quad \text {as }\varepsilon \rightarrow 0. \end{aligned}$$

Hence, there exists $\varepsilon _0>0$ such that, for every $\varepsilon \in (0,\varepsilon _0)$,

$$\begin{aligned}&\mu _i|u_i|_{2^*}^{2^*}+\sum _{j\ne i}\beta _{ij}\,\lambda _{ij}\int _\Omega |u_j|^{\alpha _{ij}}|u_i|^{\beta _{ij}} + \beta _{iM}\,\lambda _{iM}\int _\Omega |w_\varepsilon |^{\alpha _{iM}}|u_i|^{\beta _{iM}}\\&\quad =\Vert u_i\Vert _i^2 + \beta _{iM}\,\lambda _{iM}\int _\Omega |w_\varepsilon |^{\alpha _{iM}}|u_i|^{\beta _{iM}}>0,\qquad i,j=1,\ldots ,M-1, \\&\quad \mu _M|w_\varepsilon |_{2^*}^{2^*}+\sum _{j=1}^{M-1}\beta _{Mj}\,\lambda _{Mj}\int _\Omega |u_j|^{\alpha _{Mj}}|w_\varepsilon |^{\beta _{Mj}} > 0. \end{aligned}$$

Therefore we may use Lemma 2.1 in order to obtain $0<r<R<\infty$ and $s_{\varepsilon ,1},\ldots ,s_{\varepsilon , M}\in [r,R]$ such that

$$\begin{aligned} u_\varepsilon =(s_{\varepsilon ,1}u_1,\ldots ,s_{\varepsilon ,M-1}u_{M-1},s_{\varepsilon ,M}w_\varepsilon )\in \mathcal {N}. \end{aligned}$$

As $(u_1,\ldots ,u_{M-1})$ is a least energy solution to the system $(\mathscr {S}_I)$, from (3.2) and estimates (4.7) we obtain

$$\begin{aligned} \mathcal {J}(u_\varepsilon )&= \frac{1}{2}\sum _{i=1}^{M-1}s_{\varepsilon ,i}^2\Vert u_i\Vert _i^2 - \frac{1}{2^*}\sum _{i=1}^{M-1}s_{\varepsilon ,i}^{2^*}\mu _i|u_i|^{2^*}_{2^*} \\&\quad - \frac{1}{2}\sum _{\begin{array}{c} j\ne i\\ i,j\ne M \end{array}}s_{\varepsilon ,j}^{\alpha _{ij}} s_{\varepsilon ,i}^{\beta _{ij}}\lambda _{ij}\int _\Omega |u_j|^{\alpha _{ij}}|u_i|^{\beta _{ij}}+\frac{1}{2}s_{\varepsilon ,M}^2\Vert w_\varepsilon \Vert ^2 - \frac{1}{2^*}s_{\varepsilon ,M}^{2^*}\mu _M|w_\varepsilon |^{2^*}_{2^*} \\&\quad +\frac{1}{2}s_{\varepsilon ,M}^2\kappa _M|w_\varepsilon |_2^2 - \sum _{i=1}^{M-1}s_{\varepsilon ,M}^{\alpha _{iM}} s_{\varepsilon ,i}^{\beta _{iM}}\lambda _{iM}\int _\Omega |w_\varepsilon |^{\alpha _{iM}}|u_i|^{\beta _{iM}} \\&\le c_I + \frac{1}{N}\mu _M^{-\frac{N-2}{2}}S^\frac{N}{2} + O(\varepsilon ^{N-2})\\&\quad - \frac{1}{2}r^2|\kappa _M||w_\varepsilon |_2^2 + \sum _{i=1}^M R^{2^*}|\lambda _{iM}|\int _\Omega |w_\varepsilon |^{\alpha _{iM}}|u_i|^{\beta _{iM}}. \end{aligned}$$

So, if either $N\ge 6$, or $N=5$ and $\min \{\alpha _{ij},\beta _{ij}\}>\frac{4}{3}$ for all $i,j=1,\ldots ,M$, we derive from (4.8) and (4.9) that, for $\varepsilon$ small enough,

$$\begin{aligned} - \frac{1}{2}r^2|\kappa _M||w_\varepsilon |_2^2 + \sum _{i=1}^M R^{2^*}|\lambda _{iM}|\int _\Omega |w_\varepsilon |^{\alpha _{iM}}|u_i|^{\beta _{iM}} \le - C\varepsilon ^2 + o(\varepsilon ^2) \end{aligned}$$

and (4.6) follows. In the remaining cases we need to be careful when selecting $\xi$ and $\varrho$. If $N=4$ and $\alpha _{ij}=\beta _{ij}=2$ for all $i,j=1,\ldots ,M$, we choose them in such a way that

$$\begin{aligned} \max _{x\in B_\varrho (\xi )}|u_i(x)|^2\le \frac{r^2|\kappa _M|}{4M R^4|\lambda _{iM}|}\qquad \text {for every } i=1,\ldots ,M-1. \end{aligned}$$

This can be done because $u_i=0$ on $\partial \Omega$. Then, from (4.8) we get

$$\begin{aligned} - \frac{1}{2}r^2|\kappa _M||w_\varepsilon |_2^2 + \sum _{i=1}^M R^4|\lambda _{iM}|\int _\Omega |w_\varepsilon |^2|u_i|^2&\le - \frac{1}{4}r^2|\kappa _M||w_\varepsilon |_2^2 \\&\le -C\varepsilon ^2|\ln \varepsilon |+O(\varepsilon ^2). \end{aligned}$$

If $N=5$ and $\min \{\alpha _{ij},\beta _{ij}\}=\frac{4}{3}$ for some pairs $i,j=1,\ldots ,M$ we argue in a similar way. Hence, in all cases,

$$\begin{aligned} \inf _{u\in \mathcal {N}}\mathcal {J}(u)\le \mathcal {J}(u_\varepsilon )< c_I + \frac{1}{N}\mu _M^{-\frac{N-2}{2}}S^\frac{N}{2}, \end{aligned}$$

for $\varepsilon$ small enough, as claimed. $\square$

Proof of Theorem 1.3

The result follows from Lemmas 4.10, 4.11 and Theorem 3.4(a).

Remark 4.12

If $N=5$ and $\min \{\alpha _{ij},\beta _{ij}\}>\frac{4}{3}$ or $N\ge 6$, then the condition $\partial \Omega \in \mathcal {C}^2$ is not necessary because $u_i\in \mathcal {C}^0(\Omega )$ according to the results in [18, Appendix B] and we may choose any $\xi ,\varrho$ such that $\overline{B}_\varrho (\xi )\subset \Omega$.

Remark 4.13

In Proposition 3.2 we have shown that if all $|\lambda _{ij}|$ are large enough, then there are no synchronized solutions. In some cases we can show more. Suppose $\kappa _1,\ldots ,\kappa _M$ are linearly independent. We claim that so are the positive solutions $u_1,\ldots ,u_M$ to (1.4) (so in particular, they are not synchronized). Indeed, re-writing (1.4) as

$$\begin{aligned} -\Delta u_i - \left( \sum \limits _{j\ne i} \lambda _{ij}\beta _{ij}u_j^{\alpha _{ij}}u_i^{\beta _{ij}-2}\right) u_i= - \kappa _i u_i + \mu _iu_i^{2^*-1}, \end{aligned}$$

we see from the maximum principle and the fact that $\lambda _{ij},\kappa _i<0$ that $u_i(x)>0$ for all $x\in \Omega$ and all i. Suppose $\sum _{i=1}^mt_iu_i=0$. Then $\sum _{i=1}^M t_i\Delta u_i = 0$, so

$$\begin{aligned} \sum _{i=1}^Mt_i\left( -\kappa _i+\mu _iu_i^{2^*-2} +\sum _{j\ne i}\lambda _{ij}\beta _{ij}u_j^{\alpha _{ij}}u_i^{\beta _{ij}-2}\right) = 0. \end{aligned}$$

Letting $(x_k)\subset \Omega$ and $x_k\rightarrow x\in \partial \Omega$ we obtain $\sum _{i=1}^Mt_i\kappa _i=0$ and therefore $t_1=\cdots =t_m=0$. A variant of this argument shows that the same is true for (1.2).