A Sobolev-type inequality for the curl operator and ground states for the curl-curl equation with critical Sobolev exponent

Let $\Omega\subset \mathbb{R}^3$ be a Lipschitz domain and let $S_\mathrm{curl}(\Omega)$ be the largest constant such that $$ \int_{\mathbb{R}^3}|\nabla\times u|^2\, dx\geq S_{\mathrm{curl}}(\Omega) \inf_{\substack{w\in W_0^6(\mathrm{curl};\mathbb{R}^3)\\ \nabla\times w=0}}\Big(\int_{\mathbb{R}^3}|u+w|^6\,dx\Big)^{\frac13} $$ for any $u$ in $W_0^6(\mathrm{curl};\Omega)\subset W_0^6(\mathrm{curl};\mathbb{R}^3)$ where $W_0^6(\mathrm{curl};\Omega)$ is the closure of $\mathcal{C}_0^{\infty}(\Omega,\mathbb{R}^3)$ in $\{u\in L^6(\Omega,\mathbb{R}^3): \nabla\times u\in L^2(\Omega,\mathbb{R}^3)\}$ with respect to the norm $(|u|_6^2+|\nabla\times u|_2^2)^{1/2}$. We show that $S_{\mathrm{curl}}(\Omega)$ is strictly larger than the classical Sobolev constant $S$ in $\mathbb{R}^3$. Moreover, $S_{\mathrm{curl}}(\Omega)$ is independent of $\Omega$ and is attained by a ground state solution to the curl-curl problem $$ \nabla\times (\nabla\times u) = |u|^4u $$ if $\Omega=\mathbb{R}^3$. With the aid of those results, we also investigate ground states of the Brezis-Nirenberg-type problem for the curl-curl operator in a bounded domain $\Omega$ $$\nabla\times (\nabla\times u) +\lambda u = |u|^4u\quad\hbox{in }\Omega$$ with the so-called metallic boundary condition $\nu\times u=0$ on $\partial\Omega$, where $\nu$ is the exterior normal to $\partial\Omega$.


Introduction
Sobolev-type inequalities have been widely studied by a large number of authors and the best Sobolev constants play an important role e.g. in the theory of partial differential equations, differential geometry, isoperimetric inequalities as well as in mathematical physics, see e.g. [4,20,33]. In particular, if Ω is a domain in R 3 , then the best constant S in the Sobolev inequality (1.1) Ω |∇u| 2 dx ≥ S Ω |u| 6 dx 1 3 for u ∈ D 1,2 (Ω) has been computed explicitly by Talenti [33] and as is well-known, it is achieved (i.e., equality holds) if and only if Ω = R 3 and u is the Aubin-Talenti instanton U ε,y (x) := 3 1/4 (ε 2 + |x − y| 2 ) −1/2 , see [4,33]. When ε = 1, this is the unique (up to translations in R 3 ) positive solution to the equation −∆u = |u| 4 u in D 1,2 (R 3 ) and a ground state, i.e. a minimizer for the energy functional among all nontrivial solutions.
The aim of this work is to perform a similar analysis for the curl operator ∇ × (·). This is challenging from the mathematical point of view and important in mathematical physics; such operator appears e.g. in Maxwell equations as well as in Navier-Stokes problems [13,17,26]. Finding a formulation in the spirit of (1.1), but involving the curl operator, is not straightforward and there are several essential difficulties as we shall see later.
For instance, the kernel of ∇ × (·) is of infinite dimension since ∇ × (∇ϕ) = 0 for all ϕ ∈ C 2 (Ω). Hence the inequality (1.1) with ∇u replaced by ∇ × u would hold for all u ∈ C ∞ 0 (R 3 , R 3 ) only if S = 0. This makes it necessary to introduce a Sobolev-like constant in a different way which we now proceed to do.
Let Ω be a Lipschitz domain in R 3 and for 2 ≤ p ≤ 6, let Here and in the sequel | · | q denotes the L q -norm for q ∈ [1, ∞]. We also define (1.2) W p 0 (curl; Ω) := closure of C ∞ 0 (Ω, R 3 ) in W p (curl; Ω). If Ω = R 3 , these two spaces coincide, see Lemma 2.1. Although results of this kind are well known, we provide a proof for the reader's convenience. The spaces W 2 (curl; Ω) and W 2 0 (curl; Ω) are studied in detail in [13,18,26]. Extending u ∈ W p 0 (curl; Ω) by 0 outside Ω we may assume W p 0 (curl; Ω) ⊂ W p 0 (curl; R 3 ). Denote the kernel of ∇ × (·) in W 6 0 (curl; R 3 ) by W := {w ∈ W 6 0 (curl; R 3 ) : ∇ × w = 0}. Let S curl (Ω) be the largest possible constant such that the inequality |u + w| 6 dx 1 3 holds for any u ∈ W 6 0 (curl; Ω) \ W. Inequality (1.3) is in fact (trivially) satisfied also for u ∈ W 6 0 (curl; Ω) ∩ W because then both sides are zero. Note that here u but not necessarily w is supported in Ω. It is not a priori clear that S curl (Ω) is positive or that it is independent of Ω. That this is the case follows from Theorems 1.1 and 1.2(a) below: Theorem 1.1. S curl (Ω) = S curl where S curl := S curl (R 3 ).
In the next result we show that S curl is attained provided Ω = R 3 and the optimal function is (up to rescaling) a ground state solution to the curl-curl problem with critical exponent. Existence of a ground state in this case has been an open question for some time. Let (1.4) J(u) := 1 2 R 3 |∇ × u| 2 dx − 1 6 R 3 |u| 6 dx and introduce the following constraint: (1.5) N := u ∈ W 6 0 (curl; R 3 ) \ W : |u| 6 dx and div(|u| 4 u) = 0 .
As we shall see later, this set is a variant of a generalization of the Nehari manifold [27] which may be found in [28] for a Schrödinger equation. and equality holds in (1.3) for this u. If u satisfies equality in (1.3), then there are unique t > 0 and w ∈ W such that t(u + w) ∈ N and J(t(u + w)) = inf N J.
A natural question arises whether ground states must have some symmetry properties. It follows from Theorem 1.1 in [5] that any O(3)-equivariant (weak) solution to (1.6) is trivial, hence a ground state cannot be radially symmetric.
The curl-curl problem ∇ × (∇ × u) = f (x, u) in a bounded domain or in R 3 has been recently studied e.g. in [5-8, 22, 24] under different hypotheses on f but always assuming f is subcritical, i.e. f (x, u)/|u| 5 → 0 as |u| → ∞. However, the occurence of ground states to (1.6) (i.e., in the critical exponent case) has been an open problem as we have already mentioned. In view of the existence of Aubin-Talenti instantons, this is a very natural question. While the instantons are given explicitly, we have no such explicit formula for ground states in the curl-curl case. Since the instantons are radially symmetric up to translations, one can find them by ODE methods. In view of the above remark concerning O(3)-equivariant solutions, such methods do not seem available for the curl-curl problem and a different approach is needed. Note further that there is no maximum principle for the curl-curl operator and, to our knowledge, no unique continuation principle applicable to our case. An approach different than for (1.1) is also required for the proof of Ω-independence of S curl , see Section 5. Moreover concentration-compactness analysis for the curl operator is considerably different from that in [16,21,36] -see our approach in Section 3.
We would like to emphasize an important role of the analysis of nonlinear curl-curl problems from the physical point of view. Solutions u to nonlinear curl-curl equations describe the profiles of time-harmonic solutions E(x, t) = u(x) cos(ωt) to the time-dependent nonlinear electromagnetic wave equation, which together with material constitutive laws and Maxwell equations, describes the exact propagation of electromagnetic waves in a nonlinear medium [1,6,31]. Since finding propagation exactly may be very difficult, there are several simplifications in the literature which rely on approximations of the nonlinear electromagnetic wave equation. The most prominent one is the scalar or vector nonlinear Schrödinger equation. For instance, one assumes that the term ∇(div(u)) in ∇ × (∇ × u) = ∇(div(u)) − ∆u is negligible and can be dropped, or one uses the so-called slowly varying envelope approximation. However, such simplifications may produce non-physical solutions; see [2,11] and the references therein.
We also point out that the term |u| 4 u in (1.6) as well as in (1.7) below allows to consider the so-called quintic effect in nonlinear optics modelled by Maxwell equations. See for instance [1,6,14,15,23,25,31] and the references therein. We hope that our results will prompt further analytical studies of physical phenomena involving the quintic nonlinearity, e.g. the well-known cubic-quintic effect in nonlinear optics [14,25].
Using our concentration-compactness result we are also able to treat the Brezis-Nirenberg problem [10] for the curl-curl operator together with the so-called metallic boundary condition Here ν : ∂Ω → R 3 is the exterior normal and Ω ⊂ R 3 is a bounded domain. This boundary condition is natural in the theory of Maxwell equations and it holds when Ω is surrounded by a perfect conductor. If the boundary of Ω is not of class C 1 , then we assume (1.8) is satisfied in a generalized sense by which we mean u is in the space W 6 0 (curl; Ω) defined in (1.2). Weak solutions to (1.7)-(1.8) correspond to critical points of the associated energy functional J λ : W 6 0 (curl; Ω) → R given by Recall from [7,23] that the spectrum of the curl-curl operator in H 0 (curl; Ω) := W 2 0 (curl; Ω) consists of the eigenvalue λ 0 = 0 with infinite multiplicity and of a sequence of eigenvalues with corresponding finite multiplicities m(λ k ) ∈ N. Let N λ be the generalized Nehari manifold for J λ (see (6.1) for the definition), and for λ ≤ 0 let Denote the Lebesque measure of Ω by |Ω|. We introduce the following condition: (Ω) Ω is a bounded domain, either convex or with C 1,1 -boundary. The reason for this assumption will be explained in the next section.
In domains Ω = R 3 we also introduce another constant, S curl (Ω), such that the inequality holds for any u ∈ W 6 0 (curl; Ω) \ W Ω , where W Ω := {w ∈ W 6 0 (curl; Ω) : ∇ × w = 0}, and S curl (Ω) is largest with this property. As in (1.3), also here the above inequality trivially holds if u ∈ W Ω . Although S curl (Ω) seems to be more natural than S curl (Ω), we do not know whether it equals S curl . We are only able to prove the following result: Let Ω be a Lipschitz domain in R 3 , possibly unbounded, Ω = R 3 . Then S curl ≥ S curl (Ω). If Ω satisfies (Ω), then S curl (Ω) ≥ S.
Finally, the main result concerning the Brezis-Nirenberg problem for the curl-curl operator (1.7) reads as follows.
The above result is known for cylindrically symmetric domains where it is possible to reduce the curl-curl operator to a positive definite one, see [23]. However, the solution obtained there is a ground state in a subspace of functions having cylindric symmetry and we do not know whether it is a ground state in the full space.
Let us recall from earlier work that the main difficulties when treating J and J λ , also in the subcritical case, are that these functionals are strongly indefinite, i.e., they are unbounded from above and from below, even on subspaces of finite codimension. Moreover, the quadratic part of J has infinite-dimensional kernel and J ′ , J ′ λ are not (sequentially) weak-to-weak * continuous, i.e. u n ⇀ u does not imply that . This lack of continuity is caused by the fact that W p 0 (curl; Ω) is not (locally) compactly embedded in any Lebesgue space and we do not know whether necessarily u n → u a.e. in Ω. A consequence of this is that for a Palais-Smale sequence u n ⇀ u it is not clear whether u is a critical point. In the subcritical case one can overcome these difficulties since either a variant of the Palais-Smale condition is satisfied or some compactness can be recovered on a suitable topological manifold, see e.g. [6,22,24]. In the critical case however, there are additional difficulties. In Section 3 we introduce a general concentration-compactness analysis for this case. We show that the topological manifold is locally compactly embedded in L p (R 3 , R 3 ) for 1 ≤ p < 6 and that if a sequence (u n ) is contained in this manifold and u n ⇀ u, then u n → u a.e. after passing to a subsequence. This result will play a crucial role because it implies that if such (u n ) is a Palais-Smale sequence, then u is a solution for our equation. If the condition div(|u| 4 u) = 0 is violated, the embedding need not be locally compact. The paper is organized as follows. In Section 2 we introduce the functional setting and some notation. Section 3 concerns the concentration-compactness analysis as we have already mentioned. In Section 4 we prove Theorem 1.2, and in Section 5 we prove Theorems 1.1 and 1.3. The proof of Theorem 1.4 is contained in Section 6 whereas in Section 7 we state some open problems.

Functional setting and preliminaries
Throughout the paper we assume that Ω is a Lipschitz domain in R 3 and 2 ≤ p ≤ 2 * = 6. The curl of u, ∇ × u, should be understood in the distributional sense. We shall look for solutions to (1.6) and (1.7)-(1.8) in the space W 6 0 (curl; R 3 ) and W 6 0 (curl; Ω) respectively. We introduce the subspaces W Ω := w ∈ W 6 0 (curl; Ω) : The second one has already been defined in Section 1. Here and below . , . denotes the inner product in R 3 . If Ω = R 3 , we shall usually write V and W for V R 3 and W R 3 .
In the sequel Ω is always a Lipschitz domain and C denotes a generic positive constant which may vary from one equation to another.
In the following subsections we consider two cases.
This completes the proof.

General concentration-compactness analysis in R N
In this, self-contained, section we have N ≥ 3 and we work in subspaces of x ∈ Ω, F (x, 0) = 0 and f = ∂ u F : Ω × R N → R N is a Carathéodory function (i.e., f is measurable in x ∈ Ω for all u ∈ R N and continuous in u ∈ R N for a.e. x ∈ Ω); (F2) F is uniformly strictly convex with respect to u ∈ R N , i.e. for any compact set A ⊂ (F3) There are c 1 , c 2 > 0 and a ∈ L N/2 (Ω), a ≥ 0, such that for every u ∈ R N and a.e. x ∈ Ω.
In view of (F2) and (F3), for any v ∈ V we find a unique w Ω (v) ∈ W such that This implies that Denote the space of finite measures in R N by M(R N ).
Then there exists an at most countable set I ⊂ R N and nonnegative weights {µ and passing to a subsequence, in Ω and in L p loc (Ω) for any 1 ≤ p < 2 * . Remark 3.2. We shall use this theorem in Sections 4 and 6. In Section 4 we have Ω = R 3 and Z = {0}, so w = w and we will write w(v) for w R 3 (v). In Section 6, where we treat a Brezis-Nirenberg problem, Ω will be bounded and Z the subspace of V Ω on which the quadratic part of J λ (see (1.9)) is negative semidefinite.
Passing to the limit and using the Brezis-Lieb lemma [9,36] on the left-hand side above we obtain Since µ is finite and µ,μ have the same singular set, I is at most countable and µ ≥ |∇v 0 | 2 + x∈I µ x δ x . As in the proof of Theorem 1.9 in [16] it follows from (3.5) thatρ = x∈I ρ x δ x , see also Proposition 4.2 in [35]. So µ and ρ are as claimed.
We can find a sequence of open balls (B l ) ∞ l=1 such that Ω = ∞ l=1 B l . Fix l ≥ 1. In view of [19, Lemma 1.1] there exists ξ n ∈ W 1,2 * (B l ) such that w n = ∇ξ n and we may assume without loss of generality that B l ξ n dx = 0. Then by the Poincaré inequality, ξ n W 1,2 * (B l ) ≤ C|w n | L 2 * (B l ,R N ) ≤ C|w n | 2 * and passing to a subsequence, ξ n ⇀ ξ for some ξ ∈ W 1,2 * (B l ). So ξ n → ξ in L 2 * (B l ). Now take any ϕ ∈ C ∞ 0 (B l ). Since ∇(|ϕ| 2 * (ξ n − ξ)) ∈ W, in view of (3.3) we get where the right-hand side tends to 0 as n → ∞. Since w n ⇀ ∇ξ in L 2 * (B l ), hence, recalling that w Ω (v n ) = w n + z n and z n → z 0 , we obtain The convexity of F in u implies that Adding these inequalities and using (F2), we obtain for any k ≥ 1 and Let Taking into account (3.6) and using (F3), (3.7) and Hölder's inequality, we get where k is fixed. Here we have used the fact that Ω a(x)|v for any Borel set E ⊂ B l . We find an open set E k ⊃ I such that |E k | < 1/2 k+1 . Then, taking E = B l \ E k in (3.9), we have 4m k |Ω n,k ∩ (B l \ E k )| = o(1) as n → ∞ because supp(ρ) ⊂ I; hence we can find a sufficiently large n k such that |Ω n k ,k ∩ B l | < 1/2 k and we obtain k=j Ω n k ,k and x ∈ B l , then , the second and the third inequality above cannot hold on a set of positive measure for all large k. We infer that hence by the Vitali convergence theorem, v n − v 0 + w Ω (v n ) − w 0 → 0 in L p loc (Ω) after passing to a subsequence.
Step 3. We show that w Ω (v 0 ) = w 0 . Take any w ∈ W and observe that by the Vitali convergence theorem, up to a subsequence. Now (3.3) implies that w 0 = w Ω (v 0 ) which completes the proof.
Since div(v) = 0, Proof. Given ε > 0, by (4.2) we can find v = 0 such that . By the Hölder inequality, Using this and the Sobolev inequality gives , and since w(v) is a minimizer, we obtain using (4.3) and (4.6) Hence S curl + ε ≥ S for all ε > 0 and the conclusion follows.
Next we look for ground states for the curl-curl problem (1.6), i.e. nontrivial solutions with least possible associated energy J given by (1.4). Throughout the rest of the paper we shall make repeated use of the following fact: Proof. We prove this for w Ω . Using the minimizing property of w Ω (u) we obtain Since the minimizer is unique, w Ω (u) = w Ω (λu)/λ as claimed.  3) that if v ∈ V, then J ′ (v + w(v))| W = 0, and as (4.9) there is a unique t(v) > 0 such that We note that (4.11) J(m(v)) ≥ J(t(v + w)) for all t > 0 and w ∈ W.
Since J(m(v)) ≥ J(v) and there exist a, r > 0 such that J(v) ≥ a if v = r, N is bounded away from W and hence closed.
it follows that (w(v n )) is bounded and it is then clear from (4.9) that so is (t(v n )). Hence we may assume t(v n ) → t 0 and w(v n ) ⇀ w 0 in L 6 (R 3 , R 3 ). By the weak sequential lower semicontinuity of the second integral in (4.9) and by (4.11), Now it is easily seen that m| S : S := {v ∈ V : v = 1} → N is a homeomorphism with the inverse u = v + w(v) → v/ v . Note that N is an infinite-dimensional topological manifold of infinite codimension. Although J is of class C 2 , we do not know whether N is of class C 1 . However, repeating the argument in [22,Proposition 4.4(b)] or [32, Proposition 2.9] we see that J • m| S : S → R is of class C 1 and is bounded from below by the constant a > 0 introduced above. By the Ekeland variational principle [36,Theorem 8.5], there is a Palais-Smale sequence (v n ) ⊂ S such that For s > 0, y ∈ R 3 and u : R 3 → R 3 we denote T s,y (u) := s 1/2 u(s · +y)). The following lemma is a special case of [29,Theorem 1], see also [34,Lemma 5.3].
Observe that the above lemma in [29] is expressed in terms of the space H 1,2 . However, in the notation of [29], this is the same space as our D 1,2 . Lemma 4.6. T s,y is an isometric isomorphism of W 6 0 (curl; R 3 ) which leaves the functional J and the subspaces V, W invariant. In particular, w(T s,y u) = T s,y w(u).
The proof is by an explicit (and simple) computation.
Proof of Theorem 1.2. We prove part (b) first. Take a minimizing sequence (u n ) = (m(v n )) ⊂ N constructed above and write Since J(u n ) is bounded away from 0, |u n | 6 → 0 and hence by (4.12), |v ′ n | 6 → 0. Therefore, passing to a subsequence and using Lemma 4.5, v n := T sn,yn (v ′ n ) ⇀ v 0 for some v 0 = 0, (s n ) ⊂ R + and (y n ) ⊂ R 3 . Taking subsequences again we also have that v n → v 0 a.e. in R 3 and in view of Theorem 3.1, w( v n ) ⇀ w(v 0 ) and w( v n ) → w(v 0 ) a.e. in R 3 . We set u := v 0 + w(v 0 ) and by Lemma 4.6 we may assume without loss of generality that s n = 1 and y n = 0. So if z ∈ W 6 0 (curl; R 3 ), then using weak and a.e. convergence, Here we have used that |u n | 4 u n ⇀ ζ in L 6/5 (R 3 , R 3 ) for some ζ but since |u n | 4 u n → |u| 4 u a.e., ζ = |u| 4 u. So u is a solution to (1.6). To show it is a ground state, we note that using Fatou's lemma, Hence J(u) ≤ inf N J and as a solution, u ∈ N . It follows using Lemma 4.7 that J(u) = inf N J = 1 3 S 3/2 curl . If u satisfies equality in (1.3), then t(u)(u + w(u)) ∈ N and is a minimizer for J| N . But then the corresponding point v in S is a minimizer for J • m| S , see (4.13). So v is a critical point of J • m| S and m(v) = u is a critical point of J. This completes the proof of (b).
(a) By Lemma 4.1, S curl ≥ S and by part (b), there exists u = v + w(v) for which S curl is attained. Suppose S curl = S. Then all inequalities become equalities in (4.7) with ε = 0, and therefore also in (4.6). But then R 3 |∇v i | 2 dx = S|v i | 2 6 for i = 1, 2, 3 and hence all v i are instantons, up to multiplicative constants. Since v = 0 and div(v) = 0, this is impossible. It follows that S curl > S. ✷

5.
Proof of Theorems 1.1 and 1.3 Let Ω be a Lipschitz domain in R 3 . Recall from Section 2 that we have the Helmholtz decompositions (5.1) W 6 0 (curl; R 3 ) = V ⊕ W and W 6 0 (curl; Ω) = V Ω ⊕ W Ω where the second one holds if condition (Ω) in the introduction is satisfied. For u ∈ W 6 0 (curl; Ω), denote the minimizer of Ω |u + w| 6 dx, w ∈ W Ω by w Ω (u) (cf. (4.1)) and, according to our notational convention, write w(u) for w R 3 (u). Recall from (1.3) the definition of S curl (Ω): where u ∈ W 6 0 (curl; Ω) \ W and S curl (Ω) is the largest constant with this property. By (5.1) we have u = v + w ∈ V ⊕ W. We emphasize that although u = 0 in R 3 \ Ω, v and w need not be 0 there. Note that S curl (Ω) can be characterized as As we have noticed in the introduction, although this constant seems more natural, we do not know whether it equals S curl .
Proof. Let u n → u 0 . Since (w Ω (u n )) is bounded, w Ω (u n ) ⇀ w 0 after passing to a subsequence. By the maximality and uniqueness of w Ω (·), Hence all inequalities above must be equalities and it follows that w 0 = w Ω (u 0 ) and w Ω (u n ) → w Ω (u 0 ).
In view of Lemma 2.3, W Ω ⊂ W, hence we easily infer from (5.2), (5.3) that S curl (Ω) ≥ S curl (Ω). As W 6 0 (curl; Ω) ⊂ W 6 0 (curl; R 3 ), it follows that S curl ≤ S curl (Ω). Next we show that S curl (Ω) ≤ S curl . Let u 0 be a minimizer for J on N provided by Theorem 1.2(b) and find a sequence (u n ) ⊂ C ∞ 0 (R 3 , R 3 ) such that u n → u 0 . We can decompose u n as So v 0 = 0 and v n are bounded away from 0 in L 6 (R 3 , R 3 ). Assume without loss of generality that 0 ∈ Ω. There exist λ n such that u n given by u n (x) := λ 1/2 n u n (λ n x) are supported in Ω. Set w n := w( u n ) ∈ W and choose t n so that t n ( u n + w n ) ∈ N . Then According to Lemma 4.6, u n = u n and | u n + w n | 6 = |u n + w(u n )| 6 = |v n + w(v n )| 6 . As (u n ) is bounded, so is ( u n ) and as |v n + w(v n )| 6 → |v 0 + w(v 0 )| 6 , | u n + w n | 6 is bounded away from 0. So (t n ) is bounded. Moreover, | w n | 6 = |w(u n )| 6 and therefore ( w n ) is bounded. Since J( u n ) = J(u n ) → 1 3 S 3/2 curl and J ′ ( u n ) = J ′ (u n ) → 0, it follows from Lemma 5.2 that The last inequality follows from Lemma 4.7 and the fact that u n are as in (5.2), i.e. u n ∈ W 6 0 (curl; Ω). It remains to show that S curl (Ω) ≥ S if (Ω) is satisfied. But this follows by repeating the argument of Lemma 4.1 with obvious changes: S curl should be replaced by S curl (Ω), w(v) by w Ω (v) and the domain of integration should be Ω. ✷ Remark 5.3. Let Ω = R 3 and suppose S curl (Ω) is attained by some u. Extend u by 0 outside Ω. As S curl (Ω) = S curl , u also solves (1.6) in R 3 , possibly after replacing u with αu for an appropriate α > 0. In particular, if S curl (Ω) were attained in a bounded Ω, this would imply the existence of ground states in R 3 which have compact support. To our knowledge, there is no unique continuation principle which could rule out this possibility.
In view of this remark we expect that similarly as is the case for the Sobolev constant, S curl is attained if and only if Ω = R 3 . We leave this problem as a conjecture.
6. The Brezis-Nirenberg type problem and proof of Theorem 1.4 Let λ ≤ 0. In this section Ω ⊂ R 3 is a fixed bounded domain satisfying (Ω) but λ will be varying. Therefore we drop the subscript Ω from notation and replace it by λ (J λ , N λ etc.). We also write V, W for V Ω , W Ω .
Recall from the introduction and Subsection 2.2 that the spectrum of the curl-curl operator in H 0 (curl; Ω) consists of the eigenvalue λ 0 = 0 whose eigenspace is W and of a sequence of eigenvalues 0 < λ 1 ≤ λ 2 ≤ · · · ≤ λ k → ∞ with finite multiplicities m(λ k ) ∈ N. The eigenfunctions corresponding to different eigenvalues are L 2 -orthogonal and those corresponding to λ k > 0 are in V.
For λ ≤ 0 we find two closed and orthogonal subspaces V + and V of V such that the quadratic form Q : V → R given by and our functional J λ (see (1.9)) can be expressed as We shall use Theorem 3.1 with Here W := V ⊕ W (so Z = V in the notation of Section 3) and w = v + w. V, and hence V + , may be considered, after a proper extension, as closed subspaces of This extension is bounded as a mapping from V to , and hence of D 1,2 (R 3 , R 3 ), we can apply Theorem 3.1 with F as above and V + replacing V. The generalized Nehari manifold is now given by (cf. (4.10)) and J λ • m λ is of class C 1 on S + . Moreover, m λ | S + is a homeomorphism between S + and N λ . As in (4.13), we may find a Palais-Smale sequence (v + n ) ⊂ S + such that Note that Lemma 6.1. Let λ ∈ (−λ ν , −λ ν−1 ] for some ν ≥ 1. There holds Proof. The first inequality has been established in [23,Lemma 4.7]. However, for the reader's convenience we include the argument. Let e ν be an eigenvector corresponding to λ ν . Then e ν ∈ V + . Choose t > 0, v ∈ V and w ∈ W so that u = v + w = te ν + v + w ∈ N λ . Since λ k ≤ λ ν for k < ν, In the last step we have used the elementary inequality A 2 t 2 − 1 6 t 6 ≤ 1 3 A 3/2 (A > 0). Since c 0 = 1 3 S curl (Ω) 3/2 , the second inequality follows immediately.
If c λ < c 0 , then in view of [23, Theorem 2.2 (a)] there is a Palais-Smale sequence (u n ) ⊂ N λ such that J λ (u n ) → c λ > 0 and u n ⇀ u 0 = 0 in W 6 0 (curl; Ω). It has been unclear so far whether u 0 is a critical point of J λ . Now we shall show using the concentration-compactness analysis from Section 3 that u 0 is not only a solution but even a ground state for (1.7). The following lemma plays a crucial role. Lemma 6.2. If (u n ) ⊂ N λ is bounded, then, passing to a subsequence, u n → u 0 in L 2 (Ω, R 3 ) for some u 0 .

Open problems
In this section we state some open problems. Some of them have already been mentioned earlier.
(P1) Does there exist a ground state solution u whose support is a proper subset of R 3 ? In particular, can a ground state have compact support? (P2) Can one find an explicit expression for a ground state? Or at least, what can be said about the decay properties of ground states? If they are the same as for the Aubin-Talenti instantons, then one could hopefully retrieve the formulas in the middle of p. 35 in [36] which could be useful when looking for ground states for (1.6) with the right-hand side |u| 4 u + g(x, u) where g is a monotone lower order term. (P3) Do the ground state solutions to (1.6) have any symmetry properties? How regular are they?
(P4) If Ω is a bounded domain which is neither convex nor has C 1,1 boundary, then V ⊂ H s (Ω, R 3 ) where s ∈ [1/2, 1] and s may be strictly less than 1, see Subsection 2.2 and [12]. Note that the critical exponent for H s is 6/(3 − 2s) < 6 if s < 1. Do the results of Theorem 1.4 remain valid (with the same right-hand side)? Here the boundary condition (1.8) should be understood in a generalized sense, i.e. u should be in W 6 0 (curl; Ω). (P5) Can the inequality S curl ≥ S curl (Ω) ≥ S be sharpened? Do there exist domains as in (P4) for which S curl (Ω) < S?