Fractional Kirchhoff problems with critical Trudinger–Moser nonlinearity

Abstract

This paper is concerned with the existence of solutions for a class of fractional Kirchhoff-type problems with Trudinger–Moser nonlinearity:

$$\begin{aligned} {\left\{ \begin{array}{ll} M\left( \displaystyle \iint _{{\mathbb {R}}^{2N}}\frac{|u(x)-u(y)|^{N/s}}{|x-y|^{2N}}dxdy\right) (-\Delta )^{s}_{N/s}u=f(x,u)\,\, \ &{}\quad \mathrm{in}\ \Omega ,\\ u=0\ \ \ \ &{}\quad \mathrm{in}\ {\mathbb {R}}^N{\setminus } \Omega , \end{array}\right. } \end{aligned}$$

where $(-\Delta )^{s}_{N/s}$ is the fractional N / s-Laplacian operator, $N\ge 1$, $s\in (0,1)$, $\Omega \subset {\mathbb {R}}^N$ is a bounded domain with Lipschitz boundary, $M:{\mathbb {R}}^+_0\rightarrow {\mathbb {R}}^+_0$ is a continuous function, and $f:\Omega \times {\mathbb {R}}\rightarrow {\mathbb {R}}$ is a continuous function behaving like $\exp (\alpha t^{2})$ as $t\rightarrow \infty$ for some $\alpha >0$. We first obtain the existence of a ground state solution with positive energy by using minimax techniques combined with the fractional Trudinger–Moser inequality. Next, the existence of nonnegative solutions with negative energy is established by using Ekeland’s variational principle. The main feature of this paper consists in the presence of a (possibly degenerate) Kirchhoff model, combined with a critical Trudinger–Moser nonlinearity.

1 Introduction and main results

In this paper, we study the following fractional Kirchhoff-type problem:

$$\begin{aligned} {\left\{ \begin{array}{ll} M\left( \displaystyle \iint _{{\mathbb {R}}^{2N}}\frac{|u(x)-u(y)|^{N/s}}{|x-y|^{2N}}dxdy\right) (-\Delta )^{s}_{N/s}u=f(x,u)\,\, \ &{}\quad \mathrm{in}\ \Omega ,\\ u=0\ \ \ \ &{}\quad \mathrm{in}\ {\mathbb {R}}^N{\setminus } \Omega , \end{array}\right. } \end{aligned}$$

(1.1)

where $N\ge 1$, $s\in (0,1)$, $\Omega \subset {\mathbb {R}}^N$ is a bounded domain with Lipschitz boundary, $M:[0,\infty )\rightarrow [0,\infty )$ is a continuous function, $f:\Omega \times {\mathbb {R}}\rightarrow {\mathbb {R}}$ is a continuous function behaving like $\exp (\alpha |t|^{\frac{N}{N-s}})$ as $t\rightarrow \infty$ for some $\alpha >0$, and $(-\Delta )^{s}_{N/s}$ is the fractional N / s-Laplacian operator which, up to a normalization constant, is defined as

$$\begin{aligned} (-\Delta )^{s}_{N/s}\varphi (x)=2 \lim _{\varepsilon \rightarrow 0^+}\int _{\mathbb {R^N}{\setminus } B_\varepsilon (x)}\frac{|\varphi (x)-\varphi (y)|^{\frac{N}{s}-2}(\varphi (x)-\varphi (y)}{|x-y|^{2N}}\,dy,\quad x\in {\mathbb {R}}^N, \end{aligned}$$

along functions $\varphi \in C_0^\infty ({\mathbb {R}}^N)$. Throughout this paper, $B_\varepsilon (x)$ denotes the ball in ${\mathbb {R}}^N$ centered at $x\in {\mathbb {R}}^N$ with radius $\varepsilon >0$.

To study the existence of solutions for problem (1.1), let us recall some results related to the fractional Sobolev space $W_0^{s,p}(\Omega )$. Let $1<p<\infty$ and set

$$\begin{aligned} W_0^{s,p}(\Omega )=\left\{ u\in L^{p}(\Omega ): [u]_{s,p}<\infty ,\ u=0 \ \mathrm{a.e.\ in }\ {\mathbb {R}}^N{\setminus }\Omega \right\} , \end{aligned}$$

where the Gagliardo seminorm $[u]_{s,p}$ is defined as

$$\begin{aligned}{}[u]_{s,p}= \left( \iint _{{\mathbb {R}}^{2N}}\frac{|u(x)-u(y)|^{p}}{|x-y|^{N+sp}}dxdy\right) ^{1/p} . \end{aligned}$$

Equipped with the norm

$$\begin{aligned} \Vert u\Vert :=[u]_{s,p}, \end{aligned}$$

$W_0^{s,p}(\Omega )$ is a uniformly convex Banach space, and hence reflexive, see [38] for more details. The fractional critical exponent is defined by

$$\begin{aligned} p_s^*= {\left\{ \begin{array}{ll} \displaystyle \frac{Np}{N-sp}\ \ &{}\quad \mathrm{if}\ sp<N;\\ \infty \ \ &{}\quad \mathrm{if}\ sp\ge N. \end{array}\right. } \end{aligned}$$

Moreover, the fractional Sobolev embedding theorems states that $W_0^{s,p}(\Omega )\hookrightarrow L^{p_s^*}(\Omega )$ is continuous if $sp<N$ and $W_0^{s,p}(\Omega )\hookrightarrow L^{q}(\Omega )$ is continuous for all $p\le q<\infty$ if $sp=N$. For more detailed account on the properties of $W_0^{s,p}(\Omega )$, we refer to [10].

In recent years, great attention has been paid to study problems involving fractional operators. In particular, many works focus on the subcritical and critical growth of the nonlinearity which allows us to treat the problem variationally using general critical point theory. Problems like (1.1) are important in many fields of science, notably continuum mechanics, phase transition phenomena, population dynamics, minimal surfaces and anomalous diffusion, as they are the typical outcome of stochastically stabilization of Lévy processes, see [1, 4, 21] and the references therein. Moreover, such equations and the associated fractional operators allow us to develop a generalization of quantum mechanics and also to describe the motion of a chain or an array of particles that are connected by elastic springs as well as unusual diffusion processes in turbulent fluid motions and material transports in fractured media, for more details see [4, 5] and the references therein.

Recently, some authors have paid considerable attention in the limiting case of the fractional Sobolev embedding, commonly known as the Trudinger–Moser case. For example, when $n=2$, $W^{1,2}(\Omega )\hookrightarrow L^r(\Omega )$ for $1\le r<\infty$ but $W^{1,2}(\Omega )\not \hookrightarrow L^\infty (\Omega )$. To fill this gap, Trudinger [37] proved that that there exists $\tau >0$ such that $W^{1,2}_0(\Omega )$ is embedded into the Orlicz space $L_{\phi _\tau }(\Omega )$ determined by the Young function $\phi _\tau =\text{ exp }({\tau t^2}-1)$. After that, Moser [25] found the best exponent $\tau$ and in particular he obtained a result which is now referred as Trudinger–Moser inequality. In [24], Martinazzi proved that for each $u\in W_0^{s,N/s}(\Omega )$ and $\alpha >0$, there holds

$$\begin{aligned} \int _{\Omega } \exp \left( \alpha |u|^{\frac{N}{N-s}}\right) dx<\infty . \end{aligned}$$

Moreover, there exist positive constants

$$\begin{aligned}\alpha _{N,s} =\frac{N}{\omega _{N-1}}\left( \frac{\Gamma ((N-s)/2)}{\Gamma (s/2)2^s\pi ^{N/2}}\right) ^{-\frac{N}{N-s}}, \end{aligned}$$

were $\omega _{N-1}$ be the surface area of the unit sphere in ${\mathbb {R}}^N$ and $C_{N,s}$ depending only on N and s such that

$$\begin{aligned} \sup _{\begin{array}{c} u\in W_0^{s,N/s}(\Omega )\\ {[}u]_{s,N/s}\le 1 \end{array}}\int _\Omega \exp \left( \alpha |u|^{\frac{N}{N-s}}\right) dx\le C_{N,s}|\Omega |, \end{aligned}$$

(1.2)

for all $\alpha \in [0,\alpha _{N,s}]$ and there exists $a_{N,s}^*\ge \alpha _{N,s}$ such that the supremum in (1.2) is $\infty$ for $\alpha >\alpha _{N,s}$. For more details about Trudinger–Moser inequality, we also refer to [19, 30]. When $N\ne 1$ and $s\ne 1/2$, it is still an open problem whether $a_{N,s}^*=\alpha _{N,s}$ or not. However, for $N=1$ and $s=1/2$, one can calculate that $\alpha _{N,s}=\alpha _{N,s}^*=2\pi ^2$ and there exists $C>0$ such that

$$\begin{aligned} \sup _{\begin{array}{c} u\in W_0^{\frac{1}{2},2}(\Omega )\\ {[}u]_{1/2,2}\le 1 \end{array}}\int _\Omega \exp (\alpha u^{2})dx\le C|\Omega |, \end{aligned}$$

(1.3)

for all $\alpha \in [0,2\pi ^2]$ and the supremum in (1.3) is $\infty$ for $\alpha >2\pi ^2$.

In the setting of the fractional Laplacian, Iannizzotto and Squassina [17] investigated existence of solutions for the following Dirichlet problem

$$\begin{aligned} {\left\{ \begin{array}{ll} (-\Delta )^{\frac{1}{2}} u=f(u)\ \ &{}\quad \mathrm{in}\ (0,1),\\ u=0\quad \ \ &{}\quad \mathrm{in}\ {\mathbb {R}}{\setminus }(0,1), \end{array}\right. } \end{aligned}$$

(1.4)

where f(u) behaves like $\exp (\alpha |u|^2)$ as $u\rightarrow \infty$. Using the mountain pass theorem, they obtained the existence of solutions for problem (1.4). Subsequently, Giacomoni, Mishra and Sreenadh [16] studied the multiplicity of solutions for problems like (1.4) by using the Nehari manifold method. Very recently, Perera and Squassina [32] studied the bifurcation results for the following problem with Trudinger–Moser nonlinearity

$$\begin{aligned} {\left\{ \begin{array}{ll} (-\Delta )^s_{N/s}u=\lambda |u|^{(N-2s)/s}\exp (|u|^{N/(N-s)})\,\, \ &{}\quad \mathrm{in}\ \Omega ,\\ u=0\ \ \ \ &{}\quad \mathrm{in}\ {\mathbb {R}}^N{\setminus }\Omega , \end{array}\right. } \end{aligned}$$

where $\lambda >0$ is a parameter.

For unbounded domains and the general fractional p-Laplacian, Souza [12] considered the following nonhomogeneous fractional p-Laplacian equation

$$\begin{aligned} (-\Delta )_p^su+V(x)|u|^{p-2}u=f(x,u)+\lambda h\quad \ \ \mathrm{in}\ {\mathbb {R}}^N, \end{aligned}$$

(1.5)

where $(-\Delta )_p^s$ is the fractional p-Laplacian and the nonlinear term f satisfies exponential growth. The author obtained a nontrivial weak solution of the Eq. (1.5) by using fixed point theory. Li and Yang [22] studied the following equation

$$\begin{aligned} (-\Delta )_p^\zeta u+V(x)|u|^{p-2}u=\lambda A(x) |u|^{q-2}u+f(u)\ \quad \mathrm{x}\in {\mathbb {R}}^\mathrm{N}, \end{aligned}$$

where $p\ge 2$, $0<\zeta <1$, $1<q<p$, $\lambda >0$ is a real parameter, A is a positive function in $L^{\frac{p}{p-q}}({\mathbb {R}}^N)$, $(-\Delta )_p^\zeta$ is the fractional p-Laplacian and f satisfies exponential growth.

On the other hand, Li and Yang [23] studied the following Schrödinger-Kirchhoff type equation

$$\begin{aligned} \left( \int _{{\mathbb {R}}^N}(|\nabla u|^N{+}V(x)|u|^N)dx\right) ^k(-\Delta _N u {+}V(x)|u|^{N-2}u)=\lambda A(x)|u|^{p-2}u+ f(u)\quad \text{ in } {\mathbb {R}}^N, \end{aligned}$$

(1.6)

where $\Delta _N u=\mathrm{div}(|\nabla u|^{N-2}\nabla u)$ is the N-Laplacian, $k>0$, $V:{\mathbb {R}}^N\rightarrow (0,\infty )$ is continuous, $\lambda >0$ is a real parameter, A is a positive function in $L^{\frac{p}{p-q}}({\mathbb {R}}^N)$ and f satisfies exponential growth. By using the mountain pass theorem and Ekeland’s variational principle, the authors obtained two nontrivial solutions of (1.6) as the parameter $\lambda$ small enough. Actually, the study of Kirchhoff-type problems, which arise in various models of physical and biological systems, have received more and more attention in recent years. More precisely, Kirchhoff [18] established a model governed by the equation

$$\begin{aligned} \rho \frac{\partial ^2u}{\partial t^2}-\left( \frac{p_0}{h}+\frac{E}{2L}\int _0^L\left| \frac{\partial u}{\partial x}\right| ^2dx\right) \frac{\partial ^2u}{\partial x^2}=0, \end{aligned}$$

(1.7)

for all $x\in (0,L),t\ge 0$, where $u=u(x,t)$ is the lateral displacement at the coordinate x and the time t, E is the Young modulus, $\rho$ is the mass density, h is the cross-section area, L is the length and $p_0$ is the initial axial tension. Equation (1.7) extends the classical D’Alembert wave equation by considering the effects of the changes in the length of the strings during the vibrations. Recently, Fiscella and Valdinoci [14] proposed a stationary Kirchhoff model driven by the fractional Laplacian by taking into account the nonlocal aspect of the tension, see [14, Appendix A] for more details.

In particular, when $s\rightarrow 1$ and $M\equiv 1$, problem (1.1) becomes

$$\begin{aligned} {\left\{ \begin{array}{ll} -\Delta _{N}u=f(x,u)\,\, \ &{}\quad \mathrm{in}\ \Omega ,\\ u=0\ \ \ \ &{}\quad \mathrm{in}\ \mathbb {R^N}{\setminus } \Omega , \end{array}\right. } \end{aligned}$$

which studied by many authors by using variational methods, see for example, [9, 11, 15, 20]. Here $\Delta _{N}u=\mathrm{div}(|\nabla u|^{N-2}\nabla u)$ is the N-Laplacian. When $s\rightarrow 1$, problem (1.1) becomes

$$\begin{aligned} {\left\{ \begin{array}{ll} -M\left( \displaystyle \int _{{\mathbb {R}}^{N}}|\nabla u(x)|^{N}dx\right) \Delta _{N}u=f(x,u)\,\, \ &{}\quad \mathrm{in}\ \Omega ,\\ u=0\ \ \ \ &{}\quad \mathrm{in}\ \mathbb {R^N}{\setminus } \Omega , \end{array}\right. } \end{aligned}$$

(1.8)

In [13], Figueiredo and Severo studied problem (1.8) with $N=2$, and the existence of ground state solution obtained by using minimax techniques with the Trudinger–Moser inequality.

Inspired by the above works, especially by [15, 28], we are devoted to the existence of ground state solution of (1.1) and overcome the lack of compactness due to the presence of exponential growth terms as well as the degenerate nature of the Kirchhoff coefficient. To the best of our knowledge, there are no results for (1.1) in such a generality.

Throughout the paper, without explicit mention, we assume that $M:{\mathbb {R}}^+_0\rightarrow {\mathbb {R}}^+_0$ is a continuous function with $M(0)=0$, and verifies

$(M_1)$ :

for any $d>0$ there exists $\kappa :=\kappa (d)>0$ such that $M(t)\ge \kappa$ for all $t\ge d$;

$(M_2)$ :

there exists $\theta >1$ such that $\displaystyle \frac{M(t)}{t^{\theta -1}}$ is nonincreasing for $t>0$;

$(M_3)$ :

for any $t_1,t_2\ge 0$ there holds

$$\begin{aligned} {\mathscr {M}}(t_1)+{\mathscr {M}}(t_2)\le {\mathscr {M}}(t_1+t_2). \end{aligned}$$

Remark 1.1

By $(M_2)$, we can obtain that

$$\begin{aligned} \theta {\mathscr {M}}(t)-M(t)t\ \ \mathrm{is\ nondecreasing\ for\ }\ t>0, \end{aligned}$$

where ${\mathscr {M}}(t)=\int _0^t M(\tau )d\tau$. Indeed, for any $0<t_1<t_2$,

$$\begin{aligned} \theta {\mathscr {M}}(t_1)-M(t_1)t_1&=\theta {\mathscr {M}}(t_2)-\theta \int _{t_1}^{t_2}M(t)dt -\frac{M(t_1)}{t_1^{\theta -1}}t_1^{\theta }\\&\le \theta {\mathscr {M}}(t_2)-\frac{M(t_2)}{t_2^{\theta -1}}(t_2^{\theta } -t_1^{\theta }) -\frac{M(t_2)}{t_2^{\theta -1}}t_1^{\theta }\\&=\theta {\mathscr {M}}(t_2)-M(t_2)t_2, \end{aligned}$$

thanks to assumption $(M_2)$. Thus, $\theta {\mathscr {M}}(t)-M(t)t$ is nondecreasing for $t>0$. In particular, we have

$$\begin{aligned} \theta {\mathscr {M}}(t)-M(t)t\ge 0,\quad \forall t\ge 0. \end{aligned}$$

(1.9)

A typical example of M is given by $M(t)=a_0+b_0\,t^{\theta -1}$ for all $t\ge 0$ and some $\theta >1$, where $a_0,b_0\ge 0$ and $a_0+b_0>0$. When M is of this type, problem (1.1) is said to be degenerate if $a=0$, while it is called non-degenerate if $a>0$. Recently, the fractional Kirchhoff problems have received more and more attention. Some new existence results of solutions for fractional non-degenerate Kirchhoff problems are given, for example, in [33, 34, 38]. On some recent results concerning about the degenerate case of Kirchhoff-type problems, we refer to [3, 6, 7, 26, 27, 35, 39,40,41] and the references therein. It is worth mentioning that the degenerate case is rather interesting and is treated in well–known papers in Kirchhoff theory, see for example [8]. In the large literature on degenerate Kirchhoff problems, the transverse oscillations of a stretched string, with nonlocal flexural rigidity, depends continuously on the Sobolev deflection norm of u via $M(\Vert u\Vert ^2)$. From a physical point of view, the fact that $M(0)=0$ means that the base tension of the string is zero, a very realistic model.

Throughout the paper we assume that the nonlinear term $f:\Omega \times {\mathbb {R}}\rightarrow {\mathbb {R}}$is a continuous function, with $f(x,t)\equiv 0$ for $t\le 0$ and $x\in \Omega$. In the following, we also require the following assumptions $(f_1)-(f_6)$:

$(f_1)$ :

there exists $\alpha _0>0$ such that,

$$\begin{aligned} \lim _{t\rightarrow \infty }f(x,t)\exp (-\alpha |t|^{N/(N-s)})= {\left\{ \begin{array}{ll} 0,\ \ \ &{}\quad \forall \alpha >\alpha _0,\\ \infty ,\ \ &{}\quad \forall \alpha <\alpha _0, \end{array}\right. } \end{aligned}$$

uniformly in $\Omega$;

$(f_2)$ :

there exist constants $t_0, K_0>0$ such that

$$\begin{aligned} F(x,t)\le K_0 f(x,t),\quad \forall \ (x,t)\in \Omega \times [t_0,\infty ), \end{aligned}$$

where $F(x,t)=\int _0^t f(x,\tau )d\tau$;

$(f_3)$ :

$\lim _{t\rightarrow 0^+}\displaystyle \frac{f(x,t)}{t^{\frac{\theta N}{s}-1}}<\theta {\mathscr {M}}(1)\lambda ^*$ uniformly for x in $\Omega$, where

$$\begin{aligned} \lambda ^*:=\inf _{u\in W_0^{s,N/s}(\Omega ){\setminus }\{0\}} \frac{\Vert u\Vert ^{N\theta /s}}{\Vert u\Vert _{L^{N/s}(\Omega )}^{N/s}}>0, \end{aligned}$$

see [42] for more details;

$(f_4)$ :

there exists $\beta _0>\frac{sM\left( \frac{\alpha _{N,s}}{\alpha _0}\right) \frac{\alpha _{N,s}}{\alpha _0}}{NR_0}$ such that

$$\begin{aligned} \lim _{t\rightarrow \infty }\frac{f(x,t)t}{\exp (\alpha _0 |t|^{\frac{N}{N-s}})}\ge \beta _0\ \ \ \mathrm{uniformly\ in}\ x\in \Omega , \end{aligned}$$

where $R_0$ is the radius of the largest open ball contained in $\Omega$;

$(f_5)$ :

for each $x\in \Omega$, $\displaystyle \frac{f(x,t)}{t^{\frac{\theta N}{s}-1}}$ is increasing for $t>0$;

$(f_6)$ :

there exists $0\le \psi \in W_0^{s,N/s}(\Omega )$ such that $\Vert \psi \Vert =1$ and

$$\begin{aligned} \sup _{t\in {\mathbb {R}}^+}\left( \frac{s}{N}{\mathscr {M}}(t^{N/s})-\int _\Omega F(x,t\psi )dx\right) <\frac{s}{N}{\mathscr {M}}\left( \frac{\alpha _{N,s}}{\alpha _0}\right) . \end{aligned}$$

An example of function f satisfying $(f_1)-(f_5)$ with $\alpha _0=1$ is given by

$$\begin{aligned} F(x,t)=\frac{s{\mathscr {M}}(1)\lambda ^*}{2N}t^{\frac{N\theta }{s}} +t^{\frac{N\theta }{s}-\frac{s}{N-s}}\left[ \exp (t^{N/(N-s)})-1\right] ,\quad \ \forall t>0. \end{aligned}$$

In fact, by a simple calculation, one can verify that

$$\begin{aligned} f(x,t)&=\frac{\theta {\mathscr {M}}(1)\lambda ^*}{2}t^{\frac{N\theta }{s}-1} +\left( \frac{N\theta }{s}-\frac{s}{N-s}\right) t^{\frac{N\theta }{s}-\frac{N}{N-s}} \left[ \exp (t^{N/(N-s)})-1\right] \\&\quad +\frac{N}{N-s}t^{\frac{N\theta }{s}}\exp (t^{N/(N-s)}), \end{aligned}$$

Moreover, $f(x,t)/t^{N\theta /s-1}$ is increasing for all $t>0$, and

$$\begin{aligned} \lim _{t\rightarrow \infty }\frac{F(x,t)}{f(x,t)}=0,\quad \ \ \lim _{t\rightarrow \infty }\frac{f(x,t)t}{\exp (t^{N/(N-s)})} =\infty . \end{aligned}$$

uniformly in $x\in \Omega$.

Remark 1.2

We say that f satisfies exponential critical growth at $\infty$ if $(f_1)$ holds. Moreover, we observe that $(f_2)$ implies

$$\begin{aligned} F(x,t)\ge F(x,t_0)\exp (K_0(t-t_0)),\quad \ \ \forall (x,t)\in \Omega \times [t_0,\infty ), \end{aligned}$$

which is reasonable for the nonlinear term f(x, t) behaving like $\exp (\alpha _0|t|^{N/(N-s)})$ at infinity. Moreover, by $(f_2)$, for each $\mu >0$, there exists $C_\mu >0$ such that

$$\begin{aligned} \mu F(x,t)\le tf(x,t),\quad \ \forall (x,t)\in \Omega \times [C_\mu ,\infty ). \end{aligned}$$

(1.10)

Remark 1.3

If $N=1$ and $s=1/2$, then $(f_4)$ reduces to:

$(f_4^\prime )$ :

there exists $\beta _0>\frac{M\left( \frac{2\pi ^2}{\alpha _0}\right) \frac{2\pi ^2}{\alpha _0}}{b-a}$ such that

$$\begin{aligned} \lim _{t\rightarrow \infty }\frac{f(x,t)t}{\exp (\alpha _0 t^{2})}\ge \beta _0\ \ \ \mathrm{uniformly\ in}\ x\in \Omega :=(a,b), \end{aligned}$$

where $a<b$.

Remark 1.4

Using $(f_5)$ and the similar discussion as Remark 1.1, one can deduce that for each $x\in \Omega$,

$$\begin{aligned} tf(x,t)-\frac{N\theta }{s} F(x,t) \ \ \mathrm{is\ increasing\ for}\ t>0. \end{aligned}$$

In particular, $tf(x,t)-\frac{ N\theta }{s}F(x,t)\ge 0$ for all $(x,t)\in \Omega \times [0,\infty )$.

Definition 1.1

We say that $u\in W_0^{s,N/s}(\Omega )$ is a (weak) solution of problem (1.1), if there holds

$$\begin{aligned} M(\Vert u\Vert ^{N/s})\langle u,\varphi \rangle _{s,N/s} =\int _{\Omega }f(x,u)\varphi dx \end{aligned}$$

for all $\varphi \in W_0^{s,N/s}(\Omega )$, where

$$\begin{aligned} \langle u,\varphi \rangle _{s,N/s}=\iint _{{\mathbb {R}}^{2N}}\frac{|u(x)-u(y)|^{\frac{N}{s}-2} (u(x)-u(y))\cdot (\varphi (x)-\varphi (y))}{|x-y|^{2N}} dxdy. \end{aligned}$$

For general $N\ge 1$ and $s\in (0,1)$, we get the following result.

Theorem 1.1

If M fulfills $(M_1)$–$(M_3)$ and f satisfies $(f_1)$–$(f_6)$, then problem (1.1) admits a nontrivial nonnegative ground state solution in $W_{0}^{s,N/s}(\Omega )$ with positive energy.

If we consider the special case $s=1/2$ and $N=1$, then the assumption $(f_6)$ can be removed. Hence we get the second result as follows.

Theorem 1.2

Let $s=1/2$, $N=1$ and $\Omega =(a,b)$ with $a<b$. If M fulfills $(M_1)$–$(M_3)$ and f satisfies $(f_1)$–$(f_5)$, then problem (1.1) admits a nontrivial nonnegative ground state solution in $W_0^{1/2,2}(a,b)$ with positive energy.

Finally, we consider a special case of f(x, u), that is, we study the following problem:

$$\begin{aligned} {\left\{ \begin{array}{ll} M\left( \Vert u\Vert ^{N/s}\right) (-\Delta )^{s}_{N/s}u=\lambda |u|^{q-2}u\exp (\alpha _0|u|^{N/(N-s)})+h(x)|u|^{r-2}u\,\, \ &{}\mathrm{in}\ \Omega ,\\ u=0\ \ \ \ &{}\mathrm{in}\ {\mathbb {R}}^N{\setminus } \Omega , \end{array}\right. } \end{aligned}$$

(1.11)

where $M:[0,\infty )\rightarrow [0,\infty )$ is a continuous function satisfying $(M_1)$ and (1.9), $\theta N/s<q<\infty$, $1<r<N/s$ and $0\le h\in L^{\frac{N}{N-sr}}(\Omega )$.

Set

$$\begin{aligned} t_*=\left( \frac{\alpha _{N,s}}{2\alpha _0}\right) ^{(N-s)/N} \end{aligned}$$

and define

$$\begin{aligned} g(t)=\frac{{\mathscr {M}}(t_*)}{t_*^{\theta }}t^{\frac{N\theta }{s}}-\frac{\lambda }{q}C_{N,s}|\Omega |t^q\ \ \mathrm{for\ all}\ t\ge 0. \end{aligned}$$

Clearly, g has positive maximum attained at

$$\begin{aligned} t_{\mathrm{max}}=\left( \frac{{\mathscr {M}}(t_*)N\theta }{sC_{N,s}t_*^\theta \lambda }\right) ^{\frac{s}{sq-N\theta }}>0, \end{aligned}$$

being $q>N\theta /s.$ Set

$$\begin{aligned} \Lambda ^*=\frac{{\mathscr {M}}(t_*)N\theta }{sC_{N,s}t_*^{\theta }{\widetilde{\rho }}_1^{q-\frac{N\theta }{s}}}. \end{aligned}$$

and denote by $C_r$ the embedding constant from $W_0^{s,N/s}(\Omega )$ to $L^r(\Omega )$. Here $\widetilde{\rho _1}\in (0, t_*)$ is a constant. Assume that

$$\begin{aligned} \Vert h\Vert _{L^{\frac{N}{N-sr}}(\Omega )}<\frac{rg(t_{\mathrm{max}})}{C_r^rt_{\mathrm{max}}^r}. \end{aligned}$$

(1.12)

Now we give the third result as follows.

Theorem 1.3

Assume M fulfills $(M_1)$ and (1.9). If (1.12) holds, then for all $\lambda >\Lambda _*$ problem (1.11) admits a nontrivial nonnegative solution $u_\lambda$ in $W_{0}^{s,N/s}(\Omega )$ with negative energy. Moreover, $\Vert u_\lambda \Vert \rightarrow 0$ as $\lambda \rightarrow \infty .$

To get the existence of ground state solutions for problem (1.1), we first apply the mountain pass lemma without Palais–Smale condition to get a Palais–Smale sequence $\{u_n\}$ with $I(u_n)\rightarrow c_*>0$ and $I^\prime (u_n)\rightarrow 0$. The main difficulty is how one can get the strong convergence of $\{u_n\}$ and how to prove that the limit of $\{u_n\}$ is the ground state solution of problem (1.1).

To the best of our knowledge, Theorems 1.1–1.3 are the first results for the Kirchhoff-type problems involving critical Trudinger–Moser nonlinearities in the fractional setting.

The rest of the paper is organized as follows. In Sect. 2, we give some necessary properties for the functional setting. In Sect. 3, we verify that the associated functional satisfies the mountain pass geometry and give an estimate for the level value. In Sect. 4, we obtain the existence of ground state solution for problem (1.1). In Sect. 5, a nonnegative and nontrivial solution for problem (1.1) with negative energy is obtained by using Ekeland’s variational principle.

2 Preliminary results

We first provide some basic functional setting that will be used in the next sections.

Theorem 2.1

[10, Theorem 6.10] Let $s\in (0,1)$ and $N\ge 1$. Let $\Omega \subset {\mathbb {R}}^N$ be a bounded domain with Lipschitz boundary. Then there exists a positive constant $C=C(N,s,\Omega )$ such that for any $u\in W_0^{s,N/s}(\Omega )$ there holds

$$\begin{aligned} \Vert u\Vert _{L^q(\Omega )}\le C\Vert u\Vert \end{aligned}$$

for any $q\in [1,\infty )$, i.e. the space $W_0^{s,N/s}(\Omega )$ is continuously embedded in $L^q(\Omega )$ for any $q\in [1,\infty )$.

To prove the existence of weak solutions of (1.1), we shall use the following embedding theorem.

Theorem 2.2

(Compact embedding) let $s\in (0,1)$ and $N\ge 1$. Assume that $\Omega$ is a bounded domain in ${\mathbb {R}}^N$ with Lipschitz boundary $\partial \Omega$. Then, for any $\nu \ge 1$ the embedding $W_0^{s,N/s}(\Omega )\hookrightarrow \hookrightarrow L^\nu (\Omega )$ is compact.

Proof

By [10, Theorem 7.1], we know that the embedding $W_0^{s,N/s}(\Omega )\hookrightarrow \hookrightarrow L^\nu (\Omega )$ is compact for any $\nu \in [1,N/s]$. Next we prove that this result holds true for the case $\nu \in (N/s,\infty )$. Let $\{u_n\}$ is a bounded sequence in $W_0^{s,N/s}(\Omega )$. Then there exist a subsequence of $\{u_n\}$ (still denoted by $\{u_n\}$) and $u\in W_0^{s,N/s}(\Omega )$ such that $u_n\rightarrow u$ in $L^{N/s}(\Omega )$.

For any $\nu >N/s$, by the Hölder inequality we have

$$\begin{aligned} \int _{\Omega }|u_n-u|^\nu dx&=\int _{\Omega }|u_n-u|^{\frac{\sigma N}{s}}|u_n-u|^{\nu -\frac{\sigma N}{s}}dx\nonumber \\&\le \left( \int _{\Omega }|u_n-u|^{\frac{ N}{s}}dx\right) ^{\frac{1}{\sigma }} \left( \int _\Omega |u_n-u|^{(\nu -\frac{\sigma N}{s})\frac{\sigma }{\sigma -1}}dx\right) ^{\frac{\sigma -1}{\sigma }}\nonumber \\&\le \left( \int _{\Omega }|u_n-u|^{\frac{ N}{s}}dx\right) ^{\frac{1}{\sigma }} \left( \int _\Omega |u_n-u|^{\frac{\sigma }{\sigma -1}}dx\right) ^{\frac{\nu -\frac{\sigma N}{s}}{\nu }}|\Omega |^{\frac{\sigma N}{s\nu }}, \end{aligned}$$

(2.1)

where $\sigma \in (0,1)$. Since $\sigma /(\sigma -1)>1$, it follows from Theorem 2.1 that

$$\begin{aligned} \left( \int _\Omega |u_n-u|^{\frac{\sigma }{\sigma -1}}dx\right) ^{\frac{\nu -\frac{\sigma N}{s}}{\nu }}|\Omega |^{\frac{\sigma N}{s\nu }}\le C, \end{aligned}$$

which together with (2.1) yields that

$$\begin{aligned} \int _{\Omega }|u_n-u|^\nu dx \le C\left( \int _{\Omega }|u_n-u|^{\frac{ N}{s}}dx\right) ^{\frac{1}{\sigma }}, \end{aligned}$$

In view of $u_n\rightarrow u$ in $L^{N/s}(\Omega )$, we get $u_n\rightarrow u$ in $L^{\nu }(\Omega )$. $\square$

To study solutions of problem (1.1), we define the associated functional $I:W_{0}^{s,N/s}(\Omega )\rightarrow {\mathbb {R}}$ as

$$\begin{aligned}&I(u)=\frac{s}{N}{\mathscr {M}}(\Vert u\Vert ^{N/s}) -\int _{\Omega }F(x,u)dx. \end{aligned}$$

Since f is continuous and satisfies $(f_1)$ and $(f_3)$, for any $\varepsilon \in (0,\frac{s{\mathscr {M}}(1)}{N}\lambda ^*)$, $\alpha >\alpha _0$, and $q\ge 1$, there exists $C=C(\varepsilon ,\alpha ,q)>0$ such that

$$\begin{aligned} |F(x,t)|\le \left( \frac{s{\mathscr {M}}(1)}{N}\lambda ^*-\varepsilon \right) |t|^{\theta N/s}+C|t|^{q}\exp (\alpha |t|^{\frac{N}{N-s}}),\quad \ \forall (x,t)\in \Omega \times {\mathbb {R}}. \end{aligned}$$

(2.2)

Using (2.2) and the fractional Trudinger–Moser inequality, one can verify that I is well defined, of class $C^1(W_{0}^{s,N/s}(\Omega ),{\mathbb {R}})$ and

$$\begin{aligned} \langle I^\prime (u),v\rangle = M(\Vert u\Vert ^{N/s})\langle u,v\rangle _{s,N/s} -\int _{\Omega }f(x,u)vdx, \end{aligned}$$

for all $u,v\in W_{0}^{s,N/s}(\Omega )$. From now on, $\langle \cdot ,\cdot \rangle$ denotes the duality pairing between $\big (W_{0}^{s,N/s}(\Omega )\big )'$ and $W_{0}^{s,N/s}(\Omega )$. Clearly, the critical points of I are exactly the weak solutions of problem (1.1). Moreover, the following lemma shows that any nontrivial weak solution of problem (1.1) is nonnegative.

Lemma 2.1

Any nontrivial solutions of problem (1.1) is nonnegative.

Proof

Let $u\in W_{0}^{s,N/s}(\Omega ){\setminus }\{0\}$ be a critical point of functional I. Clearly, $u^-=\max \{-u,0\}\in W_{0}^{s,N/s}(\Omega )$. Then $\langle I^\prime (u),-u^-\rangle =0$, i.e.

$$\begin{aligned} M(\Vert u\Vert ^{\frac{N}{s}})\langle u,u^-\rangle _{s,N/s} =\int _{\Omega }f(x,u)(-u^-)dx. \end{aligned}$$

Observe that for a.e. $x, y\in {\mathbb {R}}^N$,

$$\begin{aligned}&|u(x)-u(y)|^{\frac{N}{s}-2}(u(x)-u(y))(-u^-(x)+u(y)^-)\\&\quad =|u(x)-u(y)|^{\frac{N}{s}-2}u^+(x)u^-(y) +|u(x)-u(y)|^{\frac{N}{s}-2}u^-(x)u^+(y)\\&\qquad +|u^-(x)-u^-(y)|^{\frac{N}{s}}\\&\quad \ge |u^-(x)-u^-(y)|^{\frac{N}{s}}, \end{aligned}$$

and $f(x,u)u^-=0$ a.e. $x\in \Omega$ by assumption. Hence,

$$\begin{aligned} M(\Vert u\Vert ^{\frac{N}{s}})\Vert u^-\Vert ^{\frac{N}{s}}\le 0. \end{aligned}$$

This, together with $\Vert u\Vert >0$ and $(M_1)$, implies that $u^-\equiv 0$, that is $u\ge 0$ a.e. in $\Omega$. This ends the proof. $\square$

3 Mountain pass geometry and minimax estimates

Let us recall that I satisfies the $(PS)_c$ condition in $W_0^{s,N/s}(\Omega )$, if any $(PS)_c$ sequence $\{u_n\}\subset W_0^{s,N/s}(\Omega )$, namely a sequence such that $I(u_n)\rightarrow c$ and $I^\prime (u_n)\rightarrow 0$ as $n\rightarrow \infty$, admits a strongly convergent subsequence in $W_0^{s, N/s}(\Omega )$.

In the sequel, we shall make use of the following general mountain pass theorem (see [2, 36]).

Theorem 3.1

Let E be a real Banach space and $J\in C^1(E,{\mathbb {R}})$ with $J(0)=0$. Suppose that

• (i) there exist $\rho ,\alpha >0$ such that $J(u)\ge \alpha$ for all $u\in E$, with $\Vert u\Vert _{E}=\rho$;

• (ii) there exists $e\in E$ satisfying $\Vert e\Vert _{E}>\rho$ such that $J(e)<0$.

Define $\Gamma =\{\gamma \in C^1([0,1];E):\gamma (0)=1,\gamma (1)=e\}$. Then

$$\begin{aligned} c=\inf _{\gamma \in \Gamma }\max _{0\le t\le 1} J(\gamma (t))\ge \alpha \end{aligned}$$

and there exists a $(PS)_c$ sequence $\{u_n\}_n\subset E$.

To find a mountain pass solution of problem (1.1), let us first verify the validity of the conditions of Theorem 3.1.

Lemma 3.1

(Mountain Pass Geometry 1) Assume that $(f_1)-(f_3)$ hold. Then there exist $\rho >0$ and $\kappa >0$ such that $I(u)\ge \kappa$ for any $u\in W_{0}^{s,N/s}(\Omega )$, with $\Vert u\Vert =\rho$.

Proof

Applying (2.2) and the definition of $\lambda ^*$ in $(f_3)$, for any $\varepsilon \in (0,\lambda ^*)$ and $q>\theta N/s$ we have

$$\begin{aligned} \int _\Omega F(x,u)dx&\le \left( \frac{s{\mathscr {M}}(1)}{N}\lambda ^*-\varepsilon \right) \int _\Omega |u|^{\frac{N\theta }{s}}dx+C \int _\Omega |u|^q\exp \left( \alpha |u|^{\frac{N}{N-s}}\right) dx\nonumber \\&\le \left( \frac{s{\mathscr {M}}(1)}{N}-\frac{\varepsilon }{\lambda ^*}\right) \Vert u\Vert ^{\frac{N\theta }{s}}+C\Vert u\Vert _{L^{2q}(\Omega )}^q \nonumber \\&\quad \cdot \left( \int _\Omega \exp (2\alpha \Vert u\Vert ^{\frac{N}{N-s}}(u/\Vert u\Vert )^{\frac{N}{N-s}})dx\right) ^{1/2}, \end{aligned}$$

(3.1)

for all $u\in W_{0}^{s,N/s}(\Omega )$.

On the other hand, (1.9) gives

$$\begin{aligned} {\mathscr {M}}(t)\ge {\mathscr {M}}(1)t^\theta \quad \text{ for } \text{ all } t\in [0,1]. \end{aligned}$$

(3.2)

Thus, by using (3.1), (3.2) and the Hölder inequality, we obtain for all $u\in W_{0}^{s,N/s}(\Omega )$, with $\Vert u\Vert \le \rho _1\le 1$ small enough,

$$\begin{aligned} I(u)&\ge \frac{\varepsilon }{\lambda ^*}\Vert u\Vert ^{\frac{N\theta }{s}}-C\Vert u\Vert _{L^{2q}(\Omega )}^q \left( \int _\Omega \exp \left( 2\alpha \Vert u\Vert ^{\frac{N}{N-s}}(u/\Vert u\Vert )^{\frac{N}{N-s}}\right) dx\right) ^{1/2}\nonumber \\&\ge \frac{\varepsilon }{\lambda ^*}\Vert u\Vert ^{\frac{N\theta }{s}}-C_1\Vert u\Vert ^q \left( \int _\Omega \exp \left( 2\alpha \rho _1^{{\frac{N}{N-s}}}(u/\Vert u\Vert )^{{\frac{N}{N-s}}}\right) dx\right) ^{1/2}. \end{aligned}$$

Choosing $2\alpha \rho _1^{{N/(N-s)}}\le \alpha _{N,s}$ and using the fractional Trudinger–Moser inequality, we get

$$\begin{aligned} I(u)\ge \Vert u\Vert ^{\frac{\theta N}{s}}\left( \frac{\varepsilon }{\lambda ^*}-C_2\Vert u\Vert ^{q-\frac{\theta N}{s}}\right) . \end{aligned}$$

Now fix $\varepsilon >0$ and choose $0<\rho<\rho _1<1$ such that $\frac{\varepsilon }{\lambda ^*}-C_2\rho ^{q-\frac{\theta N}{s}}>0$. Thus, $I(u)\ge \kappa := \rho ^{\theta N/s}\left( \frac{\varepsilon }{\lambda ^*}-C_2\rho ^{q-\frac{\theta N}{s}}\right) >0$ for all $u\in W_{0}^{s,N/s}(\Omega )$, with $\Vert u\Vert =\rho$. $\square$

Lemma 3.2

(Mountain Pass Geometry 2) Assume that $(M_2)$ and $(f_2)$ hold. Then there exists a nonnegative function $e\in C_0^\infty ({\mathbb {R}})$, such that $I(e)<0$ and $\Vert e\Vert \ge \rho$ for all $\lambda \in {\mathbb {R}}^+$.

Proof

It follows from (1.9) that

$$\begin{aligned} {\mathscr {M}}(t)\le {\mathscr {M}}(1)t^\theta \quad \text{ for } \text{ all } t\ge 1. \end{aligned}$$

(3.3)

On the other hand, taking $\mu >\theta N/s$ and using (1.10), we obtain that there exist positive constants $C_3,C_4>0$ such that

$$\begin{aligned} F(x,t)\ge C_3 t^{\mu }-C_4\quad \ \ \mathrm{for\ all}\ x\in \Omega \quad \ \mathrm{and}\quad t\ge 0. \end{aligned}$$

(3.4)

Now, choose $v_0\in W_{0}^{s,N/s}(\Omega )$ with $v_0\ge 0$ and $\Vert v_0\Vert =1$. Then for all $t\ge 1$, we have

$$\begin{aligned} I(tv_0)&\le \frac{s}{N}{\mathscr {M}}(1)t^{\frac{\theta N}{s}}\Vert v_0\Vert ^{\frac{\theta N}{s}} -C_3t^{\mu }\int _{\Omega }|v_0|^\mu dx+C_4|\Omega |. \end{aligned}$$

Hence, $I(tu_0)\rightarrow -\infty$ as $t\rightarrow \infty$, thanks to $\theta N/s<\mu$. The lemma is proved by taking $e=Tv_0$, with $T>0$ so large that $\Vert e\Vert \ge \rho$ and $I(e)<0$. $\square$

By Lemmas 3.1 and 3.2 and the mountain pass theorem (Theorem 3.1), there exists a $(PS)_c$ sequence $\{u_n\}\subset W_{0}^{s,N/s}(\Omega )$ such that

$$\begin{aligned} I(u_n)\rightarrow c_* \quad \ \ \mathrm{and}\quad \ I^\prime (u_n)\rightarrow 0\quad \ \ \mathrm{as}\ n\rightarrow \infty , \end{aligned}$$

where

$$\begin{aligned} c_{*}=\inf _{\gamma \in \Gamma }\max _{t\in [0,1]}I(\gamma (t)), \end{aligned}$$

(3.5)

where $\Gamma =\left\{ \gamma \in C([0,1];W_{0}^{s,N/s}(\Omega ):\gamma (0)=0,\ I(\gamma (1))=e\right\} .$ Obviously, $c_*>0$ by Lemma 3.1. Moreover, under assumption $(f_6)$, we have the following estimate.

Lemma 3.3

Assume that $(M_2)$, $(f_2)$ and $(f_6)$ hold. Then

$$\begin{aligned} c_*<\frac{s}{N}{\mathscr {M}}\left( \frac{\alpha _{N,s}}{\alpha _0}\right) . \end{aligned}$$

Proof

Since $\psi \ge 0$ in $\Omega$ and $\Vert \psi \Vert =1$, as in the proof of Lemma 3.2, we deduce that $I(t\psi )\rightarrow -\infty$ as $t\rightarrow \infty$. Consequently, using assumption $(f_6)$, one can deduce that

$$\begin{aligned} c_*\le \max _{t\ge 0} I(t\psi ) \le \sup _{t\in {\mathbb {R}}^+}\left( \frac{s}{N}{\mathscr {M}}(t^{N/s})-\int _\Omega F(x,t\psi )dx\right) <\frac{s}{N}{\mathscr {M}}\left( \frac{\alpha _{N,s}}{\alpha _0}\right) . \end{aligned}$$

This proves the lemma. $\square$

Actually, for the case $N=1$ and $s=1/2$, assumption $(f_6)$ naturally holds true. To get more precise information about the minimax level $c_*$ in this case, let us consider the following Moser functions which have been used in [31]:

$$\begin{aligned} {\widetilde{G}}_n(x)= \frac{1}{\sqrt{2}\pi }{\left\{ \begin{array}{ll} |\ln n|^{\frac{1}{2}}\ \ &{}\quad \mathrm{if}\ |x|\le \frac{1}{n},\\ \frac{|\ln |x||}{|\ln n|^{\frac{1}{2}}}\ \ &{}\quad \mathrm{if}\ \frac{1}{n}<|x|<1,\\ 0\ \ &{}\quad \mathrm{if}\ |x|\ge 1. \end{array}\right. } \end{aligned}$$

Let $\Omega :=(a,b)$, $x_0=\frac{a+b}{2}$ and $R_0=\frac{b-a}{2}$. It is standard verify that the functions

$$\begin{aligned} G_n(x)={\widetilde{G}}_n\left( \frac{x-x_0}{R_0}\right) ,\quad \ \ x\in \Omega , \end{aligned}$$

belongs to $W_0^{\frac{1}{2},2}(\Omega )$. Moreover, $\lim _{n\rightarrow \infty }\Vert G_n\Vert =1$ and the support of $G_n$ is contained in interval $(x_0-R_0,x_0+R_0)$, see [31].

Lemma 3.4

Assume that $(M_1), (M_2)$, $(f_1)$ and $(f_4^\prime )$ hold. Then there exists $n>0$ such that

$$\begin{aligned} \max _{t\ge 0}I(tG_n)=\max _{t\ge 0}\left\{ \frac{1}{2}{\mathscr {M}}(t^{2}\Vert G_n\Vert ^{2})-\int _{\Omega }F(x,tG_n)dx\right\} < \frac{1}{2}{\mathscr {M}}\left( \frac{2\pi ^2}{\alpha _0}\right) . \end{aligned}$$

Proof

Arguing by contradiction, we assume that

$$\begin{aligned} \max _{t\ge 0}I(tG_n)\ge \frac{1}{2}{\mathscr {M}}\left( \frac{2\pi ^2}{\alpha _0}\right) . \end{aligned}$$

(3.6)

Since the functional I possesses the mountain pass geometry, for each n there exists $t_n>0$ such that

$$\begin{aligned} I(t_nG_n)=\max _{t\ge 0} I(tG_n). \end{aligned}$$

In view of the fact that $F(x,t)\ge 0$ for all $(x,t)\in \Omega \times {\mathbb {R}}$, one can deduce that

$$\begin{aligned} {\mathscr {M}}\left( t_n^{2}\Vert G_n\Vert ^{2}\right) \ge {\mathscr {M}}\left( \frac{2\pi ^2}{\alpha _0}\right) . \end{aligned}$$

Since $M:[0,\infty )\rightarrow [0,\infty )$ is a nonnegative function, ${\mathscr {M}}$ is a nondecreasing function. Thus, we get

$$\begin{aligned} t_n^{2}\Vert G_n\Vert ^{2}\ge \frac{2\pi ^2}{\alpha _0}. \end{aligned}$$

Hence,

$$\begin{aligned} \liminf _{n\rightarrow \infty }\ t_n^{2}\ge \frac{2\pi ^2}{\alpha _0}. \end{aligned}$$

(3.7)

On the other hand,

$$\begin{aligned} \frac{d}{dt}I(tG_n)\mid _{t=t_n}=0, \end{aligned}$$

which implies that

$$\begin{aligned} M(t_n^{2}\Vert G_n\Vert ^{2})t_n^{2}\Vert G_n\Vert ^{2}=\int _\Omega f(x,t_nG_n)t_nG_ndx\ge \int _{B_{R_0}(x_0)}f(x,t_nG_n) t_nG_n dx. \end{aligned}$$

(3.8)

Using change of variable, we have

$$\begin{aligned} M(t_n^{2}\Vert G_n\Vert ^{2})t_n^{2}\Vert G_n\Vert ^{2}&\ge R_0\int _{B_{1}(0)}f(R_0x+x_0,t_n{\widetilde{G}}_n)t_n{\widetilde{G}}_ndx\\&\ge R_0\int _{B_{\frac{1}{n}}(0)}f\left( R_0x+x_0,t_n\frac{1}{\sqrt{2}\pi }(\ln n)^{1/2}\right) t_n\frac{1}{\sqrt{2}\pi }(\ln n)^{1/2}dx. \end{aligned}$$

Note that (3.7) implies that

$$\begin{aligned} \frac{t_n}{\sqrt{2}\pi }(\ln n)^{1/2}\rightarrow \infty \quad \ \ \mathrm{as}\ n\rightarrow \infty . \end{aligned}$$

It follows from $(f_4)$ that given $\delta >0$ there exists $t_\delta >0$ such that

$$\begin{aligned} f(x,t)t\ge (\beta _0-\delta )\exp (\alpha _0t^{2}),\quad \ \ \forall (x,t)\in \Omega \times [t_\delta ,\infty ). \end{aligned}$$

(3.9)

Thus, there exists $n_0\in {\mathbb {N}}$ such that

$$\begin{aligned}&f\left( R_0x+x_0,t_n\frac{1}{\sqrt{2}\pi }(\ln n)^{1/2}\right) t_n\frac{1}{\sqrt{2}\pi }(\ln n)^{1/2}\\&\qquad \ge (\beta _0-\delta )\exp \left( \alpha _0t_n^2\frac{1}{2\pi ^2}\ln n\right) , \end{aligned}$$

for all $n\ge n_0$. Hence,

$$\begin{aligned} M(t_n^{2}\Vert G_n\Vert ^{2})t_n^{2}\Vert G_n\Vert ^{2}&\ge (\beta _0-\delta )R_0\exp \left( \alpha _0t_n^2\frac{1}{2\pi ^2}\ln n\right) \frac{2}{n}\nonumber \\&=(\beta _0-\delta )2R_0\exp \left( \alpha _0t_n^2\frac{1}{2\pi ^2}\ln n\right) \exp (-\ln n)\nonumber \\&=(\beta _0-\delta )2R_0\exp \left[ \left( \alpha _0t_n^{2}\frac{1}{2\pi ^2}-1\right) \ln n\right] . \end{aligned}$$

(3.10)

Next we show that $\{t_n\}$ is a bounded sequence in ${\mathbb {R}}$. If not, there exists a subsequence of $\{t_n\}$ still labeled by $\{t_n\}$, such that $t_n\rightarrow \infty$ as $n\rightarrow \infty$. From (3.3) and (3.7), we can conclude that

$$\begin{aligned} \frac{M(t_n^{2}\Vert G_n\Vert ^{2})t_n^{2}\Vert G_n\Vert ^{2}}{\exp \left[ \left( \alpha _0t_n^{2}\frac{1}{2\pi ^2}-1\right) \ln n\right] }\rightarrow 0\ \ \mathrm{as}\ n\rightarrow \infty , \end{aligned}$$

which contradicts (3.10). Thus,

$$\begin{aligned}\limsup _{n\rightarrow \infty }t_n^{2}\le \frac{2\pi ^2}{\alpha _0},\end{aligned}$$

which together with (3.7) yields that

$$\begin{aligned} t_n^{2}\rightarrow \frac{2\pi ^2}{\alpha _0} \end{aligned}$$

(3.11)

as $n\rightarrow \infty$.

Following some arguments as in [11, 13], we are going to estimate (3.8). In view of (3.9), for $0<\delta <\beta _0$ and $n\in {\mathbb {N}}$, we set

$$\begin{aligned} U_{n,\delta }:=\{x\in B_{R_0}(x_0):t_nG_n(x)\ge t_\delta \}\quad \ \ \mathrm{and}\quad \ \ V_{n,\delta }:=B_{R_0}(x_0){\setminus } U_{n,\delta }. \end{aligned}$$

Splitting the integral (3.8) on $U_{n,\delta }$ and $V_{n,\delta }$ and using (3.10), we deduce

$$\begin{aligned}&M(t_n^{2}\Vert G_n\Vert ^{2})t_n^{2}\Vert G_n\Vert ^{2}\nonumber \\&\quad \ge (\beta _0-\delta )\int _{B_{R_0}(x_0)}\exp (\alpha _0(t_nG_n)^{2})dx \nonumber \\&\qquad - (\beta _0-\delta )\int _{V_{n,\delta }} \exp (\alpha _0(t_nG_n)^{2})dx +\int _{V_{n,\delta }} f(x,t_nG_n)t_nG_ndx. \end{aligned}$$

(3.12)

Since $G_n(x)\rightarrow 0$ a.e. in $B_{R_0}(x_0)$, we deduce that the characteristic functions $\chi _{V_{n,\delta }}$ satisfies

$$\begin{aligned} \chi _{V_{n,\delta }}\rightarrow 1\ \ \mathrm{a.e.\ in}\ B_{R_0}(x_0)\ \ \quad \mathrm{as}\ n\rightarrow \infty . \end{aligned}$$

By $t_nG_n<t_\delta$ and the Lebesgue dominated convergence theorem, we have as $n\rightarrow \infty$

$$\begin{aligned} \int _{V_{n,\delta }}\exp (\alpha _0(t_nG_n)^{2})dx\rightarrow 2 R_0 \ \ \quad \mathrm{and}\quad \ \ \int _{V_{n,\delta }}f(x,t_nG_n)t_nG_ndx\rightarrow 0. \end{aligned}$$

(3.13)

The key point is to estimate the first term on the right hand of (3.12). By (3.7) and the definition of $G_n$, we have

$$\begin{aligned} \int _{B_{R_0}(x_0)}\exp (\alpha _0(t_nG_n)^{2})dx&\ge R_0\int _{B_{1/n}(0)}\exp (\ln n)dx\nonumber \\&\quad +R_0\int _{1/n<|x|<1}\exp \left[ \frac{|\ln |x||^{2}}{\ln n}\right] dx\nonumber \\&=2R_0+R_0\int _{1/n<|x|<1}\exp \left[ \frac{|\ln |x||^{2}}{\ln n}\right] dx\nonumber \\&\ge 2R_0+2R_0\left( 1-\frac{1}{n}\right) , \end{aligned}$$

(3.14)

for n sufficiently large. Inserting (3.13) and (3.14) in (3.12) and using (3.10), we arrive at

$$\begin{aligned} M\left( \frac{2\pi ^2}{\alpha _0}\right) \frac{2\pi ^2}{\alpha _0} \ge (\beta _0-\delta )2R_0,\quad \ \ \forall \delta \in (0,\beta _0). \end{aligned}$$

Letting $\delta \rightarrow 0^+$, we obtain

$$\begin{aligned} \beta _0\le \frac{M\left( \frac{2\pi ^2}{\alpha _0}\right) \frac{2\pi ^2}{\alpha _0}}{2R_0}, \end{aligned}$$

which contradicts $(f_4^\prime )$. Therefore, the lemma is proved. $\square$

By Lemma 3.4, we obtain the desired estimate for the level $c_*$.

Lemma 3.5

Assume $(M_1)-(M_2)$ and $(f_3)$ hold. Then

$$\begin{aligned} c_*<\frac{1}{2}{\mathscr {M}}\left( \frac{2\pi ^2}{\alpha _0}\right) . \end{aligned}$$

Proof

Since $G_n\ge 0$ in $\Omega$ and $\Vert G_n\Vert \rightarrow 1$, as in the proof of Lemma 3.2, we deduce that $I(tG_n)\rightarrow -\infty$ as $t\rightarrow \infty$. Consequently,

$$\begin{aligned} c_*\le \max _{t\ge 0} I(tG_n),\quad \ \ \forall n\in {\mathbb {N}}. \end{aligned}$$

Thus, the desired result follows by using Lemma 3.4. $\square$

Consider the Nehari manifold associated to the functional I, that is,

$$\begin{aligned} {\mathcal {N}}=\left\{ u\in W_0^{s,N/s}(\Omega ){\setminus }\{0\}:\langle I^\prime (u),u\rangle =0\right\} \end{aligned}$$

and $c^*:=\inf _{u\in {\mathcal {N}}}I(u)$.

The next result is crucial in our arguments to get the existence of a ground state solution for (1.1).

Lemma 3.6

Assume that $(M_3)$ and $(f_3)$ are satisfied. Then $c_*\le c^*$.

Proof

For any $u\in {\mathcal {N}}$, we define $h:[0,+\infty )\rightarrow {\mathbb {R}}$ by $h(t)=I(tu)$. clearly, h is differentiable and

$$\begin{aligned} h^\prime (t)=\langle I^\prime (tu),u\rangle =M\left( t^{\frac{N}{s}}\Vert u\Vert ^{\frac{N}{s}}\right) t^{\frac{N}{s}-1}\Vert u\Vert ^{\frac{N}{s}} -\int _\Omega f(x,tu)udx,\ \ \quad \forall t\ge 0. \end{aligned}$$

It follows from $\langle I^\prime (u),u\rangle =0$ that

$$\begin{aligned} M\left( \Vert u\Vert ^{\frac{N}{s}}\right) \Vert u\Vert ^{\frac{N}{s}} =\int _\Omega f(x,u)udx, \end{aligned}$$

which means that $h^\prime (1)=0$. Thus,

$$\begin{aligned} h^\prime (t)&=t^{\frac{\theta N}{s}-1}\Vert u\Vert ^{\theta \frac{N}{s}}\left[ \frac{M(t^{N/s}\Vert u\Vert ^{N/s})}{t^{(\theta -1)N/s}\Vert u\Vert ^{(\theta -1)N/s}}-\frac{M(\Vert u\Vert ^{N/s})}{\Vert u\Vert ^{(\theta -1)N/s}}\right] \\&\quad +t^{\frac{\theta N}{s}-1}\int _\Omega \left[ \frac{f(x,u)}{u^{\frac{\theta N}{s}-1}}-\frac{f(x,tu)}{(tu)^{\frac{\theta N}{s}-1}}\right] u^{\frac{\theta N}{s}}dx. \end{aligned}$$

Using $(M_2)$ and $(f_5)$, we get

$$\begin{aligned} h^\prime (t)\ge 0\quad \ \ \mathrm{for}\ 0\le t<1\quad \mathrm{and}\quad \ h^\prime (t)\le 0 \quad \ \ \mathrm{for}\ t>1. \end{aligned}$$

Thus, $h(1)=\max _{t\ge 0} h(t),$ which means

$$\begin{aligned} I(u)=\max _{t\ge 0} I(tu). \end{aligned}$$

Now we define $g:[0,1]\rightarrow W_0^{s,N/s}(\Omega )$, $g(t)=tt_0u$, where $t_0$ is such that $I(t_0u)<0$. Clearly, $g\in \Gamma$ and therefore

$$\begin{aligned} c_*\le \max _{t\in [0,1]}I(g(t))\le \max _{t\ge 0} I(tu)=I(u). \end{aligned}$$

By the arbitrary of $u\in {\mathcal {N}}$, we get $c_*\le c^*$. Thus the proof is complete. $\square$

4 Proofs of Theorems 1.1 and 1.2

This section is devoted to the proof of our main result. We recall that a solution $u_0$ of problem (1.1) is a ground state if $I(u_0)=\inf _{u\in {\mathcal {A}}}I(u)$, where

$$\begin{aligned} {\mathcal {A}}:=\{u\in W_0^{s,N/s}(\Omega ){\setminus }\{0\}:I^\prime (u)=0\}. \end{aligned}$$

Since $c_*\le c^*\le I(u_0)$, in order to obtain a ground state $u_0$ for (1.1) it is enough to show that there is $u_0\in {\mathcal {A}}$ and $I(u_0)=c_*$. To this aim, we first give some useful lemmas.

Lemma 4.1

(The $(PS)_{c*}$ condition) Let $(M_1)-(M_3)$ and $(f_1)$–$(f_6)$ hold. Then the functional I satisfies the $(PS)_{c_*}$ condition.

Proof

From Lemmas 3.1 and 3.2, we deduce from Theorem 3.1 that there exists a sequence $\{u_n\}\subset W_0^{s,N/s}(\Omega )$ satisfying

$$\begin{aligned} I(u_n)\rightarrow c_*\quad \ \ \mathrm{and}\quad \ I^\prime (u_n)\rightarrow 0. \end{aligned}$$

We first show that $\{u_n\}$ is bounded in $W_0^{s,N/s}(\Omega )$. Arguing by contradiction, we assume that $\{u_n\}$ is unbounded in $W_0^{s,N/s}(\Omega )$. Then up to a subsequence, still labeled by $\{u_n\}$, $\Vert u_n\Vert \rightarrow \infty$ and $d:=\inf _{n\ge 1}\Vert u_n\Vert >0$. Using (1.9), $(M_1)$ and (1.10) with $\mu >\frac{N\theta }{s}$, we get

$$\begin{aligned} C+\Vert u_n\Vert&\ge I(u_n)-\frac{1}{\mu }\langle I^\prime (u_n),u_n\rangle \nonumber \\&\ge \left( \frac{s}{N\theta }-\frac{1}{\mu }\right) M(\Vert u_n\Vert ^{N/s})\Vert u_n\Vert ^{N/s} -\frac{1}{\mu }\int _{|u_n|\le t_mu}(\mu F(x,u_n)-f(x,u_n)u_n)dx\nonumber \\&\ge \left( \frac{s}{N\theta }-\frac{1}{\mu }\right) M(d^{N/s})\Vert u_n\Vert ^{N/s} -\frac{1}{\mu }C_\mu |\Omega |, \end{aligned}$$

(4.1)

where $C_\mu =\sup \{|f(x,t)t-\mu F(x,t)|:(x,t)\in \Omega \times [0,t_\mu ]\}$. Dividing (4.1) by $\Vert u_n\Vert ^{N/s}$ and letting $n\rightarrow \infty$, we get

$$\begin{aligned} 0\ge \left( \frac{s}{N\theta }-\frac{1}{\mu }\right) M(d^{N/s})>0, \end{aligned}$$

which is absurd. Thus, $\{u_n\}$ is bounded in $W_0^{s,N/s}(\Omega )$.

Next we show that $\{u_n\}$ has a convergence subsequence in $W_0^{s,N/s}(\Omega )$. Going if necessary to a subsequence, there exist $u\in W_0^{s,N/s}(\Omega )$ and $\xi \ge 0$ such that

$$\begin{aligned}&u_n\rightharpoonup u\ \ \mathrm{weakly\ in}\ W_0^{s,N/s}(\Omega ),\nonumber \\&u_n\rightarrow u\ \ \mathrm{strongly\ in}\ L^{\nu }(\Omega )( \nu \ge 1),\nonumber \\&u_n\rightarrow u\ \ \mathrm{a.e.\ in}\ \Omega \nonumber \\&\Vert u_n\Vert \rightarrow \xi . \end{aligned}$$

(4.2)

Here we have used the compact embedding from $W_0^{s,N/s}(\Omega )$ to $L^\nu (\Omega )$ for any $\nu \ge 1$, see Theorem 2.2.

We first show that

$$\begin{aligned} \lim _{n\rightarrow \infty }\int _{\Omega }|f(x,u_n)|dx =\int _{\Omega }|f(x,u)|dx. \end{aligned}$$

(4.3)

By $I^\prime (u_n)\rightarrow 0$ and $\{u_n\}$ is bounded in $W_0^{s,N/s}(\Omega )$, there exists $C>0$ such that

$$\begin{aligned} \int _\Omega |f(x,u_n)u_n|dx\le C. \end{aligned}$$

Since $f(x,u)\in L^1(\Omega )$, it follows that given $\varepsilon >0$ there is a $\delta >0$ such that

$$\begin{aligned} \int _{U}|f(x,u)|dx\le \varepsilon \ \ \mathrm{if}\ |U|\le \delta \end{aligned}$$

for all measurable subsets U of $\Omega$, where |U| denotes the Lebesgue measure of U. From $u\in L^1(\Omega )$, there exists $D_1>0$ such that

$$\begin{aligned} |\{x\in \Omega :|u(x)|> D_1\}|\le \delta . \end{aligned}$$

Let $D=\max \{D_1,C/\varepsilon \}$. Then we have

$$\begin{aligned}&\left| \int _{\Omega }|f(x,u_n)|dx-\int _\Omega |f(x,u)|dx\right| \\&\quad \le \int _{|u_n|> D}|f(x,u_n)|dx+\int _{|u|> D}|f(x,u)|dx\\&\qquad +\left| \int _{|u_n|\le D}|f(x,u_n)|dx-\int _{|u|\le D}|f(x,u)|dx\right| . \end{aligned}$$

By above results, we obtain

$$\begin{aligned} \int _{|u|> D}|f(x,u)|dx\le \varepsilon . \end{aligned}$$

Also, we deduce

$$\begin{aligned} \int _{|u_n|> D}|f(x,u_n)|dx= \int _{|u_n|> D}\frac{|f(x,u_n)u_n|}{|u_n|}dx \le \frac{C}{D}\le \varepsilon . \end{aligned}$$

Next we claim that as $n\rightarrow \infty$

$$\begin{aligned} \left| \int _{|u_n|\le D}|f(x,u_n)|dx-\int _{|u|\le D}|f(x,u)|dx\right| \rightarrow 0. \end{aligned}$$

Indeed, set $g_n(x)=|f(x,u_n)|\chi _{|u_n|\le D}-|f(x,u)|\chi _{|u|\le D}$, then $g_n\rightarrow 0$ a.e. in $\Omega$. Moreover, $|g_n|\le |f(x,u)|$ if $|u_n(x)|>D$; $|g_n|\le C+|f(x,u)|$ if $|u_n(x)|\le D$, where $C=\sup \{|f(x,t)|:x\in {\overline{\Omega }},|t|\le D\}$. Thus, the Lebesgue dominated convergence theorem yields the claim. Therefore, we prove (4.3).

By (4.3), we can get

$$\begin{aligned} \lim _{n\rightarrow \infty }\int _{\Omega }f(x,u_n)dx =\int _{\Omega }f(x,u)dx. \end{aligned}$$

(4.4)

Indeed, set $h_n(x)=|f(x,u_n)|-|f(x,u_n)-f(x,u)|$. Obviously, $h_n(x)\rightarrow |f(x,u)|$ a.e. in $\Omega$. Moreover,

$$\begin{aligned} |h_n(x)|\le |f(x,u)|. \end{aligned}$$

Thus, the Lebesgue dominated convergence theorem implies that

$$\begin{aligned} \lim _{n\rightarrow \infty }\int _\Omega (|f(x,u_n)|-|f(x,u_n)-f(x,u)|)dx =\int _\Omega |f(x,u)|dx, \end{aligned}$$

which means that

$$\begin{aligned} \lim _{n\rightarrow \infty }\int _\Omega |f(x,u_n)-f(x,u)|dx =0. \end{aligned}$$

Therefore, (4.4) holds true. By (4.4), $(f_2)$ and the generalized Lebesgue dominated convergence theorem, we have

$$\begin{aligned} \lim _{n\rightarrow \infty }\int _\Omega (F(x,u_n)-F(x,u))dx =0. \end{aligned}$$

(4.5)

Now, we assert that $u\ne 0$. Arguing by contradiction, we assume that $u=0$. Then, $\int _\Omega F(x,u_n)dx\rightarrow 0$ and $I(u_n)\rightarrow c_*$ gives that

$$\begin{aligned} \frac{s}{N}{\mathscr {M}}(\Vert u_n\Vert ^{N/s}) \rightarrow c_*<\frac{s}{N}{\mathscr {M}}\left( \frac{\alpha _{N,s}}{\alpha _0}\right) \end{aligned}$$

as $n\rightarrow \infty$. Thus, there exists $n_0\in {\mathbb {N}}$ and $\delta >0$ such that $\Vert u_n\Vert ^{N/s}<\delta <\frac{\alpha _{N,s}}{\alpha _0}.$ Choosing $q>1$ close to 1 and $\alpha >\alpha _0$ close to $\alpha _0$ such that we still have $q\alpha \Vert u_n\Vert ^{N/s}<\delta <\alpha _{N,s}$. Thus, it follows from (2.2) with $q=1$ that

$$\begin{aligned}&\left| \int _\Omega f(x,u_n)u_ndx\right| \\&\quad \le C\left( \int _\Omega |u_n|^{\theta N/s}dx+\int _\Omega |u_n|\exp (\alpha |u_n|^{N/s})dx\right) \\&\quad \le C(\Vert u_n\Vert ^{\theta N/s}+\Vert u_n\Vert _{L^{\frac{q}{q-1}}(\Omega )} \left( \int _{\Omega }\exp [q\alpha \Vert u_n\Vert ^{N/s}(u_n/\Vert u_n\Vert )^{N/s}]dx\right) ^{\frac{1}{q}}\\&\quad \le C\left( \Vert u_n\Vert ^{\theta N/s}+\Vert u_n\Vert _{L^{\frac{q}{q-1}}(\Omega )}\right) \rightarrow 0 \end{aligned}$$

as $n\rightarrow \infty$. Since $\{u_n\}$ is a bounded $(PS)_{c_*}$ sequence, we get

$$\begin{aligned} \langle I^\prime (u_n),u_n\rangle =M(\Vert u_n\Vert ^{N/s})\Vert u_n\Vert ^{N/s}-\int _{\Omega }f(x,u_n)u_ndx\rightarrow 0, \end{aligned}$$

which implies that

$$\begin{aligned} M(\Vert u_n\Vert ^{N/s})\Vert u_n\Vert ^{N/s}\rightarrow 0. \end{aligned}$$

From this and assumption $(M_1)$, we deduce $\Vert u_n\Vert \rightarrow 0$. Furthermore, we obtain $I(u_n)\rightarrow 0$, which contradicts the fact that $I(u_n)\rightarrow c_*>0$. Therefore, we must have $u\ne 0$. So that $\xi >0$.

We claim that $I(u)\ge 0$. Arguing by contradiction, we assume that $I(u)<0$. Set $z(t):=I(tu)$ for all $t\ge 0$. Then $z(0)=0$ and $z(1)<0$. Arguing as in the proof of Lemma 3.1, we can see that $z(t)>0$ for $t>0$ small enough. Hence there exists $t_0\in (0,1)$ such that

$$\begin{aligned} z(t_0)=\max _{t\in [0,1]}z(t),\ \ z^\prime (t_0) =\langle I^\prime (t_0u),u\rangle =0, \end{aligned}$$

which means that $t_0u\in {\mathcal {N}}$. Therefore, by Remarks 1.1 and 1.4, the semicontinuity of norm and Fatou’s lemma, we get

$$\begin{aligned} c_*\le c^*\le I(t_0u)&=I(t_0u)-\frac{s}{N\theta }\langle I^\prime (t_0u),t_0u\rangle \\&=\frac{s}{N}{\mathscr {M}}(\Vert t_0u\Vert ^{N/s})-\frac{ s}{N\theta } M(\Vert t_0u\Vert ^{N/s})\Vert t_0u\Vert ^{N/s}\\&\quad +\frac{s}{N\theta } \int _\Omega \left[ f(x,t_0u)t_0u-\frac{\theta N}{s}F(x,t_0u)\right] dx\\&<\frac{s}{N}{\mathscr {M}}(\Vert u\Vert ^{N/s})-\frac{ s}{N\theta } M(\Vert u\Vert ^{N/s})\Vert u\Vert ^{N/s}\\&\quad +\frac{ s}{N\theta } \int _\Omega \left[ f(x,u)u-\frac{\theta N}{s}F(x,u)\right] dx. \end{aligned}$$

By the weak lower semicontinuity of convex functional, we have

$$\begin{aligned} \Vert u\Vert ^{N/s}\le \liminf _{n\rightarrow \infty }\Vert u_n\Vert ^{N/s}=\xi ^{N/s}. \end{aligned}$$

In view of Remark 1.1 and the continuity of M, we deduce that

$$\begin{aligned}&\frac{s}{N}{\mathscr {M}}(\Vert u\Vert ^{N/s})-\frac{ s}{N\theta } M(\Vert u\Vert ^{N/s})\Vert u\Vert ^{N/s}\\&\quad \le \frac{s}{N}{\mathscr {M}}(\xi ^{N/s})-\frac{ s}{N\theta } M(\xi ^{N/s})\xi ^{N/s}\\&\quad =\lim _{n\rightarrow \infty }\frac{s}{N}{\mathscr {M}}(\Vert u_n\Vert ^{N/s})-\frac{ s}{N\theta } M(\Vert u_n\Vert ^{N/s})\Vert u_n\Vert ^{N/s}. \end{aligned}$$

By Fatou’s lemma, we get

$$\begin{aligned} \int _\Omega f(x,u)udx\le \liminf _{n\rightarrow \infty }\int _\Omega f(x,u_n)u_ndx. \end{aligned}$$

It follows from above results and (4.5) that

$$\begin{aligned} c_*\le c^*&<\lim _{n\rightarrow \infty }\left[ \frac{s}{N}{\mathscr {M}}(\Vert u_n\Vert ^{N/s}) -\frac{ s}{N\theta } M(\Vert u_n\Vert ^{N/s})\Vert u_n\Vert ^{N/s}\right] \\&\quad +\frac{s}{N\theta }\liminf _{n\rightarrow \infty } \int _\Omega \left[ f(x,u_n)u_n-\frac{N\theta }{s}F(x,u_n)\right] dx\\&\le \lim _{n\rightarrow \infty }\left[ I(u_n)-\frac{ s}{N\theta }\langle I^\prime (u_n),u_n\rangle \right] =c_* \end{aligned}$$

which is absurd. Thus the claim holds true.

Now we claim that

$$\begin{aligned} I(u)=c_*. \end{aligned}$$

(4.6)

Obviously, by (4.5) and semicontinuity of norm, we have $I(u)\le c_*$. Next we are going to show that $I(u_0)<c_*$ can not occur. Actually, if $I(u)<c_*$, then

$$\begin{aligned} \Vert u\Vert <\xi . \end{aligned}$$

Note that (4.5) yields that

$$\begin{aligned} \frac{s}{N}{\mathscr {M}}(\xi ^{N/s}) =\lim _{n\rightarrow \infty }\frac{s}{N}{\mathscr {M}}(\Vert u_n\Vert ^{N/s}) =c_*+\int _\Omega F(x,u)dx. \end{aligned}$$

(4.7)

This gives that

$$\begin{aligned} \xi ^{\frac{N}{s}}={\mathscr {M}}^{-1}\left( \frac{N}{s}c_*+\frac{N}{s}\int _\Omega F(x,u)dx\right) . \end{aligned}$$

Set $v_n=u_n/\Vert u_n\Vert$. Then $v_n\rightharpoonup v_0=u_0/\xi$ in $W_0^{s,N/s}(\Omega )$ and $\Vert v_0\Vert <1$. Thus, it follows from [32, Theorem 2.2] that

$$\begin{aligned} \sup _{n\in {\mathbb {N}}}\int _\Omega \exp (\alpha ^\prime v_n^{N/s})dx<\infty ,\quad \ \ \forall \ \alpha ^\prime <\frac{\alpha _{N,s}}{1-\Vert v_0\Vert ^{N/s}}. \end{aligned}$$

(4.8)

On the other hand, by (4.7), we have

$$\begin{aligned} \frac{N}{s}c_*-\frac{N}{s}I(u)={\mathscr {M}}(\xi ^{N/s}) -{\mathscr {M}}(\Vert u\Vert ^{N/s}). \end{aligned}$$

Thus, it follows from $I(u)\ge 0$ that

$$\begin{aligned} {\mathscr {M}}(\xi ^{N/s})\le \frac{N}{s}c_*+{\mathscr {M}}(\Vert u\Vert ^{N/s}) <{\mathscr {M}}\left( \frac{\alpha _{N,s}}{\alpha _0}\right) +{\mathscr {M}}(\Vert u\Vert ^{N/s}). \end{aligned}$$

Furthermore, by $(M_1)$, we get

$$\begin{aligned} \xi ^{N/s}<{\mathscr {M}}^{-1}\left[ {\mathscr {M}}\left( \frac{\alpha _{N,s}}{\alpha _0}\right) +{\mathscr {M}}(\Vert u\Vert ^{N/s})\right] \le \frac{\alpha _{N,s}}{\alpha _0}+\Vert u\Vert ^{N/s}. \end{aligned}$$

(4.9)

Note that

$$\begin{aligned} \xi ^{N/s}=\frac{\xi ^{N/s}-\Vert u\Vert ^{N/s}}{1-\Vert v\Vert ^{N/s}}. \end{aligned}$$

Hence, it follows from (4.9) that

$$\begin{aligned} \xi ^{N/s}<\frac{\alpha _{N,s}/\alpha _0}{1-\Vert v\Vert ^{N/s}}. \end{aligned}$$

Thus, there exist $n_0\in {\mathbb {N}}$ and $\alpha ^{\prime \prime }>0$ such that

$$\begin{aligned} \alpha _0\Vert u_n\Vert ^{N/(N-s)}<\alpha ^{\prime \prime }<\frac{\alpha _{N,s}}{1-\Vert v\Vert ^{N/s}} \end{aligned}$$

for all $n\ge n_0$. We choose $\nu >1$ close to 1 and $\alpha >\alpha _0$ close to $\alpha _0$ such that

$$\begin{aligned} \nu \alpha \Vert u_n\Vert ^{N/(N-s)}\le \alpha ^{\prime \prime }<\frac{\alpha _{N,s}/\alpha _0}{1-\Vert v\Vert ^{N/s}}. \end{aligned}$$

In view of (4.8), for some $C>0$ and n large enough, we obtain

$$\begin{aligned} \int _\Omega \exp (\nu \alpha |u_n|^{N/(N-s)})dx \le \int _\Omega \exp (\alpha ^{\prime \prime } |v_n|^{N/(N-s)})dx\le C. \end{aligned}$$

Therefore, we deduce from (2.2) that

$$\begin{aligned}&\left| \int _\Omega f(x,u_n)(u_n-u)dx\right| \\&\quad \le C\left( \int _\Omega |u_n-u|^{ N\theta /s}dx+\int _\Omega |u_n-u|\exp (\alpha |u_n|^{N/(N-s)})dx\right) \\&\quad \le C\Vert u_n\Vert _{L^{\frac{N\theta }{s}}(\Omega )}^{N\theta /s}+C\Vert u_n-u\Vert _{L^{\frac{\nu }{\nu -1}}(\Omega )} \rightarrow 0 \end{aligned}$$

as $n\rightarrow \infty$.

Since $\{u_n\}$ is a bounded $(PS)_{c_*}$ sequence in $W_{0}^{s,N/s}(\Omega )$, we have

$$\begin{aligned} o(1)&=\langle I^\prime (u_n),u_n-u\rangle \nonumber \\&=M(\Vert u_n\Vert ^{N/s})\langle u_n,u_n-u\rangle _{s,N/s}-\int _{\Omega }f(x,u_n)(u_n-u)dx. \end{aligned}$$

(4.10)

Define a functional L as follows:

$$\begin{aligned} \langle L(v),w\rangle =\langle v,w\rangle _{s,N/s} \end{aligned}$$

for all $v,w\in W_{0}^{s,N/s}(\Omega )$. By the Hölder inequality, one can see that

$$\begin{aligned} |\langle L(v),w\rangle |\le \Vert v\Vert \Vert w\Vert , \end{aligned}$$

which together with the definition of L implies that for each v, L(v) is a bounded linear functional on $W_{0}^{s,N/s}(\Omega )$. Thus, $\langle L(u),u_n-u\rangle =o(1)$, that is,

$$\begin{aligned} \langle u,u_n-u\rangle _{s,N/s}=o(1). \end{aligned}$$

In conclusion, we can deduce from (4.10) that

$$\begin{aligned} M(\Vert u_n\Vert ^{N/s})\Vert u_n-u\Vert ^{N/s}=M(\Vert u_n\Vert ^{N/s})(\langle u_n,u_n-u\rangle _{s,N/s}-\langle u,u_n-u\rangle _{s,N/s})=o(1). \end{aligned}$$

In view of the fact that $\Vert u_n\Vert \rightarrow \xi$ and $\xi >0$, by using $(M_1)$, we obtain that $u_n\rightarrow u$ in $W_{0}^{s,N/s}(\Omega )$. Furthermore, using (4.5) and the continuity of ${\mathscr {M}}$, we have $I(u)=c_*$, which is a contradiction. Thus, the assertion (4.6) holds true.

Combining $I(u)=c_*$ with $I(u_n)\rightarrow c_*$ and $\Vert u_n\Vert \rightarrow \xi$, we conclude that

$$\begin{aligned} {\mathscr {M}}(\xi ^{N/s})={\mathscr {M}}(\Vert u\Vert ^{N/s}), \end{aligned}$$

which implies that $\xi =\Vert u\Vert$. By the uniform convexity of norm, we obtain that $u_n\rightarrow u$ in $W_0^{s,N/s}(\Omega )$. This finishes the proof. $\square$

Proof of Theorem 1.1

By Lemmas 3.1 and 3.2, we know that I satisfies all the assumptions of Theorem 3.1. Hence there exists a $(PS)_{c_*}$ sequence $\{u_n\}\subset W_{0}^{s,N/s}(\Omega )$. Moreover, by Lemma 4.1, there exists a subsequence of $\{u_n\}$ (still labeled by $\{u_n\}$) such that $u_n\rightarrow u$ in $W_{0}^{s,N/s}(\Omega )$ and $\xi =\Vert u\Vert$. It follows from $I^\prime (u_n)\rightarrow 0$ that

$$\begin{aligned} M(\Vert u_n\Vert ^{N/s})\langle u_n,\varphi \rangle _{s,N/s} =\int _\Omega f(x,u_n)\varepsilon dx,\quad \ \ \forall \varphi \in W_{0}^{s,N/s}(\Omega ). \end{aligned}$$

Furthermore, we have

$$\begin{aligned} M(\Vert u\Vert ^{N/s})\langle u,\varphi \rangle _{s,N/s} =\int _\Omega f(x,u)\varphi dx\quad \ \ \forall \varphi \in W_{0}^{s,N/s}(\Omega ), \end{aligned}$$

which means that u is a solution of (1.1) satisfying $I(u)=c_*$, that is, $I^\prime (u)=0$ and $I(u)=c_*$. Therefore, by the definition of $c^*$ and $c_*\le c^*$, we know that u is a ground state solution of problem (1.1). Moreover, Lemma 2.1 shows that u is nonnegative. $\square$

Proof of Theorem 1.2

Indeed, Theorem 1.2 is a special case of Theorem 1.1. Its proof follows from the same discussion as the proof of Theorem 1.1 by using the corresponding lemmas. $\square$

5 Proof of Theorem 1.3

To study the solutions of problem (1.11), we define the following functional

$$\begin{aligned}&{\mathcal {I}}_\lambda (u)=\frac{s}{N}{\mathscr {M}}(\Vert u\Vert ^{N/s}) -\lambda \int _{\Omega }F(x,u^+)dx-\frac{1}{r}\int _\Omega h(x)|u^+|^rdx \end{aligned}$$

for all $u\in W_0^{s,N/s}(\Omega )$, where $u^+{=}\max \{u,0\}$ and $F(x,u){=}\int _0^u |t|^{q{-}2}t\exp (\alpha _0|t|^{N/(N-s)})dt$. Clearly, one can verify that ${\mathcal {I}}_\lambda$ is of class $C^1$ and the critical points of ${\mathcal {I}}_\lambda$ are the nonnegative solutions of (1.11).

In this section, without further mentioning, we always assume that M fulfills $(M_1)$ and (1.9), $q>\theta N/s$, $r\in (1,N/s)$ and $0\le h\in L^{N/(N-sr)}(\Omega )$.

Lemma 5.1

There exist $\Lambda ^*>0$, ${\widetilde{\rho }}_\lambda >0$ and ${\widetilde{\kappa }}_\lambda >0$ such that for all $\lambda >\Lambda ^*$, ${\mathcal {I}}_\lambda (u)\ge {\widetilde{\kappa }}_\lambda$ for any $u\in W_{0}^{s,N/s}(\Omega )$, with $\Vert u\Vert ={\widetilde{\rho }}_\lambda$.

Proof

Set $t_*=\left( \frac{\alpha _{N,s}}{2\alpha _0}\right) ^{(N-s)/N}$. By (1.9), one can get

$$\begin{aligned} {\mathscr {M}}(t)\ge \frac{{\mathscr {M}}(t_*)}{t_*^{\theta }}t^\theta ,\quad \ \ \mathrm{for\ all}\ 0\le t\le t_*. \end{aligned}$$

By the Hölder inequality, we obtain for all $u\in W_{0}^{s,N/s}(\Omega )$, with $\Vert u\Vert \le {\widetilde{\rho }}_1< t_*$,

$$\begin{aligned} {\mathcal {I}}_\lambda (u)&\ge \frac{{\mathscr {M}}(t_*)}{t_*^{\theta }}\Vert u\Vert ^{\theta N/s}-\frac{\lambda }{q}\Vert u\Vert _{L^{2q}(\Omega )}^q \left[ \int _\Omega \exp \left( 2\alpha _0\Vert u\Vert ^{N/(N-s)}\big |\frac{u}{\Vert u\Vert }\big |^{N/(N-s)}\right) dx\right] ^{1/2}\\&\quad -\frac{1}{r}\int _\Omega h(x)|u|^rdx\nonumber \\&\ge \frac{{\mathscr {M}}(t^*)}{t_*^{\theta }}\Vert u\Vert ^{\theta N/s}-\frac{\lambda }{q}\Vert u\Vert ^q \left[ \int _\Omega \exp \left( 2\alpha _0 {\widetilde{\rho }}_1^{{N/(N-s)}}\big |\frac{u}{\Vert u\Vert }\big |^{{N/(N-s)}}\right) dx\right] ^{1/2}\\&\quad -\frac{C_r^r\Vert h\Vert _{L^{\frac{N}{N-sr}}(\Omega )}}{r}\Vert u\Vert ^r, \end{aligned}$$

where $C_r>0$ denotes the embedding constant of from $W_0^{s,N/s}(\Omega )$ to $L^r(\Omega )$. Since $2\alpha _0 {\widetilde{\rho }}_1^{{N/(N-s)}}< \alpha _{N,s}$, it follows from the fractional Trudinger–Moser inequality that

$$\begin{aligned} {\mathcal {I}}_\lambda (u)\ge \frac{{\mathscr {M}}(t^*)}{t_*^{\theta }}\Vert u\Vert ^{\theta N/s}-\frac{\lambda }{q} C_{N,s}\Vert u\Vert ^{q}-\frac{ C_r^r\Vert h\Vert _{L^{\frac{N}{N-sr}}(\Omega )}}{r}\Vert u\Vert ^r. \end{aligned}$$

Let

$$\begin{aligned} g(t)=\frac{{\mathscr {M}}(t_*)}{t_*^{\theta }}t^{\frac{N\theta }{s}}-\frac{\lambda }{q}C_{N,s}t^q,\ \ \mathrm{for\ all}\ t\ge 0. \end{aligned}$$

It is easy to check that g has positive maximum attained at

$$\begin{aligned} t_{\mathrm{max}}=\left( \frac{{\mathscr {M}}(t_*)N\theta }{sC_{N,s}t_*^\theta \lambda }\right) ^{\frac{s}{sq-N\theta }}>0, \end{aligned}$$

being $q>N\theta /s.$ Set

$$\begin{aligned} \Lambda ^*=\frac{{\mathscr {M}}(t_*)N\theta }{sC_{N,s}t_*^{\theta }{\widetilde{\rho }}_1^{q-\frac{N\theta }{s}}}. \end{aligned}$$

Then for $\lambda \ge \Lambda ^*$ we have $t_{\mathrm{max}}\le {\widetilde{\rho }}_1<t_*$. Since

$$\begin{aligned} \Vert h\Vert _{L^{\frac{N}{N-sr}}(\Omega )}<\frac{rg(t_{\mathrm{max}})}{C_r^rt_{\mathrm{max}}^r}, \end{aligned}$$

we conclude that ${\mathcal {I}}_\lambda (u)\ge {\widetilde{\kappa }}:= {\widetilde{\rho }}_\lambda ^{r}\left( \frac{{\mathscr {M}}(t^*)}{t_*^{\theta }}{\widetilde{\rho }}_\lambda ^{\theta N/s-r}-\frac{\lambda }{q} C_{N,s}{\widetilde{\rho }}_\lambda ^{q-r}-\frac{ 1}{r}C_r^r\Vert h\Vert _{L^{\frac{N}{N-sr}}(\Omega )}\right) >0$ for all $u\in W_{0}^{s,N/s}(\Omega )$, with $\Vert u\Vert ={\widetilde{\rho }}_\lambda :=t_{\mathrm{max}}$. $\square$

Lemma 5.2

Set

$$\begin{aligned}{\widetilde{c}}_\lambda =\inf \left\{ {\mathcal {I}}_\lambda (u): u\in \overline{B_{{\widetilde{\rho }}_\lambda }}\right\} ,\end{aligned}$$

where $B_{{\widetilde{\rho }}_\lambda } =\{u\in W_0^{s,N/s}(\Omega )\,:\, \Vert u\Vert <{\widetilde{\rho }}_\lambda \}$ and ${\widetilde{\rho }}_\lambda \in (0,1]$ is given by Lemma 5.1. Then ${\widetilde{c}}_\lambda <0$ for all $\lambda >\Lambda ^*$.

Proof

Choose a nonnegative function $\varphi \in C_0^\infty (\Omega )$ such that $\Vert \varphi \Vert =1$ and $\int _{\Omega }h(x)\varphi ^r dx>0$. Fix $\lambda >\Lambda ^*$. Then, by $(H_1)$ and (3.3), for all $\tau$, with $0<\tau <1$, we have

$$\begin{aligned} {\mathcal {I}}_\lambda (\tau \varphi )&\le \frac{s}{N} {\mathscr {M}}(\Vert \varphi \Vert ^{N/s}\tau ^{N/s}) -\frac{1}{r} \tau ^r\int _{{\mathbb {R}}^3}h(x)\varphi ^rdx\\&\le \frac{s}{N}\left( \sup _{0\le \tau \le 1}M(\tau )\right) \tau ^{N/s}-\frac{1}{r} \tau ^r\int _{{\mathbb {R}}^3}h(x)\varphi ^rdx. \end{aligned}$$

Since $1<r<N/s$, fixing $\tau >0$ even smaller so that we have that $\tau \varphi \in B_{{\widetilde{\rho }}}$ and ${\mathcal {I}}_\lambda (\tau \varphi )<0$. This gives that ${\widetilde{c}}_\lambda <0$ for all $\lambda >\Lambda ^*$, as desired. $\square$

By Lemmas 5.1 and 5.2 and the Ekeland variational principle (see [2]), applied in $\overline{B_{{\widetilde{\rho }}_\lambda }}$, there exists a sequence $\{u_n\}_n$ such that

$$\begin{aligned} {\mathcal {I}}_\lambda (u_n)\rightarrow {\widetilde{c}}_\lambda ,\quad \ \ {\mathcal {I}}^\prime _\lambda (u_n)\rightarrow 0 \end{aligned}$$

(5.1)

as $n\rightarrow \infty$.

Next we show that $\{u_n\}$ has a convergent subsequence in $W_0^{s,N/s}(\Omega )$.

Lemma 5.3

Up to a subsequence, $\{u_n\}$ is strongly convergent to some function in $W_0^{s,N/s}(\Omega )$.

Proof

Since $\{u_n\}\subset \overline{B_{{\widetilde{\rho }}_\lambda }}$, there exist a subsequence of $\{u_n\}$, still denoted by $\{u_n\}$, $u_\lambda$ and $\omega _\lambda \ge 0$ such that

$$\begin{aligned} {\left\{ \begin{array}{ll} u_n\rightarrow u_\lambda \ \ \mathrm{weakly\ in}\ W_0^{s,N/s}(\Omega ),\\ u_n\rightarrow u_\lambda \ \ \mathrm{strongly\ in}\ L^\nu (\Omega )\ \mathrm{for\ any}\ \nu \in [1,\infty ),\\ \Vert u_n\Vert \rightarrow \omega _\lambda \ \ \mathrm{and}\ \ u_n\rightarrow u_\lambda \ \ \mathrm{a.e.\ in}\ \Omega . \end{array}\right. } \end{aligned}$$

We first show that

$$\begin{aligned} \lim _{n\rightarrow \infty }\int _\Omega |u_n^+|^{q-2}u_n\exp (|u_n^+|^{\frac{N}{N-s}})(u_n-u_\lambda )dx=0. \end{aligned}$$

(5.2)

Indeed, by the Hölder inequality, we have

$$\begin{aligned}&\left| \int _\Omega |u_n^+|^{q-2}u_n^+\exp (|u_n^+|^{\frac{N}{N-s}})(u_n-u_\lambda )dx\right| \nonumber \\&\quad \le \left( \int _{\Omega }|u_n|^{2q}dx\right) ^{\frac{q-1}{2q}} \left( \int _{\Omega }|u_n-u_\lambda |^{2q}dx\right) ^{\frac{1}{2q}} \left( \int _{\Omega }\exp (2\alpha _0|u_n|^{\frac{N}{N-s}})dx\right) ^{\frac{1}{2}}. \end{aligned}$$

(5.3)

In view of Lemma 5.1, we have $2\alpha _0\Vert u_n\Vert ^{N/(N-s)}\le 2\alpha _0\rho _\lambda <\alpha _{N,s}$. Thus, the fractional Trudinger–Moser inequality gives that

$$\begin{aligned} \int _{\Omega }\exp (2\alpha _0|u_n|^{\frac{N}{N-s}})dx&=\int _{\Omega }\exp \left( 2\alpha _0\Vert u_n\Vert ^{\frac{N}{N-s}}\left| \frac{u_n}{\Vert u_n\Vert }\right| ^{\frac{N}{N-s}}\right) dx \\ {}&\le \int _{\Omega }\exp \left( 2\alpha _0{\widetilde{\rho }}^{\frac{N}{N-s}}\left| \frac{u_n}{\Vert u_n\Vert }\right| ^{\frac{N}{N-s}}\right) dx\\&\le C_{N,s}. \end{aligned}$$

It follows from (5.3) that

$$\begin{aligned}&\left| \int _\Omega |u_n^+|^{q-2}u_n^+\exp (|u_n^+|^{\frac{N}{N-s}})(u_n-u_\lambda )dx\right| \nonumber \\&\quad \le C \left( \int _{\Omega }|u_n-u_\lambda |^{2q}dx\right) ^{\frac{1}{2q}}\rightarrow 0 \end{aligned}$$

as $n\rightarrow \infty ,$ which means that (5.2) holds true.

Since $h(x)\in L^{\frac{N}{N-sr}}(\Omega )$ and $1<r<N/s$, by Vitali’s convergence theorem one can prove that

$$\begin{aligned} \lim _{n\rightarrow \infty }\int _\Omega h(x)|u_n^+|^{r-2}u_n^+(u_n-u_\lambda )dx=0. \end{aligned}$$

(5.4)

By the weak convergence of $\{u_n\}$ in $W_0^{s,N/s}(\Omega )$, one can easily get that

$$\begin{aligned} \lim _{n\rightarrow \infty }M(\Vert u_n\Vert ^{N/s})\langle u_\lambda , u_n-u_\lambda \rangle _{s,N/s}=0. \end{aligned}$$

(5.5)

Due to the fact that $\{u_n\}$ is a (PS) sequence, we have

$$\begin{aligned}\langle {\mathcal {I}}_\lambda ^\prime (u_n),u_n-u_\lambda \rangle -M(\Vert u_n\Vert ^{N/s})\langle u_\lambda ,u_n-u_\lambda \rangle _{s,N/s}=o(1).\end{aligned}$$

Then,

$$\begin{aligned} o(1)&=M(\Vert u_n\Vert ^{N/s})\left( \langle u_{n},u_n-u_\lambda \rangle _{s,N/s} -\langle u_\lambda ,u_n-u_\lambda \rangle _{s,N/s}\right) \\&\quad -\int _{\Omega }|u_n^+|^{q-2}u_n^+\exp (\alpha _0|u_n^+|^{\frac{N}{N-s}})(u_n-u_\lambda )dx\\&\quad +\int _\Omega h(x)|u_n^+|^{r-2}u_n^+(u_n-u_\lambda )dx. \end{aligned}$$

Combining (5.2), (5.4) and (5.5), we get

$$\begin{aligned} \lim _{n\rightarrow \infty }M(\Vert u_n\Vert ^{N/s})\left( \langle u_n,u_n-u_\lambda \rangle _{s,N/s}-\langle u_\lambda ,u_n-u_\lambda \rangle _{s,N/s}\right) =0. \end{aligned}$$

(5.6)

If $\omega _\lambda =0$, then by ${\mathcal {I}}_\lambda (u_n)\rightarrow {\widetilde{c}}_\lambda$ and (4.5) we obtain

$$\begin{aligned} 0>{\widetilde{c}}_\lambda =\lim _{n\rightarrow \infty } \left[ \frac{s}{N}{\mathscr {M}}(\Vert u_n\Vert ^{N/s})-\int _\Omega F(x,u_n^+)dx-\frac{1}{r}\int _\Omega h(x)|u_n^+|^rdx\right] =0, \end{aligned}$$

which is impossible. Thus, we get $\omega _\lambda >0$. Therefore, from (5.6) and $(M_1)$, we conclude that $\Vert u_n-u_\lambda \Vert \rightarrow 0$ as $n\rightarrow \infty$. In conclusion, the proof is complete. $\square$

Proof of Theorem 1.3

By Lemmas 5.1 and 5.2, there exists a (PS) sequence $\{u_n\}$ such that

$$\begin{aligned} {\mathcal {I}}_\lambda (u_n)\rightarrow {\widetilde{c}}_\lambda , \ \ I^\prime _\lambda (u_n)\rightarrow 0. \end{aligned}$$

Furthermore, by Lemma 5.3, there exist a subsequence of $\{u_n\}$ (still denoted by $\{u_n\}$) and $u_\lambda \in W_0^{s,N/s}(\Omega )$ such that

$$\begin{aligned} u_n\rightarrow u_\lambda . \end{aligned}$$

Moreover, $u_\lambda$ is a nonnegative and nontrivial solutions of problem (1.11). Finally, according to the following fact

$$\begin{aligned} \Vert u_\lambda \Vert \le {\widetilde{\rho }}_\lambda :=\left( \frac{{\mathscr {M}}(t_*)N\theta }{sC_{N,s}t_*^\theta \lambda }\right) ^{\frac{s}{sq-N\theta }} \quad \ \ \mathrm{and}\quad \ \ t_*= \left( \frac{\alpha _{N,s}}{2\alpha _0}\right) ^{(N-s)/N}, \end{aligned}$$

we deduce that $\Vert u_\lambda \Vert \rightarrow 0$ as $\lambda \rightarrow \infty .$ $\square$