1 Introduction and main results

In this paper we are interested in the nonexistence of stable solutions to the quasilinear Schrödinger equation

$$\begin{aligned} -\Delta u- \bigl[\Delta\bigl(1+u^{2}\bigr)^{1/2} \bigr]\frac{ u}{2(1+u^{2})^{1/2}}=h(x) \vert u \vert ^{q-1}u,\quad x\in R^{N}, \end{aligned}$$
(1.1)

where \(N\ge3\) and \(q\ge5/2\). Equation (1.1) can be obtained as a stationary problem of the modified Schrödinger equation

$$\begin{aligned} iz_{t}=-\Delta z+W(x)z-h\bigl(x, \vert z \vert ^{2}\bigr)z- \bigl[\Delta l\bigl( \vert z \vert ^{2} \bigr) \bigr]l'\bigl( \vert z \vert ^{2}\bigr)z, \quad x\in R^{N}, \end{aligned}$$
(1.2)

where \(z: R\times R^{N} \to\mathbb{ C}\), \(W:R^{N}\to R\) is a given potential, h and l are real functions.

It is well known that the standing wave solutions of the form \(z(t,x)=\exp(-i\omega t)u(x)\) satisfy (1.2) if and only if the real function \(u(x)\) solves the equation of elliptic type

$$\begin{aligned} -\Delta u+V(x)u- \bigl[\Delta l\bigl(u^{2}\bigr) \bigr]l'\bigl(u^{2}\bigr)u=g(x,u), \quad x\in R^{N}, \end{aligned}$$
(1.3)

where \(V(x)=W(x)-\omega, \omega\in R\) and \(g(x,u)\equiv h(x,u^{2})u\).

Quasilinear Schrödinger equations as in (1.3) appear naturally in mathematical physics and have been derived as models of several physical phenomena corresponding to various types of nonlinear term l. When \(l(s)=s\), we get the superfluid film equation in plasma physics [16]:

$$\begin{aligned} -\Delta u+V(x)u-\Delta\bigl(u^{2}\bigr)u=g(x,u), \quad x \in R^{N}. \end{aligned}$$
(1.4)

In the case \(l(s)=(1+s)^{1/2}\), (1.3) turns into the following equation:

$$\begin{aligned} -\Delta u+V(x)u- \bigl[\Delta\bigl(1+u^{2} \bigr)^{\frac{1}{2}} \bigr]\frac {u}{2(1+u^{2})^{\frac{1}{2}}}=g(x,u), \quad x\in R^{N}, \end{aligned}$$
(1.5)

which models the self-channeling of a high-power ultrashort laser in matter, see [2, 10].

The existence of positive solutions for (1.5) has been extensively studied recently. In [23], the authors proved that (1.5) has a positive solution under the assumptions: \(g(x,u)=\lambda|u|^{p-1}u, \sqrt{6}-1\le p<\frac{N+2}{N-2}, \lambda>0 \) and \(0<\inf_{x\in {\mathbb {R}}^{N}}V(x)\le V(x)\le V(\infty):=\lim_{|x|\to\infty}V(x)<\infty\).

Li [20] studied the existence of quasilinear Schrödinger equations of the form

$$\begin{aligned} -\Delta u+V(x)u- \bigl[\Delta\bigl(1+u^{2} \bigr)^{\alpha/2} \bigr]\frac{\alpha u}{2(1+u^{2})^{(2-\alpha)/2}}=g(x,u), \quad x\in R^{N}, \end{aligned}$$
(1.6)

where the parameter \(\alpha\in[1,2]\) and the functions \(V(x), g(x,u)\) are 1-periodic in \(x_{j}\) for \(j=1,2,\dots, N\).

Similar works can be found in [6, 7, 11, 24, 27] and the references therein. It is noted that, in the above works, one always assumes that the potential function \(V(x)\ge0\) and \(V(x)\not\equiv 0\) in \({\mathbb {R}}^{N}\).

On the other hand, the nonexistence of solutions and either the stable or unstable solutions for Lane–Emden problems are investigated to some extent. The results can be found in [1, 3, 8, 14, 15, 18, 19, 21, 25, 26, 28], and the references therein. To the best of our knowledge, there are no results on the nonexistence of solutions for (1.1). Motivated by [4, 5, 9, 13, 17, 22], our purpose in this paper is to study the nonexistence of nonnegative and stable solutions of (1.1) under some assumptions on the weighted function \(h(x)\) and the exponent q.

Usually, we make the change of variables \(z=f^{-1}(u)\), where f is defined by

$$\begin{aligned} f'(t)= \biggl[1+\frac{f^{2}(t)}{2(1+f^{2}(t))} \biggr]^{-1/2}=\sqrt {2}\bigl(1+f^{2}(t)\bigr)^{\frac{1}{2}} \bigl(2+3f^{2}(t)\bigr)^{-\frac{1}{2}},\quad t\ge0, f(0)=0 \end{aligned}$$
(1.7)

and by \(f(t)=-f(-t)\) on \((-\infty, 0]\).

Lemma 1.1

([20, 23])

The function \(f(t)\) satisfies the following properties:

\((f_{1})\) :

f is uniquely defined, odd, increasing, invertible and \(C^{\infty}\) in \(R=(-\infty, +\infty)\),

\((f_{2})\) :

\(0< f'(t)\le1\), \(\forall t\in R\),

\((f_{3})\) :

\(|f(t)|\le|t|\), \(\forall t\in R\),

\((f_{4})\) :

\(\frac{f(t)}{t}\to1\) as \(t\to0\),

\((f_{5})\) :

\(f(t)\le2 tf'(t)\le2 f(t)\), \(\forall t\in R^{+}=[0, \infty)\),

\((f_{6})\) :

\(\lim_{t\to+\infty}\frac{f(t)}{t}=\sqrt{\frac{2}{3}}\).

If we take \(u=f(z)\) or \(z=f^{-1}(u)\), then (1.1) becomes the following semilinear elliptic equation:

$$\begin{aligned} -\Delta z=h(x)f'(z) \bigl\vert f(z) \bigr\vert ^{q-1}f(z),\quad x\in R^{N}. \end{aligned}$$
(1.8)

As usual, we study the existence and the nonexistence of weak solutions of (1.1) via (1.8).

Definition 1.1

([12])

The function \(z\in C_{\mathrm{loc}}^{1,\delta}({\mathbb {R}}^{N})\) \((0<\delta<1)\) is said to be a weak solution of (1.8) if

$$\begin{aligned} \int_{{\mathbb {R}}^{N}}\nabla z\nabla\zeta \,dx= \int_{{\mathbb {R}}^{N}}h(x)g(z)\zeta \,dx,\quad \forall\zeta\in C_{0}^{1}\bigl({\mathbb {R}}^{N}\bigr), \end{aligned}$$
(1.9)

where (and in the sequel) \(g(z)=f'(z)|f(z)|^{q-1}f(z)\). A weak solution z of (1.8) is stable if

$$\begin{aligned} \int_{{\mathbb {R}}^{N}} \vert \nabla\zeta \vert ^{2}\,dx\ge \int_{{\mathbb {R}}^{N}}h(x)g'(z)\zeta^{2}\,dx ,\quad \forall\zeta\in C_{0}^{1}\bigl({\mathbb {R}}^{N}\bigr). \end{aligned}$$
(1.10)

In other words, the stability condition translates into the fact that the second variation of the energy functional is nonnegative. Thus, all the minima of the functional are stable solutions of (1.8).

As in [11, 20, 23], we give:

Definition 1.2

A function \(u\in\mathcal{D}^{1,2}({\mathbb {R}}^{N})\) is called a weak solution of (1.1) if \(z=f^{-1}(u)\) is a weak solution of (1.8). A weak solution u of (1.1) is stable in \({\mathbb {R}}^{N}\) if \(z=f^{-1}(u)\) is a stable solution of (1.8).

So in order to prove the nonexistence of stable solutions for (1.1), it is sufficient to prove that there is no nonnegative and stable weak solution to (1.8).

The main result in this paper is as follows.

Theorem 1.2

Suppose that the positive function \(h(x)\in C^{1}_{\mathrm{loc}}({\mathbb {R}}^{N})\) is such that there exist \(a>-2\), \(a_{0}>0\) and \(R_{0}>0\) such that

$$\begin{aligned} h(x)\ge a_{0} \vert x \vert ^{a},\quad \forall \vert x \vert \ge R_{0}. \end{aligned}$$
(1.11)

Denote

$$\begin{aligned} \gamma_{0}(a):=6+2a,\qquad\gamma_{\infty}(a):=10+4a. \end{aligned}$$
(1.12)

Let \(q_{c}\) be the positive root of the equation \(N=X(q)\) with

$$\begin{aligned} X(t)=\frac{2[t+k_{0}(t)]+a[k_{0}(t)+1]}{t-1},\quad t\ge5/2, \end{aligned}$$
(1.13)

and

$$\begin{aligned} k_{0}(t)=2t-3+\sqrt{4t^{2}-12t+5},\quad t \ge5/2. \end{aligned}$$
(1.14)

Moreover, let \(u\in C_{\mathrm{loc}}^{1,\delta}({\mathbb {R}}^{N})\) (\(0<\delta<1\)) be a nonnegative and stable solution of (1.1). Then, we have \(u\equiv0\) in \({\mathbb {R}}^{N}\) if one of the following conditions is satisfied:

\((A_{1})\) :

\(q\ge\frac{5}{2}\) and \(3\le N\le\gamma_{0}{(a)}\);

\((A_{2})\) :

\(q>q_{c}\) and \(\gamma_{0}{(a)}< N<\gamma_{\infty}{(a)}\).

Remark 1.3

When \(N\in(\gamma_{0}, \gamma_{\infty})\), it is not difficult to get

$$\begin{aligned} q_{c}=\frac{6-d^{2}+\sqrt{16-3d^{2}}}{4-d^{2}}\quad\mbox{with } d= \frac{N-6-2a}{2+a}. \end{aligned}$$
(1.15)

The condition \(\gamma_{0}< N<\gamma_{1}\) implies that \(d\in(0,2)\) and \(q_{c}>5/2\).

2 Proof of Theorem 1.2

In order to prove the nonexistence of solution to (1.8), we use the test function method, which has been used in [5, 9] to deal with the m-Laplace equation. The proof is by contradiction which involves obtaining an a priori estimate for a solution of (1.8) by carefully choosing a special test function and then applying the scaling argument. We first establish:

Lemma 2.1

Assume \(q>1\). Let \(f(t)\) be the function defined by (1.7) and \(k>1\). Suppose \(g(t)=f'(t)|f(t)|^{q-1}f(t)\) and

$$\begin{aligned} G(t)= \biggl(\frac{g(t)}{g'(t)}t^{k} \biggr)^{1/2}, \quad\forall t\ge0. \end{aligned}$$
(2.1)

Then there exist \(d_{1},d_{1}'>0\) such that

$$\begin{aligned} d_{1}'t^{k+1}\le G^{2}(t)\le d_{1}t^{k+1}, \quad\forall t\ge0. \end{aligned}$$
(2.2)

Furthermore, if \(q\ge2\), we have for all \(t\ge0\),

$$\begin{aligned} \bigl(G'(t)\bigr)^{2}\le \alpha_{k} t^{k-1}, \quad\textit{where } \alpha_{k}=\frac{(k+1)^{2}}{2q-1}. \end{aligned}$$
(2.3)

Proof

Let \(f=f(t)\ (t\ge0)\). Direct computation shows that

$$\begin{aligned} g'(t)=2\beta(t)f^{q-1}\bigl[2+3f^{2} \bigr]^{-2}, \quad \mbox{where } \beta(t)=2q+(5q-1)f^{2}(t)+3qf^{4}(t) \end{aligned}$$
(2.4)

and

$$\begin{aligned} G^{2}(t)=\frac{g(t)}{g'(t)}t^{k}= \frac{ft^{k}(1+f^{2})^{\frac {1}{2}}(2+3f^{2})^{\frac{3}{2}}}{\sqrt{2}\beta(t)}. \end{aligned}$$
(2.5)

From \(f(0)=0\), as well as \((f_{4})\) and \((f_{6})\) in Lemma 1.1, it follows that

$$\begin{aligned} \lim_{t\to0^{+}}\frac{G^{2}(t)}{t^{k+1}}=\frac{1}{q} \quad\mbox{and}\quad \lim_{t\to+\infty}\frac{G^{2}(t)}{t^{k+1}}= \frac{1}{q}. \end{aligned}$$
(2.6)

Furthermore, noticing the fact that the function \(H(t)=t^{-k-1}G^{2}(t)\) is positive and continuous on any bounded interval \([c, d]\subset(0, +\infty)\), we obtain (2.2) from (2.6).

In the following we prove (2.3). First, we have from (2.5) that

$$\begin{aligned} \sqrt{2}\beta(t)G^{2}(t)=ft^{k} \bigl(1+f^{2}\bigr)^{\frac{1}{2}}\bigl(2+3f^{2} \bigr)^{\frac {3}{2}},\quad t\ge0. \end{aligned}$$
(2.7)

Differentiating (2.7) with respect to t gives

$$\begin{aligned} \begin{aligned}[b] &2\sqrt{2}G(t)G'(t) \beta(t)+\sqrt{2}G^{2}(t)\beta '(t)=\bigl[ft^{k} \bigl(1+f^{2}\bigr)^{\frac{1}{2}}\bigl(2+3f^{2} \bigr)^{\frac{3}{2}}\bigr]' \\ &\quad =kt^{k-1}f\bigl(1+f^{2}\bigr)^{\frac{1}{2}} \bigl(2+3f^{2}\bigr)^{\frac{3}{2}}+\sqrt {2}t^{k} \bigl(2+16f^{2}+15f^{4}\bigr). \end{aligned} \end{aligned}$$
(2.8)

Since

$$\begin{aligned} \sqrt{2}G^{2}(t)\beta'(t)=2\sqrt {2}t^{k}f^{2}\bigl(1+f^{2}\bigr) \bigl(2+3f^{2}\bigr) \bigl(5q-1+6qf^{2}\bigr) \beta^{-1}(t), \end{aligned}$$
(2.9)

we have from (2.5), (2.8) and (2.9) that

$$\begin{aligned} 8\bigl(G'(t)\bigr)^{2}=t^{k-1} \bigl(A_{1}(t)+A_{2}(t)+A_{3}(t)\bigr),\quad\forall t\ge0, \end{aligned}$$
(2.10)

where

$$\begin{aligned} \begin{aligned} &A_{1}(t)=\frac{\sqrt{2}k^{2}f(1+f^{2})^{\frac{1}{2}}(2+3f^{2})^{\frac {3}{2}}}{t\beta(t)},\\ &A_{2}(t)= \frac{4k\gamma(t)}{\beta^{2}(t)},\\ & A_{3}(t)=\frac{2\sqrt{2} t\gamma^{2}(t)}{f\beta^{3}(1+f^{2})^{\frac {1}{2}}(2+3f^{2})^{\frac{3}{2}}} \end{aligned} \end{aligned}$$
(2.11)

and

$$\begin{aligned} \begin{aligned}[b] \gamma(t)&=\beta(t) \bigl(2+16f^{2}+15f^{4}\bigr)-2f^{2} \bigl(1+f^{2}\bigr) \bigl(2+3f^{2}\bigr) \bigl(5q-1+6qf^{2} \bigr) \\ &=4q+2(11q+1)f^{2}+6(7q-1)f^{4}+3(11q-3)f^{6}+9qf^{8}. \end{aligned} \end{aligned}$$
(2.12)

Using the property \((f_{5})\) in Lemma 1.1, we have

$$\begin{aligned} \frac{f(t)}{t}\le 2f'(t)=2\sqrt{2} \bigl(1+f^{2}\bigr)^{\frac{1}{2}}\bigl(2+3f^{2} \bigr)^{-\frac{1}{2}},\quad \forall t\ge0, \end{aligned}$$
(2.13)

and then

$$\begin{aligned} A_{1}(t)\le \frac{4k^{2}(1+f^{2})(2+3f^{2})}{\beta(t)}=\frac {4k^{2}(2+5f^{2}+3f^{4})}{2q+(5q-1)f^{2}+3qf^{4}}\le \frac{20k^{2}}{5q-1},\quad \forall t\ge0. \end{aligned}$$
(2.14)

On the other hand, directive computation gives

$$\begin{aligned} \beta^{2}(t)=4q^{2}+4q(5q-1)f^{2}+ \bigl(37q^{2}-10q+1\bigr)f^{4}+6q(5q-1)f^{6}+9q^{2}f^{8}. \end{aligned}$$
(2.15)

Then an application of (2.12) and (2.15) yields \(\gamma(t)\le\frac{3}{2q}\beta^{2}(t)\), and so

$$\begin{aligned} A_{2}(t)\le\frac{6k}{q},\quad\forall t\ge0. \end{aligned}$$
(2.16)

Furthermore, we see from \((f_{5})\) in Lemma 1.1 that

$$\begin{aligned} \frac{t}{f(t)}\le \frac{1}{f'(t)}=2^{-\frac{1}{2}} \bigl(1+f^{2}\bigr)^{-\frac {1}{2}}\bigl(2+3f^{2} \bigr)^{\frac{1}{2}} , \end{aligned}$$
(2.17)

and then

$$\begin{aligned} A_{3}(t)\le\frac{2\gamma^{2}(t)}{\beta^{3}(t)(2+5f^{2}+3f^{4})},\quad\forall t\ge0. \end{aligned}$$
(2.18)

We now claim

$$\begin{aligned} \frac{\gamma^{2}(t)}{\beta^{3}(t)(2+5f^{2}+3f^{4})}\le\frac{2}{q},\quad \forall t\ge0. \end{aligned}$$
(2.19)

In fact, we have

$$\begin{aligned} \begin{aligned}[b] \gamma ^{2}(t)={}&16q^{2}+16q(11q+1)f^{2}+4 \bigl(205q^{2}+10q+1\bigr)f^{4}+24\bigl(88q^{2}-7q-1 \bigr)f^{6}\\ &{}+24q(137q-32)f^{8} +36\bigl(88q^{2}-31q+3\bigr)f^{10}+9\bigl(205q^{2}-78q+9 \bigr)f^{12}\\ &{}+54q(11q-3)f^{14}+81q^{2}f^{16} \end{aligned} \end{aligned}$$
(2.20)

and

$$\begin{aligned} \begin{aligned}[b] \bigl(2+5f^{2}+3f^{4} \bigr)\beta ^{3}(t)={}&\bigl(2+5f^{2}+3f^{4}\bigr) \bigl[2q+(5q-1)f^{2}+3qf^{4}\bigr]^{3}\\ ={}&16q^{3}+8q^{2}(20q-3)f^{2} +12q\bigl(58q^{2}-15q+1\bigr)f^{4} \\ &{}+2\bigl(860q^{3}-279q^{2}+30q-1 \bigr)f^{6}\\ &{}+\bigl(2641q^{3}-915q^{2}+111q-5 \bigr)f^{8}\\ &{} +3\bigl(860q^{3}-279q^{2}+30q-1\bigr)f^{10}\\ &{}+27q \bigl(58q^{2}-15q+1\bigr)f^{12}+27q^{2}(20q-3)f^{14}\\ &{}+81q^{3}f^{16}. \end{aligned} \end{aligned}$$
(2.21)

Clearly, it is not difficult to verify (2.19) by the use of (2.20) and (2.21). Then, from (2.10), (2.14), (2.16), (2.18) and (2.19), we obtain

$$\begin{aligned} \bigl(G'(t)\bigr)^{2}\le t^{k-1} \biggl(\frac{5k^{2}}{2(5q-1)}+\frac{3k}{4q}+\frac{1}{2q} \biggr)\le \frac{(k+1)^{2}}{2q-1}t^{k-1},\quad\forall t\ge0, \end{aligned}$$
(2.22)

and the proof of Lemma 2.1 is completed. □

On the other hand, for the function \(g(t)\) defined in Lemma 2.1, we have the following result.

Lemma 2.2

Assume \(q>1\). Let \(f(t)\) be the function defined by (1.7) and \(k>1\). Suppose \(g(t)=f'(t)|f(t)|^{q-1}f(t)\). Then, there exists \(M_{1}>0\) such that

$$\begin{aligned} \biggl\vert \frac{t^{q}}{g(t)} \biggr\vert \le M_{1},\quad\forall t> 0. \end{aligned}$$
(2.23)

Proof

Obviously, the function \(Y(t)=\frac{t^{q}}{g(t)}\) is continuous on \((0, +\infty)\). From \((f_{4})\) and \((f_{6})\) in Lemma 1.1, we derive

$$\begin{aligned} \lim_{t\to0^{+}}Y(t)=\lim_{t\to 0^{+}} \frac{t^{q}(2+3f^{2})^{\frac{1}{2}}}{\sqrt{2}(1+f^{2})^{\frac {1}{2}}f^{q}(t)}=\lim_{t\to 0^{+}}\frac{t^{q}}{f^{q}(t)}=1 \end{aligned}$$
(2.24)

and

$$\begin{aligned} \lim_{t\to +\infty}Y(t)= \biggl(\frac{3}{2} \biggr)^{\frac{1}{2}}\lim_{t\to +\infty}\frac{t^{q}}{f^{q}(t)}= \biggl( \frac{3}{2} \biggr)^{(q+1)/2}. \end{aligned}$$
(2.25)

Then the conclusion (2.23) follows, and the proof of Lemma 2.2 is completed. □

Lemma 2.3

Let \(z\in C_{\mathrm{loc}}^{1,\delta}({\mathbb {R}}^{N})\) \((0<\delta<1)\) be a nonnegative and stable weak solution of (1.8) with \(q\ge5/2\). Then for every \(k\in(1, k_{0}(q))\), where \(k_{0}(t)\) is defined by (1.14), there exists a constant \(C=C(q,k)\) such that

$$\begin{aligned} \int_{{\mathbb {R}}^{N}}h(x)g(z)z^{k}\varphi^{2}\,dx\le C \int_{{\mathbb {R}}^{N}} \bigl\vert h(x) \bigr\vert ^{-\frac{k+1}{q-1}} \vert \nabla \varphi \vert ^{\frac{2(q+k)}{q-1}}\varphi^{-\frac{2(k+1)}{q-1}} \,dx, \end{aligned}$$
(2.26)

where \(\varphi=\varphi(x)\in C_{0}^{1}({\mathbb {R}}^{N})\) is a nonnegative cut-off function, in which \(\varphi(x)=\varphi_{0}(\frac{|x|}{R})\) with \(R>0\), \(\varphi_{0}(s)\in C_{0}^{1}({\mathbb {R}}^{+})\), \(0\le\varphi_{0}(s)\le1\), and it is defined by

$$\begin{aligned} \varphi_{0}(s)= \textstyle\begin{cases} 1,& 0\le s\le1, \\ 2(2-s)^{m}-(2-s)^{2m},& 1< s< 2, \\ 0, & s>2, \end{cases}\displaystyle \end{aligned}$$
(2.27)

with \(m=\frac{q+k}{q-1}>1\).

Remark 2.4

It is not difficult to verify that \(0\le\varphi_{0}(s)\le1\) and \(|\varphi'_{0}(s)|\le\beta_{0}\varphi_{0}^{1-1/m}(s)\) with \(\beta_{0}=2^{1/m}m\).

Proof

Let \(z\in C^{1,\delta}_{\mathrm{loc}}({\mathbb {R}}^{N})\) be a nonnegative and stable weak solution of (1.8) and \(k>1\). Setting \(\zeta=z^{k}\varphi^{2}\) in (1.9), we find

$$\begin{aligned} k \int_{{\mathbb {R}}^{N}} \vert \nabla z \vert ^{2}z^{k-1} \varphi^{2}\,dx\le2 \int_{{\mathbb {R}}^{N}} \vert \nabla z \vert \vert \nabla\varphi \vert z^{k}\varphi \,dx+ \int_{{\mathbb {R}}^{N}}h(x)g(z)z^{k}\varphi^{2}\,dx. \end{aligned}$$
(2.28)

Applying Young’s inequality with parameter \(\epsilon\in(0,1)\), we obtain

$$\begin{aligned} 2 \int_{{\mathbb {R}}^{N}} \vert \nabla z \vert \vert \nabla\varphi \vert z^{k}\varphi \,dx\le \epsilon \int_{{\mathbb {R}}^{N}} \vert \nabla z \vert ^{2}z^{k-1} \varphi^{2}\,dx+C \int_{{\mathbb {R}}^{N}} \vert \nabla\varphi \vert ^{2}z^{k+1} \,dx. \end{aligned}$$
(2.29)

Here and in the sequel, let C be a positive constant depending on ϵ and \(q, k\), which may vary from line to line. Then it follows from (2.28) and (2.29) that

$$\begin{aligned} (k-\epsilon) \int_{{\mathbb {R}}^{N}} \vert \nabla z \vert ^{2}z^{k-1} \varphi^{2}\,dx\le \int_{{\mathbb {R}}^{N}}h(x) \bigl\vert g(z) \bigr\vert z^{k} \varphi^{2}\,dx+C \int_{{\mathbb {R}}^{N}} \vert \nabla\varphi \vert ^{2}z^{k+1} \,dx. \end{aligned}$$
(2.30)

On the other hand, taking \(\zeta=G(z)\varphi\) in (1.10), we find

$$\begin{aligned} \vert \nabla\zeta \vert ^{2}=\varphi^{2} \bigl(G'(z)\bigr)^{2} \vert \nabla z \vert ^{2}+2\varphi G'(z)G(z)\nabla z\nabla \varphi+G^{2}(z) \vert \nabla\varphi \vert ^{2} , \end{aligned}$$
(2.31)

and then

$$\begin{aligned} \int_{{\mathbb {R}}^{N}} h(x)g(z)z^{k}\varphi^{2}\,dx \le& \int_{{\mathbb {R}}^{N}} \vert \nabla z \vert ^{2} \bigl\vert G'(z) \bigr\vert ^{2}\varphi^{2}\,dx + 2 \int_{{\mathbb {R}}^{N}} \vert \nabla z \vert \vert \nabla \varphi \vert \bigl\vert G'(z)G(z) \bigr\vert \varphi \,dx \\ &{}+ \int_{{\mathbb {R}}^{N}} \vert \nabla \varphi \vert ^{2}G^{2}(z) \,dx. \end{aligned}$$
(2.32)

By Young’s inequality with \(\epsilon>0\), one derives

$$\begin{aligned} 2 \int_{{\mathbb {R}}^{N}} \vert \nabla z \vert \vert \nabla\varphi \vert \bigl\vert G'(z)G(z) \bigr\vert \varphi \,dx \le& \epsilon \int_{{\mathbb {R}}^{N}} \vert \nabla z \vert ^{2} \bigl\vert G'(z) \bigr\vert ^{2}\varphi^{2}\,dx \\ &{}+C \int_{{\mathbb {R}}^{N}} \vert \nabla\varphi \vert ^{2} \bigl\vert G(z) \bigr\vert ^{2}\,dx. \end{aligned}$$
(2.33)

Then, it follows from (2.2), (2.3), (2.30) and (2.32) that

$$\begin{aligned} \begin{aligned}[b] \int_{{\mathbb {R}}^{N}} h(x)g(z)z^{k}\varphi^{2}\,dx\le{}& (1+\epsilon) \int_{{\mathbb {R}}^{N}} \bigl\vert G'(z) \bigr\vert ^{2} \vert \nabla z \vert ^{2}\varphi^{2}\,dx+C \int_{{\mathbb {R}}^{N}} \vert \nabla\varphi \vert ^{2} \bigl\vert G(z) \bigr\vert ^{2}\,dx \\ \le{}&\alpha_{k}(1+\epsilon) \int_{{\mathbb {R}}^{N}} \vert \nabla z \vert ^{2} \varphi^{2}z^{k-1}\,dx+C \int_{{\mathbb {R}}^{N}} \vert \nabla\varphi \vert ^{2} \bigl\vert G(z) \bigr\vert ^{2}\,dx \\ \le{}& \frac{\alpha_{k}(1+\epsilon)}{k-\epsilon} \int_{{\mathbb {R}}^{N}} h(x)g(z)z^{k}\varphi^{2}\,dx+ C \int_{{\mathbb {R}}^{N}} \vert \nabla\varphi \vert ^{2}z^{k+1} \,dx\\ &{}+C \int_{{\mathbb {R}}^{N}} \bigl\vert G(z) \bigr\vert ^{2} \vert \nabla\varphi \vert ^{2}\,dx \\ \le{}& \frac{\alpha_{k}(1+\epsilon)}{k-\epsilon} \int_{{\mathbb {R}}^{N}} h(x)g(z)z^{k}\varphi^{2}\,dx+ C_{1} \int_{{\mathbb {R}}^{N}} \vert \nabla\varphi \vert ^{2}z^{k+1} \,dx. \end{aligned} \end{aligned}$$
(2.34)

Then one sees from (2.34) that

$$\begin{aligned} \delta_{\epsilon} \int_{{\mathbb {R}}^{N}} h(x)g(z)z^{k}\varphi^{2}\,dx\le C \int_{{\mathbb {R}}^{N}} \vert \nabla\varphi \vert ^{2}z^{k+1} \,dx, \end{aligned}$$
(2.35)

where

$$\begin{aligned} \delta_{\epsilon}=1-\frac{\alpha_{k}(1+\epsilon)}{k-\epsilon} =1-\frac{(1+\epsilon)(k+1)^{2}}{(k-\epsilon)(2q-1)}. \end{aligned}$$
(2.36)

Clearly,

$$\begin{aligned} \delta_{0}=\lim_{\epsilon\to 0+} \delta_{\epsilon}=1-\frac{(k+1)^{2}}{k(2q-1)} . \end{aligned}$$
(2.37)

Moreover, the elementary inequality \(\delta_{0}>0\) implies that \(k\in (1, \frac{k_{0}(q)}{2})\), where \(k_{0}(t)\) is defined by (1.14). Now, an application of (2.35) yields

$$\begin{aligned} \int_{{\mathbb {R}}^{N}}h(x)g(z)z^{k}\varphi^{2}\,dx\le C \int_{{\mathbb {R}}^{N}} \vert \nabla\varphi \vert ^{2}z^{k+1} \,dx. \end{aligned}$$
(2.38)

Let \(\lambda=\frac{q+k}{k+1}\), \(\lambda'=\frac{q+k}{q-1}\). Then it follows from Lemma 2.2 that, if \(z=z(x)\) is nonnegative in \({\mathbb {R}}^{N}\), the function \(\psi(x)=z^{(k+1)\lambda'}(g(z)z^{k})^{-\frac{\lambda'}{\lambda }}=(\frac{z^{q}}{g(z)})^{\frac{k+1}{q-1}}\) is nonnegative and bounded in \({\mathbb {R}}^{N}\). Moreover, we obtain from the Hölder inequality, (2.23) and (2.38) that

$$\begin{aligned} \begin{aligned}[b] & \int_{{\mathbb {R}}^{N}} h(x)g(z)z^{k}\varphi^{2}\,dx\\ &\quad \le C \int_{{\mathbb {R}}^{N}} \vert \nabla\varphi \vert ^{2}z^{k+1} \,dx \\ &\quad \le C \biggl( \int_{{\mathbb {R}}^{N}} h(x)g(z)z^{k}\varphi^{2}\,dx \biggr)^{\frac {1}{\lambda}} \biggl( \int_{{\mathbb {R}}^{N}} \bigl\vert h(x) \bigr\vert ^{-\frac{\lambda'}{\lambda}} \vert \nabla \varphi \vert ^{2\lambda'}\varphi^{-\frac{2\lambda'}{\lambda }}z^{(k+1)\lambda'} \bigl(gz^{k}\bigr)^{-\frac{\lambda'}{\lambda}}\,dx \biggr)^{\frac{1}{\lambda'}} \\ &\quad \le C \biggl( \int_{{\mathbb {R}}^{N}}h(x)g(z)z^{k}\varphi^{2}\,dx \biggr)^{\frac{1}{\lambda}} \biggl( \int_{{\mathbb {R}}^{N}} \bigl\vert h(x) \bigr\vert ^{-\frac{\lambda'}{\lambda}} \vert \nabla \varphi \vert ^{2\lambda'}\varphi^{-\frac{2\lambda'}{\lambda}} \,dx \biggr)^{\frac{1}{\lambda'}}. \end{aligned} \end{aligned}$$
(2.39)

Obviously, inequality (2.39) implies

$$\begin{aligned} \int_{{\mathbb {R}}^{N}}h(x)g(z)z^{k}\varphi^{2}\,dx\le C \int_{{\mathbb {R}}^{N}} \bigl\vert h(x) \bigr\vert ^{-\frac{\lambda'}{\lambda}} \vert \nabla \varphi \vert ^{2\lambda'}\varphi^{-\frac{2\lambda'}{\lambda}} \,dx \end{aligned}$$
(2.40)

and (2.26) follows. □

Proof of Theorem 1.2

Setting \(x=R\xi\) in (2.26), we get

$$\begin{aligned} \begin{aligned}[b] \int_{{\mathbb {R}}^{N}}h(x)g(z)z^{k}\varphi^{2}\,dx&\le C \int_{{\mathbb {R}}^{N}} \bigl\vert h(x) \bigr\vert ^{-\frac{k+1}{q-1}} \vert \nabla \varphi \vert ^{\frac{2(q+k)}{q-1}}\varphi^{-\frac{2(k+1)}{q-1}} \,dx \\ &\le CR^{\theta} \int_{1\le \vert \xi \vert \le 2} \biggl[\frac{ \vert \varphi_{0}'( \vert \xi \vert ) \vert }{\varphi_{0}^{1-\frac {q-1}{q+k}}( \vert \xi \vert )} \biggr]^{\frac{2(q+k)}{q-1}}\,d\xi \le CR^{\theta}\beta_{0}^{\frac{2(q+k)}{q-1}}, \end{aligned} \end{aligned}$$
(2.41)

where assumption (1.11) on \(h(x)\) has been used, C is a positive constant independent of R, and

$$\begin{aligned} \theta=N-\frac{2(q+k)+a(k+1)}{q-1}. \end{aligned}$$
(2.42)

Clearly, if \(\theta<0\), the desired result follows by letting \(R\to\infty\) in (2.41). In the following, we will show that some appropriate \(k=k(q)\) can be chosen such that \(\theta<0\). Let \(X(t)\) be the function defined in (1.13). Obviously, we have

$$\begin{aligned} \lim_{t\to5/2}X(t)=\gamma_{0}(a):=6+2a, \qquad \lim_{t\to+\infty}X(t)=\gamma_{\infty}(a):=10+4a. \end{aligned}$$
(2.43)

Note that \(X(t)=2+(2+a)Y(t)\ (t\ge5/2)\) with

$$\begin{aligned} Y(t)=\frac{k_{0}(t)+1}{t-1}=2+\frac{\sqrt{4t^{2}-12t+5}}{t-1} \end{aligned}$$
(2.44)

and

$$\begin{aligned} X'(t)=\frac{(2+a)(2t+1)}{(t-1)^{2}}\bigl(4t^{2}-12t+5 \bigr)^{-1/2}>0,\quad\forall t>5/2. \end{aligned}$$
(2.45)

So, the function \(X(t)\) is increasing and \(6+2a< X(t)<10+4a\) for \(t>5/2\).

Therefore, if \(N\le6+2a\), we have \(N< X(t)\) for any \(t>5/2\). Hence, if we fix \(k\in(1, k_{0}(t))\) suitably near \(k_{0}(t)\), we obtain

$$\begin{aligned} N< \frac{2[t+k_{0}(t)]+a[k_{0}(t)+1]}{t-1}. \end{aligned}$$
(2.46)

For this reason, the desired result follows by letting \(R\to\infty\) in (2.41).

Assume now \(\gamma_{0}(a)< N<\gamma_{\infty}(a)\). Since \(X(t)\) is increasing, we get in this case a critical value \(q_{c}>5/2\) such that \(N< X(q)\) for \(q_{c}< q\). From this, the desired result follows again by letting \(R\to\infty\) in (2.41). Clearly, \(q_{c}\) may be determined from the equation \(N=X(q_{c})\). Then we complete the proof of Theorem 1.2. □

3 Concluding remarks

In this paper, we have considered a model described by the quasilinear Schrödinger equation. Nonexistence of stable solutions is proved.