1 Introduction and preliminaries

Consider the the following Schrödinger-Poisson system:

$$ \textstyle\begin{cases} -\Delta u+\phi u = f(x,u) , & x\in\Omega,\\ -\Delta\phi=u^{2}, & x \in\Omega,\\ u=\phi=0, & x \in\partial\Omega, \end{cases} $$
(1.1)

where Ω is a smooth bounded domain in \(\mathbb{R}^{3}\), and \(f\in C(\Omega\times\mathbb{R},\mathbb{R})\).

System (1.1) is related to the stationary analogue of the nonlinear parabolic Schrödinger-Poisson system

$$ \textstyle\begin{cases} -i\frac{\partial\psi}{\partial t}=-\Delta\psi+\phi(x)\psi- \vert \psi \vert ^{p-2}\psi & \text{in } \Omega,\\ -\Delta\phi= \vert \psi \vert ^{2} & \text{in }\Omega,\\ \psi=\phi=0 & \text{on } \partial\Omega. \end{cases} $$
(1.2)

The first equation in (1.2) is called the Schrödinger equation, which describes quantum particles interacting with the electromagnetic field generated by the motion. An interesting class of Schrödinger equations is the case where the potential \(\phi(x)\) is determined by the charge of the wave function itself, that is, when the second equation in (1.2) (Poisson equation) holds. For more details as regards the physical relevance of the Schrödinger-Poisson system, we refer to [1–4].

Recently, Schrödinger-Poisson systems on unbounded domains or on the whole space \(\mathbb{R}^{N}\) have attracted a lot of attention. Many solvability conditions on the nonlinearity have been given to obtain the existence and multiplicity of solutions for Schrödinger-Poisson systems in \(\mathbb{R}^{N}\), we refer the readers to [4–23] and references therein.

Compared with the whole space case, there are few works concerning the Schrödinger-Poisson system on a bounded domain; see, for instance, [20, 24–31]. Ruiz and Siciliano [26] studied the following system:

$$ \textstyle\begin{cases} -\Delta u+\lambda\phi u =f(x,u) & \text{in } \Omega,\\ -\Delta\phi=u^{2}, & x \in\Omega,\\ u=\phi=0, & x\in\partial\Omega, \end{cases} $$
(1.3)

where \(\lambda>0\) is a parameter. Using variational methods, the authors investigate the existence, nonexistence, and multiplicity of solutions when \(f(x,u)= \vert u \vert ^{p-1}u\) with \(p\in(1,5)\). Alves and Souto [29] studied system (1.3) when f has a subcritical growth. They obtained the existence of least-energy nodal solution by means of variational methods. Siciliano [25] studied system (1.3) with \(f(x,u)= \vert u \vert ^{p-2}u, p\in(2,6)\). By means of Ljusternik-Schnirelmann theory the author proved that problem (1.3) has at least \(\operatorname{cat}_{\overline{\Omega}}(\overline{\Omega })+1\) solutions for p near the critical Sobolev exponent 6, where cat denotes the Ljusternik-Schnirelmann category. Using a new sign-changing version of the symmetric mountain pass theorem, Batkam [27] proved the existence of infinitely many sign changing solutions for the following class of Schrödinger-Poisson systems:

$$ \textstyle\begin{cases} -\Delta u+\lambda\phi u =f(x,u)+\lambda u^{5} & \text{in } \Omega,\\ -\Delta\phi=u^{2}, & x \in\Omega,\\ u=\phi=0, & x\in\partial\Omega, \end{cases} $$
(1.4)

where \(\lambda\geq0\) is a parameter, and \(f\in C(\Omega\times \mathbb{R},\mathbb{R})\) satisfies the well-known Ambrosetti-Rabinowitz condition, that is, there exists \(\mu>4\) such that

$$ 0< \mu F(x,u)\leq uf(x,u),\quad \forall u\neq0, $$
(1.5)

where \(F(x,u)=\int_{0}^{u} f(x,s)\,ds\). Ba and He [28] considered system (1.1) with a general 4-superlinear nonlinearity f. They proved the existence of ground state solution for system (1.1) by the aid of the Nehari manifold. Moreover, they also obtained the existence of infinitely many solutions for system (1.1)

Before we state the main results of this paper, we first introduce the variational framework of problem (1.1).

Let \(H:=H_{0}^{1}(\Omega)\) be the Sobolev space equipped with the inner product and norm

$$ (u,v)= \int_{\Omega}\nabla u\cdot\nabla v \,dx, \qquad\Vert u \Vert =(u,u)^{\frac{1}{2}}. $$

We denote by \(\vert \cdot \vert _{p}\) the usual \(L^{p}\)-norm. Since Ω is a bounded domain, \(H \hookrightarrow L^{p}(\Omega)\) continuously for \(p \in[1,6]\) and compactly for \(p\in[1,6)\), and for every \(p\in[1,6]\), there exists \(\gamma_{p}>0\) such that

$$ \vert u \vert _{p}\leq\gamma_{p} \Vert u \Vert ,\quad \forall u \in H. $$
(1.6)

Recall that a function \(u\in H\) is called a weak solution of (1.1) if

$$ \int_{\Omega}\nabla u \nabla v + \int_{\Omega}\phi u v \,dx = \int_{\Omega}f(x,u)v\,dx,\quad \forall v\in H. $$
(1.7)

We have the following lemma from [1, 20].

Lemma 1.1

For each \(u\in H\), there exists a unique element \(\phi_{u}\in H\) such that \(-\Delta\phi_{u}=u^{2}\); moreover, \(\phi_{u}\) has the following properties:

  1. (1)

    there exists \(a>0\) such that \(\Vert \phi_{u} \Vert \leq a \Vert u \Vert ^{2}\) and

    $$ \int_{\Omega} \vert \nabla\phi_{u} \vert ^{2}\,dx= \int_{\Omega}\phi_{u} u^{2}\,dx\leq a \Vert u \Vert ^{4},\quad \forall u\in H; $$
    (1.8)
  2. (2)

    \(\phi_{u}\geq0\), \(\forall u\in H\);

  3. (3)

    \(\phi_{tu}=t^{2}\phi_{u}\), \(\forall t>0\) and \(u\in H\);

  4. (4)

    if \(u_{n}\rightharpoonup u\) in H, then \(\phi _{u_{n}}\rightharpoonup\phi_{u}\) in H, and

    $$ \lim_{n\rightarrow+\infty} \int_{\Omega}\phi_{u_{n}}u_{n}^{2}\,dx= \int _{\Omega}\phi_{u} u^{2}\,dx. $$
    (1.9)

By the lemma we have that \((u,\phi)\in H\times H\) is a solution of (1.1) if and only if \(\phi= \phi_{u}\) and \(u \in H\) is a solution of the following nonlocal problem:

$$ \textstyle\begin{cases} -\Delta u+\phi_{u} u=f(x,u) &\text{in } \Omega,\\ u=0 & \text{on } \partial\Omega. \end{cases} $$

We define the functional \(I:H \rightarrow\mathbb{R}\) by

$$ I(u)=\frac{1}{2} \int_{\Omega} \vert \nabla u \vert ^{2}\,dx + \frac{1}{4} \int_{\Omega}\phi_{u} u^{2}\,dx- \int_{\Omega}F(x,u)\,dx. $$
(1.10)

Using \((F_{1})\) and the Sobolev embedding theorem, we can prove easily that \(I\in C^{1}(H,\mathbb{R})\) with

$$ I^{\prime}(u)v= \int_{\Omega}\nabla u \nabla v \,dx + \int_{\Omega}\phi_{u} uv\,dx- \int_{\Omega}f(x,u)v\,dx,\quad \forall u,v\in H. $$
(1.11)

Consider the following eigenvalue problems:

$$ \textstyle\begin{cases} -\Delta u=\lambda u & \text{in } \Omega,\\ u=0 &\text{on } \partial\Omega, \end{cases} $$
(1.12)

and

$$ \textstyle\begin{cases} - \Vert u \Vert ^{2}\Delta u=\mu u^{3} & \text{in } \Omega,\\ u=0 &\text{on } \partial\Omega. \end{cases} $$
(1.13)

Denote by \(0<\lambda_{1}<\lambda_{2}<\cdots\) the distinct eigenvalues of the problem (1.12). It is well known that \(\lambda_{1}\) can be characterized as

$$ \lambda_{1}=\inf \bigl\{ \Vert u \Vert ^{2}: u\in H, \vert u \vert _{2}=1 \bigr\} , $$

and \(\lambda_{1}\) is achieved by the first eigenfunction \(\varphi_{1}>0\).

We say that μ is an eigenvalue of problem (1.13) if there is a nonzero \(u\in H\) such that

$$ \Vert u \Vert ^{2} \int_{\Omega}\nabla u \nabla v \,dx=\mu \int_{\Omega}u^{3} v\,dx,\quad v\in H, $$

and u is called an eigenvector corresponding to the eigenvalue μ. Denote by \(0<\mu_{1}<\mu_{2}<\cdots\) all distinct eigenvalues of problem (1.13). Furthermore, \(\mu_{1}\) can be characterized as

$$ \mu_{1}:=\inf \bigl\{ \Vert u \Vert ^{4}: u\in H, \vert u \vert _{4}=1 \bigr\} , $$
(1.14)

and \(\mu_{1}\) can be achieved by some function \(\psi_{1}\) with \(\psi_{1} >0\) in Ω (see [32, 33]).

Motivated by the works mentioned, in this paper, we study the existence of nontrivial state solutions of problem (1.1) by means of the mountain pass theorem. Moreover, establish the existence of infinitely many solutions by using the symmetric mountain pass theorem. To state the main results of this paper, we impose the following assumptions on f and its primitive F:

\((F_{1})\) :

There exist \(p\in(2,6)\) and a positive constant C such that

$$ \bigl\vert f(x,u) \bigr\vert \leq C\bigl(1+ \vert u \vert ^{p-1}\bigr); $$
\((F_{2})\) :

\(\limsup_{t\rightarrow0}\frac {2F(x,t)}{t^{2}}<\lambda_{1}\) uniformly in \(x \in\Omega\);

\((F_{3})\) :

\(\liminf_{ \vert t \vert \rightarrow\infty}\frac {4F(x,t)}{at^{4}}>\mu_{1}\) uniformly in \(x \in\Omega\), where a is the constant defined in Lemma 1.1(1);

\((F_{4})\) :

There exist \(\rho\in(0,\lambda_{1})\) and a constant \(L\gg 1\) such that

$$ 4F(x,t) \leq f(x,t)t+ \rho \vert t \vert ^{\delta},\quad \forall x \in \Omega, \vert t \vert \geq L, $$

where \(\delta\in[1,2]\).

\((F_{5})\) :

\(f(x,-t)=-f(x,t)\) for all \((x,t)\in \Omega\times \mathbb{R}\).

The main results of this paper are the following:

Theorem 1.2

Assume that \((F_{1})\)-\((F_{4})\) hold. Then system (1.1) has at least one nontrivial solution.

Theorem 1.3

Assume that \((F_{1})\)-\((F_{5})\) hold. Then, system (1.1) possesses an unbounded sequence of nontrivial solutions \(\{(u_{k},\phi_{k})\}\in H\times H\) such that

$$ \frac{1}{2} \int_{\Omega} \vert \nabla u_{k} \vert ^{2}\,dx +\frac{1}{4} \int_{\Omega}\phi_{k} u_{k}^{2}\,dx- \int_{\Omega}F(x,u_{k})\,dx\rightarrow+\infty $$

as \(k\rightarrow\infty\).

Remark 1.4

  1. (1)

    In this paper, we do not need the well-known Ambrosetti-Rabinowitz condition (1.5), which plays a very important role in proving the boundedness of the Palais-Smale sequence. Moreover, it is easy to prove that (AR) condition implies that

    $$ \lim_{t\rightarrow\infty}\frac{F(x,t)}{t^{4}}=+\infty. $$

    Therefore, Theorem 1.3 extends and sharply improves Theorem 1.1 in [27].

  2. (2)

    Our assumptions \((F_{2})\)-\((F_{3})\) are weaker than the following assumptions:

    \((F'_{2})\) :

    \(\lim_{t\rightarrow0}\frac{f(x,t)}{t}=0\) uniformly in \(x \in\Omega\);

    \((F'_{3})\) :

    \(\lim_{ \vert t \vert \rightarrow\infty}\frac {F(x,t)}{t^{4}}=+\infty\) uniformly in \(x \in\Omega\).

    On the other hand, noting that the variant Nehari monotonicity condition,

    (VNC):

    \(\frac{f(x,u)}{ \vert u \vert ^{3}}\) is nondecreasing on \((-\infty ,0)\cup(0,+\infty)\),

    implies that

    $$ 4F(x,u)\leq f(x,u)u, \quad\forall u\in\mathbb{R}. $$

    Then, assumption \((F_{4})\) it is also weaker than (VNC). Consequently, our results generalize and improve the results of Ba and He [28].

  3. (3)

    As a function f satisfying \((F_{1})\)-\((F_{5})\), set

    $$ F(x,s)=\frac{a}{4}\mu_{2} s^{4}+\frac{\lambda_{1}}{4}s^{2},\quad s\in \mathbb{R}. $$

    Then by a simple computation we obtain

    $$ f(x,s)= a\mu_{2} s^{3}+\frac{\lambda_{1}}{2}s. $$

    So, it is easy to check that f satisfies \((F_{1})\), \((F_{2})\), \((F_{3})\), and \((F_{5})\). Furthermore, we have

    $$ f(x,u)u-4F(x,u)= -\frac{\lambda_{1}}{2}u^{2}, $$

    which implies that f satisfies \((F_{4})\). On the other hand, for \(\mu >4\) and \(u>1\), we have

    $$ f(x,u)u-\mu F(x,u)= - \biggl(\frac{\mu}{4}-1 \biggr)a\mu_{2} u^{4}-\frac {\lambda_{1}}{2} \biggl(\frac{\mu}{2}-1 \biggr)u^{2} \rightarrow-\infty \quad\text{as } u\rightarrow\infty. $$

    Hence f does not satisfy (AR) condition. Moreover, it is clear that f does not satisfy \((F'_{2})\)-\((F'_{3})\).

This paper is organized as follows. Using the mountain pass theorem, we prove Theorem 1.2 in Section 2. In Section 3, by using the symmetric mountain pass theorem we prove Theorem 1.3.

2 Proof of Theorem 1.2

First, we introduce the mountain pass theorem, which is the main tool to prove Theorem 1.2.

Definition 2.1

The functional I satisfies the Palais-Smale condition at level \(c\in \mathbb{R}\), denoted by \((PS)_{c}\), if every sequence \(\{u_{n}\}\subset H\) such that

$$ I(u_{n})\rightarrow c \quad\text{and}\quad I'(u_{n}) \rightarrow0 $$
(2.1)

as \(n \rightarrow+\infty\) possesses a strongly convergent subsequence.

Proposition 2.2

([34], mountain pass theorem)

Let E be a real Banach space, and let \(I\in C^{1}(E,R)\) with \(I(0)=0\) satisfying the (PS) condition. Suppose that

\((I_{1})\) :

there exist two constants \(r,\alpha>0\) such that \(I| _{\partial B_{r}}\geq\alpha\).

\((I_{2})\) :

there exists \(e \in E\setminus\overline{B}_{r}\) such that \(I(e)\leq0\).

Then I possesses a critical value \(c\geq\alpha\), which can be characterized as

$$ c =\inf_{\gamma\in\Gamma}\max_{t \in[0,1]} I \bigl(\gamma(t)\bigr), $$
(2.2)

where \(\Gamma= \{\gamma\in C([0,1],E):\gamma(0)=0,\gamma(1)=e\}\).

Lemma 2.3

Under assumptions \((F_{1})\) and \((F_{4})\), I satisfies the \((PS)\) condition.

Proof

Let \({u_{n}}\subset H\) be such that

$$ I(u_{n})\rightarrow c \quad\text{and}\quad I'(u_{n}) \rightarrow0 \quad\text{as } n\rightarrow\infty. $$
(2.3)

We claim that \({u_{n}}\) is bounded in H. Otherwise, we can assume that \(\Vert u_{n} \Vert \rightarrow\infty\). For large n, set \(\Omega_{n}=\{x\in \Omega: \vert u_{n}(x) \vert \geq L\}\) and \(H(x,u_{n})=f(x,u_{n})u_{n}-4F(x,u_{n})\). Then, for large n, it follows from (2.3) and \((F_{4})\) that there exists a constant \(C_{1}>0\) such that

$$\begin{aligned} 1+c+ \Vert u_{n} \Vert &\geq I(u_{n})-\frac{1}{4} I'(u_{n})u_{n} \\ &=\frac{1}{4} \Vert u_{n} \Vert ^{2}+ \frac{1}{4} \int_{\Omega}H(x,u_{n})\,dx \\ &=\frac{1}{4} \Vert u_{n} \Vert ^{2}+ \frac{1}{4} \int_{\Omega_{n}} H(x,u_{n})\,dx+\frac {1}{4} \int_{\Omega\setminus\Omega_{n}} H(x,u_{n})\,dx \\ &\geq\frac{1}{4} \Vert u_{n} \Vert ^{2}- \frac{\rho}{4} \int_{\Omega_{n}} \bigl\vert u_{n}(x) \bigr\vert ^{\delta}\,dx -C_{1} \\ &\geq\frac{1}{4} \Vert u_{n} \Vert ^{2}- \frac{\rho}{4} \int_{\Omega_{n}} \bigl\vert u_{n}(x) \bigr\vert ^{2} \,dx -C_{1} \\ &\geq\frac{1}{4} \Vert u_{n} \Vert ^{2}- \frac{\rho}{4\lambda_{1}} \Vert u_{n} \Vert ^{2} -C_{1} \\ &=\frac{\lambda_{1}-\rho}{4\lambda_{1}} \Vert u_{n} \Vert ^{2} -C_{1}, \end{aligned}$$

which is a contradiction since \(\rho\in(0,\lambda_{1})\). Therefore \(\{ u_{n}\}\) is bounded in H. Since \(\{u_{n}\}\) is bounded in Hm we may assume that there exists \(u\in H\) such that

$$ \begin{aligned} & u_{n} \rightharpoonup u \quad\text{in } H, \\ & u_{n} \rightarrow u \quad\text{in } L^{p}(\Omega), p\in[1,6), \\ & u_{n}(x)\rightarrow u \quad\text{for a.e. } x\in\Omega. \end{aligned} $$
(2.4)

Hence, by \((F_{1})\) we know that there is \(C_{1}>0\) such that

$$\begin{aligned} \int_{\Omega}f(x,u_{n}) (u-u_{n})\,dx&\leq \biggl( \int_{\Omega}\bigl\vert f(x,u_{n}) \bigr\vert ^{\frac{p}{p-1}}\,dx \biggr)^{\frac{p-1}{p}} \biggl( \int_{\Omega} \vert u-u_{n} \vert ^{p} \,dx \biggr)^{\frac{1}{p}} \\ &\leq2C \biggl[ \int_{\Omega}\bigl( \vert u_{n} \vert ^{p}+1\bigr)\,dx \biggr]^{\frac{p-1}{p}} \vert u-u_{n} \vert _{p} \\ &\leq C_{1} \vert u-u_{n} \vert _{p} \rightarrow0, \quad\text{as } n\rightarrow\infty. \end{aligned}$$
(2.5)

On the other hand, by Lemma 1.1, (2.7), and the Hölder inequality we have

$$\begin{aligned} \int_{\Omega}\phi_{u_{n}}u_{n}(u_{n}-u)\,dx & \leq \int_{\Omega} \vert \phi _{u_{n}} \vert \vert u_{n} \vert \vert u_{n}-u \vert \,dx \\ & \leq \vert \phi_{u_{n}} \vert _{6} \vert u_{n} \vert _{3} \vert u_{n}-u \vert _{2} \\ &\leq\gamma_{6} \Vert \phi_{u_{n}} \Vert \gamma_{3} \Vert u_{n} \Vert \vert u_{n}-u \vert _{2} \\ &\leq C \Vert u_{n} \Vert ^{3} \vert u_{n}-u \vert _{2}\rightarrow0 \end{aligned}$$
(2.6)

as \(n\rightarrow\infty\). Therefore it follows from (2.1), (2.7), (2.8), and (2.9) that

$$\begin{aligned} &\Vert u_{n} \Vert ^{2}-(u_{n},u)+ \int_{\Omega}\phi_{u_{n}}u_{n}(u_{n}-u)\,dx- \int_{\Omega}f(x,u_{n}) (u-u_{n})\,dx\\ &\quad =I'(u_{n}) (u_{n}-u) \rightarrow0 \quad\text{as } n\rightarrow\infty, \end{aligned}$$

which implies that

$$ \Vert u_{n} \Vert \rightarrow \Vert u \Vert \quad\text{as } n \rightarrow\infty. $$

Hence, \(u_{n}\rightarrow u\) in H due to the uniform convexity of H. Consequently, \(\{u_{n}\}\) has a convergent subsequence in H, and then I satisfies the (PS) condition. The proof is completed. □

Lemma 2.4

Suppose that \((F_{1}),(F_{2})\), and \((F_{3})\) hold. Then the functional I satisfies conditions \((I_{1})\)-\((I_{2})\) in Proposition 2.2.

Proof

We first claim that there exist \(r,\alpha>0\) such that \(I(u)\geq \alpha\) for all \(u\in H\) with \(\Vert u \Vert =r\). Indeed, for small \(\varepsilon>0\), by \((F_{1})\)-\((F_{2})\) there exists a constant \(C_{2}>0\) such that

$$ F(x,u)\leq\frac{1}{2}(\lambda_{1} - \varepsilon)u^{2}+C_{2} \vert u \vert ^{p}. $$
(2.7)

Therefore (1.6) and (2.7) imply that

$$\begin{aligned} I(u)&\geq\frac{1}{2} \Vert u \Vert ^{2}+\frac{1}{4} \int\phi_{u} u^{2}\,dx-\frac {1}{2}( \lambda_{1}-\varepsilon) \int_{\Omega} \vert u \vert ^{2}\,dx-C_{2} \int_{\Omega} \vert u \vert ^{p}\,dx \\ &\geq\frac{1}{2} \biggl( 1-\frac{\lambda_{1}-\varepsilon}{\lambda _{1}} \biggr) \Vert u \Vert ^{2}-C_{2}\gamma_{p}^{p} \Vert u \Vert ^{p}. \end{aligned}$$
(2.8)

Since \(2< p<6\), we can choose small \(r>0\) such that

$$ I(u)\geq\frac{1}{2} \biggl(1-\frac{\lambda_{1}-\varepsilon}{\lambda _{1}}-C_{2} \gamma_{p}^{p} r^{p-2} \biggr)r^{2}:= \alpha>0 $$

whenever \(u\in H\) with \(\Vert u \Vert =r\).

Next, we prove that there exists \(e\in H\) with \(\Vert e \Vert >r\) such that \(I(e)<0\). Indeed, for small \(\varepsilon>0\), by the definition of \(\mu_{1}\) we can choose \(v \in H, \vert v \vert _{4}=1\), satisfying

$$ \Vert v \Vert ^{4}\leq\mu_{1}+ \frac{\varepsilon}{2}. $$
(2.9)

It follows from \((F_{1})\) and \((F_{3})\) that there exists a constant \(M>0\) such that

$$ F(x,t)\geq\frac{a}{4}(\mu_{1}+ \varepsilon)t^{4}-M. $$
(2.10)

Hence, combining (2.9) and (2.10) with Lemma 1.1(1), we get

$$\begin{aligned} I(tv)&= \frac{1}{2}t^{2} \Vert v \Vert ^{2}+\frac{1}{4}t^{4} \int_{\Omega}\phi_{v} v^{2}\,dx- \int_{\Omega}F(x,tv)\,dx \\ &\leq\frac{1}{2}t^{2} \Vert v \Vert ^{2}+ \frac{a}{4}t^{4} \Vert v \Vert ^{4}- \frac{a}{4}(\mu _{1}+\varepsilon)t^{4}+M \vert \Omega \vert \\ &\leq\frac{1}{2}t^{2} \Vert v \Vert ^{2}+ \frac{a}{4}t^{4} \biggl(\mu_{1}+\frac {\varepsilon}{2} \biggr)-\frac{a}{4}(\mu_{1}+\varepsilon)t^{4}+M \vert \Omega \vert \\ &\leq-\frac{a}{8}\varepsilon t^{4}+\frac{1}{2} t^{2} \Vert v \Vert ^{2}+M \vert \Omega \vert , \end{aligned}$$

which implies that

$$ I(tv)\rightarrow-\infty \quad\text{as } \vert t \vert \rightarrow\infty. $$

Hence we conclude that there exists a sufficiently large \(t^{*}>0\) such that \(t^{*} v > \rho\) and \(I(t^{*} v)<0\). The conclusion follows by taking \(e=t^{*}v\). □

Proof of Theorem 1.2

Under the conditions of Theorem 1.2, we have that \(I\in C^{1}(E,\mathbb{R})\) with \(I(0)=0\) and I satisfies the (PS) condition due to Lemma 2.3. Moreover, by Lemma 2.4, I satisfies conditions \((I_{1})\)-\((I_{2})\) in Proposition 2.2. Then I has at least one critical point \(u\in H\) such that \(I(u)\geq\alpha\). Thus system (1.1) has at least one nontrivial solution. □

3 Proof of Theorem 1.3

In this section, we prove Theorem 1.3 by using the following symmetric mountain pass theorem.

Proposition 3.1

([35])

Let E be an infinite-dimensional Banach space, and let \(I\in C^{1}(E,R)\) be even and satisfy the (PS) condition and \(I(0)=0\). Let \(X=Y\oplus Z\), where Y is finite-dimensional, and I satisfies

\((H_{1})\) :

there exist two constants \(r,\alpha>0\) such that \(I| _{\partial B_{r}\cap Z}\geq\alpha\);

\((H_{2})\) :

for each finite-dimensional subspace \(\widetilde {E}\subset E\), there exists \(R=R(\widetilde{E})>0\) such that \(I\leq0\) on \(\widetilde{E}\setminus B_{R}\).

Then I possesses an unbounded sequence of critical values.

Let \(\{e_{i}\}\) be an orthonormal basis of H and define \(X_{i}=\mathbb{R}e_{i}\),

$$ Y_{k}=\bigoplus_{i=1}^{k} X_{i}, \qquad Z_{k}=\bigoplus_{i=k}^{\infty} X_{i},\quad k\in\mathbb{Z}. $$
(3.1)

Lemma 3.2

Assume that \((F_{1})\) and \((F_{2})\) hold. Then, there exist constants \(r,\alpha>0\) and \(m\in\mathbb{N}\) such that \(I| _{\partial{B_{r}}\cap Z_{m}}\geq\alpha\).

Proof

Set

$$ \beta_{k}(p)=\sup_{u\in Z_{k}, \Vert u \Vert =1} \vert u \vert _{p}, \quad\forall k\in\mathbb {N}, 1 \leq p < 6. $$
(3.2)

Since H is compactly embedded into \(L^{p}(\Omega)\) for \(1\leq p<6\), we know from [34, Lemma 3.8] that

$$ \beta_{k}(p)\rightarrow0\quad \text{as } k\rightarrow\infty. $$
(3.3)

Combining (1.10) and (2.7) with (3.2) we have

$$\begin{aligned} I(u)&\geq\frac{1}{2} \Vert u \Vert ^{2}+\frac{1}{4} \int\phi_{u} u^{2}\,dx-\frac {1}{2}( \lambda_{1}-\varepsilon) \int_{\Omega} \vert u \vert ^{2}\,dx-C_{2} \int_{\Omega} \vert u \vert ^{p}\,dx \\ &\geq\frac{1}{2} \Vert u \Vert ^{2}-\frac{1}{2}( \lambda_{1}-\varepsilon)\beta _{k}^{2}(2) \Vert u \Vert ^{2}-C_{2}\beta_{k}^{p}(p) \Vert u \Vert ^{p}. \end{aligned}$$
(3.4)

It follows from (3.3) that there exist a large positive integer \(m\in\mathbb{N}\) such that

$$ \beta_{k}^{2}(2)\leq\frac{1}{2(\lambda_{1}-\varepsilon)} \quad\text{and}\quad \beta_{k}^{p}(p)\leq\frac{1}{4C_{2}}, \quad\forall k\geq m. $$

Then, we conclude from (3.4) that

$$ I(u)\geq\frac{1}{4}\bigl( \Vert u \Vert ^{2}- \Vert u \Vert ^{p}\bigr). $$

Hence, since \(p>2\), there exist \(r\in(0,1)\) such that

$$ I(u)\geq\frac{1}{4} r^{2}\bigl(1-r^{p-2}\bigr)= \alpha>0, \quad \forall u\in Z_{m}, \Vert u \Vert =r. $$

The proof is completed. □

Lemma 3.3

Assume that \((F_{1})\), \((F_{2})\), and \((F_{3})\) hold. Then, for any finite-dimensional subspace \(\widetilde{H}\subset H\), there exists \(R=R(\widetilde{H})>0\) such that

$$ I(u)\leq0,\quad \forall u\in\widetilde{H}\setminus B_{R}. $$

Proof

Let \(\widetilde{H}\subset H\) be a finite-dimensional subspace. By the equivalence of norms in finite-dimensional spaces, there exists a constant \(b_{p} > 0\) such that

$$ \vert u \vert _{p}\geq b_{p} \Vert u \Vert , \quad\forall u \in\widetilde{H}, p\in[2,6). $$
(3.5)

Therefore, combining (1.10), (2.10), (3.5), and Lemma 1.1(1), we have

$$\begin{aligned} I(u)&= \frac{1}{2} \Vert u \Vert ^{2}+\frac{1}{4} \int_{\Omega}\phi_{u} u^{2}\,dx- \int_{\Omega}F(x,u)\,dx \\ &\leq\frac{1}{2} \Vert u \Vert ^{2}+\frac{a}{4} \Vert u \Vert ^{4}-\frac{a}{4}(\mu _{1}+ \varepsilon) \int_{\Omega} \vert u \vert ^{4}\,dx+M \vert \Omega \vert \\ &\leq\frac{1}{2} \Vert u \Vert ^{2}+\frac{a}{4} \bigl(1-\mu_{1}b_{4}^{4}-\varepsilon b_{4}^{4} \bigr) \Vert u \Vert ^{4}+M \vert \Omega \vert . \end{aligned}$$

Choosing \(\varepsilon=\frac{1}{b_{4}^{4}}\), it follows from the last inequality that

$$ I(u)\leq\frac{1}{2} \Vert u \Vert ^{2}-\frac{a}{4} \mu_{1} b_{4}^{4} \Vert u \Vert ^{4}+M \vert \Omega \vert ,\quad \forall u \in\widetilde{H}. $$

Hence there exists \(R=R(\widetilde{H})>0\) large enough such that \(I| _{\widetilde{H}\setminus B_{R}}\leq0\). This completes the proof. □

Proof of Theorem 1.3

Clearly, \(I\in C^{1}(E,\mathbb{R})\), \(I(0)=0\), and I is even by \((F_{5})\). Lemma 2.3 implies that I satisfies the (PS) condition. On the other hand, Lemmas 3.2 and 3.3 imply that I satisfies conditions \((H_{1})\)-\((H_{2})\) of Proposition 3.1. Hence I has a sequence of nontrivial critical points \(\{ (u_{k},\phi_{k})\}\subset H\times H\) such that

$$ \lim_{k\rightarrow\infty}I(u_{k})=\frac{1}{2} \int_{\Omega} \vert \nabla u_{k} \vert ^{2}\,dx +\frac{1}{4} \int_{\Omega}\phi_{u_{k}} u_{k}^{2}\,dx- \int_{\Omega}F(x,u_{k})\,dx=+\infty. $$

Thus problem (1.1) possesses infinitely many nontrivial solutions. □

4 Conclusions

In this paper, we have established two results on the existence of nontrivial solutions and infinitely many solutions. Moreover, compared with the existing results on this problem, we have introduced somewhat weaker assumptions on the nonlinearity f. Therefore, our results extend and improve some recent results in the literature.