1 Introduction

Let

$$ (-{{\Delta }_{\mathbb{H}}})_{p}^{s}u(\xi )={{C}_{Q,s,p}}\mathit{PV} \int _{{{ \mathbb{H}}^{n}}} \frac{{{ \vert u(\xi )-u(\eta ) \vert }^{p-2}}(u(\xi )-u(\eta ))}{ \vert \eta ^{-1}\circ \xi \vert _{\mathbb{H}}^{Q+sp}}\,d \eta $$
(1.1)

be the fractional p-subLaplacian on the Heisenberg group \({{\mathbb{H}}^{n}}\), where \(0< s <1\), \(Q=2n+2\), \({{C}_{Q,s}}\) is a positive constant, and PV is the Cauchy principal value. In this paper we study the properties of cylindrical solutions to the fractional p-subLaplace equation

$$ (-{{\Delta }_{\mathbb{H}}})_{p}^{s}u(\xi )=f \bigl(u(\xi ) \bigr), $$
(1.2)

where \(2\leq p< \infty \).

Recall that the fractional Laplacian in \({{\mathbb{R}}^{n}}\) is a nonlocal pseudodifferential operator defined by

$$ {{(-\Delta )}^{\alpha }}u(x)={{C}_{n,\alpha }} \lim _{\varepsilon \to 0} \int _{{{\mathbb{R}}^{n}} \setminus {{B}_{\varepsilon }}(x)}{ \frac{u(x)-u(y)}{{{ \vert x-y \vert }^{n+2\alpha }}}}\,dy, $$
(1.3)

where \(0<\alpha <1\), \({{C}_{n,\alpha }}\) is a constant, and u belongs to the Schwartz space. Since the nonlocal property of the operator \({(-\Delta )}^{\alpha }\) brings new difficulties to investigate, Caffarelli and Silvestre in [4] developed the extension method which can reduce the nonlocal problem relating to \({(-\Delta )}^{\alpha }\) to a local one in higher dimensions. This method has been applied to deal with equations involving the fractional Laplacian, and fruitful results have been obtained, see [3] and the references therein. Chen et al. [7] developed a direct method of moving planes to handle the problem involving \({(-\Delta )}^{\alpha }\) for \(0<\alpha <1\), and this direct method has been used successfully to study symmetry, monotonicity, and nonexistence for many fractional Laplace equations, see [6, 7] and the references therein. Recently, Chen and Li [6] considered the fractional p-Laplacian

$$ (-\Delta )_{p}^{\alpha }u(x)={{C}_{n,\alpha ,p}} \mathit{PV} \int _{{{\mathbb{R}}^{n}}}{ \frac{{{ \vert u(x)-u(y) \vert }^{p-2}}(u(x)-u(y))}{{{ \vert x-y \vert }^{n+\alpha p }}}}\,dy $$
(1.4)

and obtained the radial symmetry and monotonicity of solutions to the equations involving operator (1.4).

To the elliptic equation

$$ -\Delta u=g(u) $$
(1.5)

in \({{\mathbb{R}}^{n}}\), Li and Ni [19] proved that the positive solutions to (1.5) are radially symmetric with the assumptions that the limit of u is zero at the infinity and \({g}'\le 0\) if u is sufficiently small. Under the same conditions, the authors in [6] extended the result in [19] to the fractional p-Laplace equation

$$ (-\Delta )_{p}^{\alpha }u=g(u), $$
(1.6)

and got the radial symmetry and monotonicity of the solutions. They also pointed out that the fractional p-Laplacian becomes p-Laplacian as \(\alpha \rightarrow 1\) and, furthermore, it reduces to −Δ when \(p=2\).

There are many interesting results about subLaplace and p-subLaplace equations on the Heisenberg group (see [13, 15, 17, 18] and [10, 11, 20, 21, 25–27]). There have been several different definitions of the fractional power subLaplacian in \({{\mathbb{H}}^{n}}\) (see [12, 14, 22] etc.). The definition of fractional power subLaplacian given by Roncal and Thangavelu in [22] is indeed a generalization of the definition given by Cowling and Haagerup in [9] about the heat semigroup. The fractional power subLaplace equations can also be studied by generalizing the extension method in [4] to \({{\mathbb{H}}^{n}}\), although the fractional power subLaplacian \((-{{\Delta }_{\mathbb{H}}})^{s}\) (\(0< s<1\)) does not have the concrete integral expression, for example, see [14] and [8] for \(s=\frac{1}{2}\). There are also some results of the fractional power subLaplacian which are the extension of [8], see [23, 24]. Note that the expression of fractional power subLaplacian on \({{\mathbb{H}}^{n}}\) (see [22])

$$ (-{{\Delta }_{\mathbb{H}}})^{s}u(\xi )={{C}_{Q,s}}\mathit{PV} \int _{{{ \mathbb{H}}^{n}}}{ \frac{u(\xi )-u(\eta )}{{{ \vert {{\eta }^{-1}}\circ \xi \vert }_{\mathbb{H}}}^{Q+2s}}}\,d \eta $$
(1.7)

is the special form of fractional p-subLaplacian (1.1). By extending the method of moving planes in [5–7] to \({{\mathbb{H}}^{n}}\), in this paper, we study the properties of the solutions to (1.2) on \({{\mathbb{H}}^{n}}\) and \(\mathbb{H}_{+}^{n}=\{\xi \in {{\mathbb{H}}^{n}}\mid t>0\}\).

Our main results are the following.

Theorem 1.1

Let\(0< s<1\), \(2\leq p<\infty \), and\(u\in {{L}_{sp}}({{\mathbb{H}}^{n}})\cap C_{\mathrm{loc}}^{1,1}({{\mathbb{H}}^{n}})\)be a nonnegative cylindrical solution to (1.2) with

$$ \lim_{{{ \vert \xi \vert }_{\mathbb{H}}}\to \infty } u(\xi )=0, $$
(1.8)

and suppose that\({f}'(a)\)is nonpositive and locally bounded forasufficiently small. Thenumust be symmetric and monotone with respect totabout some point in\({{\mathbb{H}}^{n}}\).

Theorem 1.2

Let\(0< s<1\), \(2\leq p<\infty \), and\(u\in {{L}_{sp}}({{\mathbb{H}}_{+}^{n}})\cap C_{\mathrm{loc}}^{1,1}({{ \mathbb{H}}_{+}^{n}})\)be a nonnegative cylindrical solution to the problem

$$ \textstyle\begin{cases} (-{{\Delta }_{\mathbb{H}}})_{p}^{s}u(\xi )=f(u(\xi )),\quad \xi \in \mathbb{H}_{+}^{n}, \\ u(\xi )=0,\quad \xi \notin \mathbb{H}_{+}^{n}, \end{cases} $$
(1.9)

and suppose thatusatisfies (1.8) and is lower semicontinuous on\(\bar{\mathbb{H}}_{+}^{n}\). If\(f(0)=0\), \({f}'(a)\)is nonpositive and locally bounded forasufficiently small, then\(u\equiv 0\).

Observe that Theorem 1.1 is the extension of symmetry and monotonicity of solutions to the fractional p-Laplace equation on \({{\mathbb{R}}^{n}}\) in [6] to the Heisenberg group, and Theorem 1.2 is the Liouville property on a half space in \({{\mathbb{H}}^{n}}\). When \(f(a)=-a+a^{q}\) (\(q>1\)), our results still hold.

The authors in [22] assumed that \(u\in C_{0}^{\infty }({{\mathbb{H}}^{n}})\) in (1.7). We point out that (1.1) is also well defined for \(u\in {{L}_{sp}}({{\mathbb{H}}^{n}})\cap C_{\mathrm{loc}}^{1,1}({{\mathbb{H}}^{n}})\), where

$$ {{L}_{sp}} \bigl({{\mathbb{H}}^{n}} \bigr)= \biggl\{ u:{{ \mathbb{H}}^{n}}\to \mathbb{R}\Bigm| \int _{{{\mathbb{H}}^{n}}}{ \frac{ \vert u(\xi ) \vert ^{p-1}}{1+{{ \vert \xi \vert }_{\mathbb{H}}}^{Q+sp}}\,d \xi < \infty } \biggr\} . $$

The paper is organized as follows. Section 2 collects some well-known results on \({\mathbb{H}}^{n}\), and we show that (1.1) is well defined for \(u\in {{L}_{sp}}({{\mathbb{H}}^{n}})\cap C_{\mathrm{loc}}^{1,1}({{\mathbb{H}}^{n}})\). In Sect. 3, we establish three maximum principles. Theorem 1.1 and Theorem 1.2 are proved in Sect. 4.

2 Preliminaries

The Heisenberg group \({\mathbb{H}}^{n}\) is the Euclidean space \({{\mathbb{R}}^{2n+1}}(n\ge 1)\) endowed with the group law ∘:

$$ \bar{\xi }\circ \xi = \Biggl(x+\bar{x},y+\bar{y},t+\bar{t}+2\sum _{i=1}^{n}{({{x}_{i}} {{{ \bar{y}}}_{i}}-{{y}_{i}} {{{\bar{x}}}_{i}})} \Biggr), $$
(2.1)

where \(\xi =({{x}_{1}},\ldots,{{x}_{n}},{{y}_{1}},\ldots,{{y}_{n}},t):=(x,y,t) \in {\mathbb{R}}^{n}\times {\mathbb{R}}^{n}\times {\mathbb{R}}\) and \(\bar{\xi }=(\bar{x},\bar{y},\bar{t})\). Denote by \(\delta _{\kappa }\) the dilations on \({{\mathbb{H}}^{n}}\), i.e.,

$$ {{\delta }_{\kappa }}(\xi )= \bigl(\kappa x,\kappa y,{{ \kappa }^{2}}t \bigr), \quad \kappa >0, $$
(2.2)

which satisfy \({{\delta }_{\kappa }}(\bar{\xi }\circ \xi )={{\delta }_{\kappa }}( \bar{\xi })\circ {{\delta }_{\kappa }}(\xi )\).

The left invariant vector fields corresponding to \({\mathbb{H}}^{n}\) are

$$\begin{aligned}& {{X}_{i}}=\frac{\partial }{\partial {{x}_{i}}}+2{{y}_{i}} \frac{\partial }{\partial t},\quad i=1,\ldots,n, \\& {{Y}_{i}}=\frac{\partial }{\partial {{y}_{i}}}-2{{x}_{i}} \frac{\partial }{\partial t},\quad i=1,\ldots,n, \\& T=\frac{\partial }{\partial {t}}. \end{aligned}$$

It is easy to check that \({{X}_{i}}\) and \({{Y}_{j}}\) satisfy

$$ [{{X}_{i}},{{Y}_{j}}]=-4T{{\delta }_{ij}}, \quad \quad [{{X}_{i}},{{X}_{j}}]=[{{Y}_{i}},{{Y}_{j}}]=0, \quad i,j=1,\ldots,n. $$

The Heisenberg gradient of a function u is defined by

$$ {{\nabla }_{\mathbb{H}}}u=({{X}_{1}}u, \ldots,{{X}_{n}}u,{{Y}_{1}}u,\ldots,{{Y}_{n}}u), $$
(2.3)

and the subLaplacian \({{\Delta }_{\mathbb{H}}}\) on \({\mathbb{H}}^{n}\) is

$$\begin{aligned} {{\Delta }_{\mathbb{H}}}:={}&\sum_{i=1}^{n} \bigl({{{X}_{i}}^{2}+{{Y}_{i}}^{2}} \bigr) \\ ={}&\sum_{i=1}^{n} \biggl({ \frac{{{\partial }^{2}}}{\partial {{x}_{i}}^{2}}+ \frac{{{\partial }^{2}}}{\partial {{y}_{i}}^{2}}+4{{y}_{i}} \frac{{{\partial }^{2}}}{\partial {{x}_{i}}\partial t}}-4{{x}_{i}} \frac{{{\partial }^{2}}}{\partial {{y}_{i}}\partial t}+4 \bigl({{x}_{i}}^{2}+{{y}_{i}}^{2} \bigr) \frac{{{\partial }^{2}}}{\partial {{t}^{2}}} \biggr). \end{aligned}$$
(2.4)

The family \(\{ {{X}_{1}},\ldots,{{X}_{n}},{{Y}_{1}},\ldots, {{Y}_{n}} \} \) satisfies Hörmander’s rank condition (see [16]) which implies that \({{\Delta }_{\mathbb{H}}}\) is hypoelliptic and the maximum principle holds for solutions to the equation involving \({{\Delta }_{\mathbb{H}}}\) (see [2]).

The integer \(Q=2n+2\) is called the homogeneous dimension of \({\mathbb{H}}^{n}\). Denote by \({{ \vert \xi \vert }_{\mathbb{H}}}\) the distance from ξ to the zero (see [13])

$$ {{ \vert \xi \vert }_{\mathbb{H}}}={{ \Biggl(\sum _{i=1}^{n}{ \bigl({{x}_{i}}^{2}+{{y}_{i}}^{2}} \bigr)^{2}}}+{{t}^{2}} {{ \Biggr)}^{ \frac{1}{4}}}. $$
(2.5)

Authors in [22] used the norm \(\vert (z,w) \vert ={{(\sum_{i=1}^{n}{({{x}_{i}}^{2}+{{y}_{i}}^{2}})^{2}}}+16{{t}^{2}}{{)}^{ \frac{1}{4}}}\) for \((x,y,t):=(z,w)\in {{\mathbb{H}}^{n}}\), which is equivalent to (2.5). The distance between two points of \({\mathbb{H}}^{n}\) is defined by

$$ {{d}_{\mathbb{H}}}(\xi ,\eta )={{ \bigl\vert {{\eta }^{-1}}\circ \xi \bigr\vert }_{\mathbb{H}}}, $$

where \(\eta ^{-1}\) denotes the inverse of η with respect to ∘, that is, \(\eta ^{-1}=-\eta \). The open ball of radius \(R>0\) centered at ξ is the set

$$ {{B}_{\mathbb{H}}}({{\xi }},R)= \bigl\{ \eta \in {\mathbb{H}^{n}} \mid {{d}_{\mathbb{H}}}(\eta ,{{\xi }})< R \bigr\} . $$

It is well known that \(\xi \to {{ \vert \xi \vert }_{\mathbb{H}}}\) is homogeneous of degree one with respect to \(\delta _{\kappa }\) and

$$ \bigl\vert {{B}_{\mathbb{H}}}({{\xi }},R) \bigr\vert = \bigl\vert {{B}_{ \mathbb{H}}}(0,R) \bigr\vert = \bigl\vert {{B}_{\mathbb{H}}}(0,1) \bigr\vert {{R}^{Q}}, $$

where \(\vert \cdot \vert \) denotes the Lebesgue measure.

A function u is called the cylindrical function if

$$ u(x,y,t)=u(r,t), $$

where \((x,y,t)\in {{\mathbb{H}}^{n}}\), \(r={{({{ \vert x \vert }^{2}}+{{ \vert y \vert }^{2}})}^{ \frac{1}{2}}}\).

Proposition 2.1

For\(u\in {{L}_{sp}}({{\mathbb{H}}^{n}})\cap C_{\mathrm{loc}}^{1,1}({{\mathbb{H}}^{n}})\), the operator in (1.1) is well defined.

Proof

For any \(\xi \in {{\mathbb{H}}^{n}}\),

$$\begin{aligned} & \mathit{PV} \int _{\mathbb{H}^{n}} \frac{ \vert u(\xi )-u(\eta ) \vert ^{p-2}(u(\xi )-u(\eta ))}{ \vert \eta ^{-1}\circ \xi \vert _{\mathbb{H}}^{Q+sp}}\,d \eta \\ &\quad =\lim_{\varepsilon \to 0} \biggl[ \int _{B_{\mathbb{H}}(\xi ,1) \setminus B_{\mathbb{H}}(\xi ,\varepsilon )} \bigl(\big\vert \big\langle -(\nabla _{\mathbb{H}}u,Tu),\eta ^{-1}\circ \xi \big\rangle +o\big( \big\vert \eta ^{-1}\circ \xi \big\vert _{\mathbb{H}}^{2}\big) \big\vert ^{p-2} \\ &\quad\quad{}\times \big( \big\langle -(\nabla _{\mathbb{H}}u,Tu ),\eta ^{-1}\circ \xi \big\rangle +o\big( \big\vert \eta ^{-1}\circ \xi \big\vert _{\mathbb{H}}^{2} \big) \big) \bigr) \bigl(\big\vert \eta ^{-1}\circ \xi \big\vert _{\mathbb{H}}^{Q+ sp}\bigr)^{-1} \,d \eta \\ & \quad \quad {} + \int _{\mathbb{H}^{n}\setminus B_{\mathbb{H}}(\xi ,1)} \frac{ \vert u(\xi )-u(\eta ) \vert ^{p-2}(u(\xi )-u(\eta ))}{ \vert \eta ^{-1}\circ \xi \vert _{\mathbb{H}}^{Q+sp}}\,d \eta \biggr] \\ &\quad \leq C\lim_{\varepsilon \to 0} \biggl[ \int _{{{B}_{\mathbb{H}}}(\xi ,1) \setminus {{B}_{\mathbb{H}}}(\xi ,\varepsilon )} \frac{o(1)}{ \vert \eta ^{-1}\circ \xi \vert _{\mathbb{H}}^{Q+sp}}\,d\eta + \int _{{{B}_{ \mathbb{H}}}(\xi ,1)\setminus {{B}_{\mathbb{H}}}(\xi ,\varepsilon )} \frac{o(1)}{ \vert \eta ^{-1}\circ \xi \vert _{\mathbb{H}}^{Q+sp}}\,d\eta \\ & \quad \quad {} + \int _{{{\mathbb{H}}^{n}}\setminus {{B}_{\mathbb{H}}}( \xi ,1)}\frac{{{u}^{p-1}}(\xi )}{ \vert \eta ^{-1}\circ \xi \vert _{\mathbb{H}}^{Q+sp}}\,d\eta + \int _{{{ \mathbb{H}}^{n}}\setminus {{B}_{\mathbb{H}}}(\xi ,1)} \frac{{{u}^{p-1}}(\eta )}{ \vert \eta ^{-1}\circ \xi \vert _{\mathbb{H}}^{Q+sp}}\,d \eta \biggr] \\ &\quad :=C\lim_{\varepsilon \to 0}({{I}_{1}}+{{I}_{2}}+{{I}_{3}}+{{I}_{4}}), \end{aligned}$$

where ε is sufficiently small. Noting that \(u\in C_{\mathrm{loc}}^{1,1}({{\mathbb{H}}^{n}})\), \(Q+sp-p< Q\), and \(Q+sp-2p+2< Q\), we know that \(I_{1}\) and \(I_{2}\) are finite; \(I_{3}\) is clearly convergent when \({{ \vert \xi \vert }_{\mathbb{H}}}\to \infty \); and \(I_{4}\) is finite from \(u\in {{L}_{sp}}({{\mathbb{H}}^{n}})\). Hence, (1.1) is well defined. □

3 Maximum principles

In this section, we prove three maximum principles which will be used in the process of moving planes. These maximum principles are on a bounded domain in \({{\mathbb{H}}^{n}}\), on a bounded domain in the left domain of some hyperplane, and on a narrow region.

Lemma 3.1

LetΩbe a bounded domain in\({{\mathbb{H}}^{n}}\). Assume\(u\in {{L}_{sp}}({{\mathbb{H}}^{n}})\cap C_{\mathrm{loc}}^{1,1}({{\mathbb{H}}^{n}})\)is lower semicontinuous onΩ̄and satisfies

$$ \textstyle\begin{cases} (-{{\Delta }_{\mathbb{H}}})_{p}^{s}u(\xi )\ge 0, \quad \xi \in \varOmega , \\ u(\xi )\ge 0, \quad \xi \in \mathbb{H}^{n}\setminus \varOmega , \end{cases} $$
(3.1)

then

$$ u(\xi )\ge 0,\quad \xi \in \varOmega . $$
(3.2)

Furthermore, if\(u=0\)at some point inΩ, then

$$ u(\xi )=0 \quad \textit{almost everywhere in } {{\mathbb{H}}^{n}}. $$

These conclusions also hold on the unbounded regionΩif we further assume that

$$ \mathop{\underline{\lim }}_{{{ \vert \xi \vert }_{\mathbb{H}}}\to \infty } u(\xi )\ge 0. $$

Proof

Suppose that (3.2) is not true, then by the lower semicontinuity of u on Ω̄ there exists \({{\xi }^{0}}\in \bar{\varOmega }\) such that

$$ u \bigl({{\xi }^{0}} \bigr)=\min_{{\bar{\varOmega }}} u< 0. $$

From (3.1), we know that \({{\xi }^{0}}\) is a point in Ω, and

$$\begin{aligned} (-{{\Delta }_{\mathbb{H}}})_{p}^{s}u \bigl({{\xi }^{0}} \bigr)&={{C}_{Q,s,p}}\mathit{PV} \int _{{{\mathbb{H}}^{n}}}{ \frac{{{ \vert u({{\xi }^{0}})-u(\eta ) \vert }^{p-2}}(u({{\xi }^{0}})-u(\eta ))}{{{ \vert {{\eta }^{-1}}\circ {{\xi }^{0}} \vert }_{\mathbb{H}}}^{Q+sp}}}\,d \eta \\ &\le {{C}_{Q,s,p}} \int _{{{\mathbb{H}}^{n}}\setminus \varOmega }{ \frac{{{ \vert u({{\xi }^{0}})-u(\eta ) \vert }^{p-2}}(u({{\xi }^{0}})-u(\eta ))}{{{ \vert {{\eta }^{-1}}\circ {{\xi }^{0}} \vert }_{\mathbb{H}}}^{Q+sp}}}\,d \eta \\ &< 0, \end{aligned}$$

which contradicts (3.1). This implies (3.2).

If there exists some point \({{\xi }^{0}}\in \varOmega \) such that \(u({{\xi }^{0}})=0\), then

$$ 0\le (-{{\Delta }_{\mathbb{H}}})_{p}^{s}u \bigl({{\xi }^{0}} \bigr)={{C}_{Q,s,p}} \int _{{{\mathbb{H}}^{n}}}{ \frac{{{ \vert u(\eta ) \vert }^{p-2}}(-u(\eta ))}{{{ \vert {{\eta }^{-1}}\circ {{\xi }^{0}} \vert }_{\mathbb{H}}}^{Q+sp}}}\,d \eta . $$

Using \(u(\xi )\ge 0\), we have \(u(\xi )=0\) almost everywhere in \({{\mathbb{H}}^{n}}\).

For an unbounded region Ω, the condition \(\underline{\lim }_{{{ \vert \xi \vert }_{\mathbb{H}}}\to \infty } u(\xi )\ge 0\) implies that the negative minimum \({{\xi }^{0}}\) of u cannot be reached at infinity. Then the condition of lower semicontinuity ensures that the proof can go on as above. The proof is ended. □

Let \(T_{\lambda }\) be a hyperplane in \({{\mathbb{H}}^{n}}\) defined by

$$ T_{\lambda }= \bigl\{ \xi \in {{\mathbb{H}}^{n}}\mid t=\lambda , \lambda \in \mathbb{R} \bigr\} . $$

Denote by \(\tilde{\xi }=(y,x,2\lambda -t)\) the H-reflection of \(\xi =(x,y,t)\) about the plane \(T_{\lambda }\) and by

$$ \varSigma _{\lambda }= \bigl\{ \xi \in {{\mathbb{H}}^{n}}\mid t< \lambda \bigr\} $$

the region in the left of the plane \(T_{\lambda }\). Letting

$$ {{u}_{\lambda }}(\xi )={{u}_{\lambda }} \bigl( \bigl\vert (x,y) \bigr\vert ,t \bigr):=u \bigl( \bigl\vert (x,y) \bigr\vert ,2\lambda -t \bigr) $$

and using the H-refection (see [1]), we have

$$ {{u}_{\lambda }}(\xi )=u(y,x,2\lambda -t)=u \bigl({{\xi }^{\lambda }} \bigr). $$

Set

$$ {{w}_{\lambda }}(\xi )={{u}_{\lambda }}(\xi )-u(\xi ). $$

Lemma 3.2

LetΩbe a bounded domain in\({{\varSigma }_{\lambda }}\). Assume that the cylindrical function\(u\in {{L}_{sp}}({{\mathbb{H}}^{n}})\cap C_{\mathrm{loc}}^{1,1}({{\mathbb{H}}^{n}})\)is lower semicontinuous onΩ̄and satisfies

$$ \textstyle\begin{cases} (-{{\Delta }_{\mathbb{H}}})_{p}^{s}{{u}_{\lambda }}(\xi )-(-{{ \Delta }_{\mathbb{H}}})_{p}^{s}u(\xi )\geq 0, \quad \xi \in \varOmega , \\ {{w}_{\lambda }}(\xi )\ge 0, \quad \xi \in {{\varSigma }_{\lambda }} \setminus \varOmega , \\ w_{\lambda }(\xi ^{\lambda })=-w_{\lambda }(\xi ), \quad \xi \in {{\varSigma }_{ \lambda }}, \end{cases} $$
(3.3)

then

$$ w_{\lambda }(\xi )\ge 0,\quad \xi \in \varOmega . $$
(3.4)

Furthermore, if\(w_{\lambda }=0\)at some point inΩ, then

$$ w_{\lambda }(\xi )=0 \quad \textit{almost everywhere in } {{ \mathbb{H}}^{n}}. $$

These conclusions also hold for the unbounded regionΩif we further assume that

$$ \mathop{\underline{\lim }}_{{{ \vert \xi \vert }_{\mathbb{H}}}\to \infty } w_{\lambda }(\xi )\ge 0. $$

Proof

Suppose that (3.4) is incorrect. By the lower semicontinuity of \(w_{\lambda }\) on Ω̄, there exists \({{\xi }^{0}}\in \bar{\varOmega }\) such that

$$ w_{\lambda } \bigl({{\xi }^{0}} \bigr)=\min_{{\bar{\varOmega }}} w_{\lambda }< 0. $$

For simplicity, we denote

$$ G(a)={{ \vert a \vert }^{p-2}}a, \quad a\geq 0. $$

Note that \(G(a)\) is increasing and \({G}'(a)=(p-1){{ \vert a \vert }^{p-2}}\ge 0\). A direct calculation gives

$$\begin{aligned}& (-{{\Delta }_{\mathbb{H}}})_{p}^{s}{{u}_{\lambda }} \bigl({{\xi }^{0}} \bigr)-(-{{ \Delta }_{\mathbb{H}}})_{p}^{s}u \bigl({{\xi }^{0}} \bigr) \\& \quad = {{C}_{Q,s,p}}\mathit{PV} \int _{{{\mathbb{H}}^{n}}}{ \frac{G({{u}_{\lambda }}({{\xi }^{0}})-{{u}_{\lambda }}(\eta ))-G(u({{\xi }^{0}})-u(\eta ))}{{{ \vert {{\eta }^{-1}}\circ {{\xi }^{0}} \vert }_{\mathbb{H}}}^{Q+sp}}}\,d \eta \\& \quad = {{C}_{Q,s,p}}\mathit{PV} \int _{{{\varSigma }_{\lambda }}}{ \frac{G({{u}_{\lambda }}({{\xi }^{0}})-{{u}_{\lambda }}(\eta ))-G(u({{\xi }^{0}})-u(\eta ))}{{{ \vert {{\eta }^{-1}}\circ {{\xi }^{0}} \vert }_{\mathbb{H}}}^{Q+sp}}}\,d \eta \\& \quad\quad{} +{{C}_{Q,s,p}}\mathit{PV} \int _{{{\varSigma }_{\lambda }}}{ \frac{G({{u}_{\lambda }}({{\xi }^{0}})-u(\eta ))-G(u({{\xi }^{0}})-{{u}_{\lambda }}(\eta ))}{{{ \vert {{({{\eta }^{\lambda }})}^{-1}}\circ {{\xi }^{0}} \vert }_{\mathbb{H}}}^{Q+sp}}}\,d \eta \\& \quad = {{C}_{Q,s,p}}\mathit{PV} \int _{{{\varSigma }_{\lambda }}} \biggl( \frac{1}{{{ \vert {{\eta }^{-1}}\circ {{\xi }^{0}} \vert }_{\mathbb{H}}}^{Q+sp}}- \frac{1}{{{ \vert {{({{\eta }^{\lambda }})}^{-1}}\circ {{\xi }^{0}} \vert }_{\mathbb{H}}}^{Q+sp}} \biggr) \\& \quad\quad{}\times \bigl(G \bigl({{u}_{\lambda }} \bigl({{\xi }^{0}} \bigr)-{{u}_{\lambda }}(\eta ) \bigr)-G \bigl(u \bigl({{ \xi }^{0}} \bigr)-u(\eta ) \bigr) \bigr)\,d\eta \\& \quad\quad{} +{{C}_{Q,s,p}}\mathit{PV} \int _{{{\varSigma }_{\lambda }}} \bigl(G\bigl({{u}_{\lambda }}\bigl({{\xi }^{0}}\bigr)-{{u}_{\lambda }}(\eta )\bigr) -G\bigl(u\bigl({{\xi }^{0}}\bigr)-u_{\lambda }(\eta )\bigr)+ G\bigl({{u}_{\lambda }}\bigl({{\xi }^{0}}\bigr)-{{u}}(\eta )\bigr) \\& \quad\quad{}-G\bigl(u\bigl({{\xi }^{0}}\bigr)-u(\eta )\bigr) \bigr) \bigl( {{ \big\vert {{\bigl({{\eta }^{\lambda }}\bigr)}^{-1}}\circ {{\xi }^{0}} \big\vert }_{\mathbb{H}}}^{Q+ sp}\bigr)^{-1} \,d \eta \\& \quad := {{C}_{Q,s,p}}({{J}_{1}}+{{J}_{2}}). \end{aligned}$$
(3.5)

For \(J_{1}\), we have for any \({{\xi }^{0}},\eta \in {{\varSigma }_{\lambda }}\),

$$ \frac{1}{{{ \vert {{\eta }^{-1}}\circ {{\xi }^{0}} \vert }_{\mathbb{H}}}^{Q+sp}}- \frac{1}{{{ \vert {{({{\eta }^{\lambda }})}^{-1}}\circ {{\xi }^{0}} \vert }_{\mathbb{H}}}^{Q+sp}}>0. $$

By the monotonicity of G and the fact that

$$ \bigl({{u}_{\lambda }} \bigl({{\xi }^{0}} \bigr)-{{u}_{\lambda }}( \eta ) \bigr)- \bigl(u \bigl({{\xi }^{0}} \bigr)-u( \eta ) \bigr)={{w}_{\lambda }} \bigl({{\xi }^{0}} \bigr)-{{w}_{\lambda }}( \eta ) $$

is nonpositive but not identity to 0, we deduce that

$$ G \bigl({{u}_{\lambda }} \bigl({{\xi }^{0}} \bigr)-{{u}_{\lambda }}( \eta ) \bigr)-G \bigl(u \bigl({{\xi }^{0}} \bigr)-u( \eta ) \bigr) $$

is also nonpositive but not identity to 0. So we have

$$ J_{1}< 0. $$
(3.6)

For \(J_{2}\), by the mean value theorem,

$$\begin{aligned} {{J}_{2}}&= \int _{{{\varSigma }_{\lambda }}}{ \frac{G({{u}_{\lambda }}({{\xi }^{0}})-{{u}_{\lambda }}(\eta ))-G(u({{\xi }^{0}})-{{u}_{\lambda }}(\eta ))+G({{u}_{\lambda }}({{\xi }^{0}})-u(\eta ))-G(u({{\xi }^{0}})-u(\eta ))}{{{ \vert {{({{\eta }^{\lambda }})}^{-1}}\circ {{\xi }^{0}} \vert }_{\mathbb{H}}}^{Q+ sp}}}\,d \eta \\ &=w_{\lambda } \bigl(\xi ^{0} \bigr) \int _{{{\varSigma }_{\lambda }}}{ \frac{{G}'(g(\eta ))+{G}'(h(\eta ))}{{{ \vert {{({{\eta }^{\lambda }})}^{-1}}\circ {{\xi }^{0}} \vert }_{\mathbb{H}}}^{Q+sp}}}\,d \eta \\ &\le 0. \end{aligned}$$
(3.7)

In fact, if \(u_{\lambda }(\eta )\ge u(\eta )\), then we have \(w_{\lambda }(\eta )\ge 0\), i.e., (3.4) holds. If \(u_{\lambda }(\eta )> u(\eta )\), we know G is strictly increasing, then \({G}'(g(\eta ))\ge 0\) and \({G}'(h(\eta ))\ge 0\). Hence, we have (3.7).

Putting (3.6) and (3.7) into (3.5) implies

$$ (-{{\Delta }_{\mathbb{H}}})_{p}^{s}{{u}_{\lambda }} \bigl({{\xi }^{0}} \bigr)-(-{{ \Delta }_{\mathbb{H}}})_{p}^{s}u \bigl({{\xi }^{0}} \bigr)< 0. $$

This contradicts (3.3) and we obtain (3.4).

If there exists some point \({{\xi }^{0}}\in \varOmega \) such that \({{w}_{\lambda }}({{\xi }^{0}})=0\), then (3.5) holds and \({{J}_{2}}\ge 0\). Hence from the first inequality in (3.3) we have \({{J}_{1}}\ge 0\), and by the monotonicity of G,

$$ G \bigl({{u}_{\lambda }} \bigl({{\xi }^{0}} \bigr)-{{u}_{\lambda }}( \eta ) \bigr)-G \bigl(u \bigl({{\xi }^{0}} \bigr)-u( \eta ) \bigr)\ge 0. $$

We have, for almost all \(\eta \in {{\varSigma }_{\lambda }}\),

$$ \bigl({{u}_{\lambda }} \bigl({{\xi }^{0}} \bigr)-{{u}_{\lambda }}( \eta ) \bigr)- \bigl(u \bigl({{\xi }^{0}} \bigr)-u( \eta ) \bigr)={{w}_{\lambda }} \bigl({{\xi }^{0}} \bigr)-{{w}_{\lambda }}( \eta )=-{{w}_{ \lambda }}(\eta )\ge 0. $$

Using (3.4), we have

$$ {{w}_{\lambda }}(\xi )=0\quad \text{almost everywhere in } {{\varSigma }_{ \lambda }}. $$

From the antisymmetry of \({{w}_{\lambda }}\),

$$ {{w}_{\lambda }}(\xi )=0\quad \text{almost everywhere in } {{ \mathbb{H}}^{n}}. $$

 □

Lemma 3.3

LetΩbe a bounded narrow domain in\({{\varSigma }_{\lambda }}\)and locate in\(\{\xi\mid \lambda -l< t<\lambda \}\)for small l. Assume that the cylindrical function\(u\in {{L}_{sp}}({{\mathbb{H}}^{n}})\cap C_{\mathrm{loc}}^{1,1}({{\mathbb{H}}^{n}})\)is lower semicontinuous onΩ̄. If\(c(x)\)is bounded from below inΩandusatisfies

$$ \textstyle\begin{cases} (-{{\Delta }_{\mathbb{H}}})_{p}^{s}{{u}_{\lambda }}(\xi )-(-{{ \Delta }_{\mathbb{H}}})_{p}^{s}u(\xi )+c(\xi ){{w}_{\lambda }}(\xi ) \ge 0, \quad \xi \in \varOmega , \\ {{w}_{\lambda }}(\xi )\ge 0, \quad \xi \in {{\varSigma }_{\lambda }} \setminus \varOmega , \\ {{w}_{\lambda }}({{\xi }^{\lambda }})=-{{w}_{\lambda }}(\xi ),\quad \xi \in {{\varSigma }_{\lambda }}, \end{cases} $$
(3.8)

then

$$ w_{\lambda }(\xi )\ge 0,\quad \xi \in \varOmega . $$
(3.9)

Furthermore, if\(w_{\lambda }=0\)at some point inΩ, then

$$ w_{\lambda }(\xi )=0 \quad \textit{almost everywhere in } {{ \mathbb{H}}^{n}}. $$

These conclusions also hold for the unbounded regionΩif we further assume that

$$ \mathop{\underline{\lim }}_{{{ \vert \xi \vert }_{\mathbb{H}}}\to \infty } w_{\lambda }(\xi )\ge 0. $$

Proof

By the proof of Lemma 3.2, we have

$$\begin{aligned} & (-{{\Delta }_{\mathbb{H}}})_{p}^{s}{{u}_{\lambda }} \bigl({{\xi }^{0}} \bigr)-(-{{ \Delta }_{\mathbb{H}}})_{p}^{s}u \bigl({{\xi }^{0}} \bigr) \\ &\quad ={{C}_{Q,s,p}}\mathit{PV} \int _{{{\varSigma }_{\lambda }}} \biggl( \frac{1}{{{ \vert {{\eta }^{-1}}\circ {{\xi }^{0}} \vert }_{\mathbb{H}}}^{Q+sp}}- \frac{1}{{{ \vert {{({{\eta }^{\lambda }})}^{-1}}\circ {{\xi }^{0}} \vert }_{\mathbb{H}}}^{Q+sp}} \biggr) \\ &\quad\quad{}\times \bigl(G \bigl({{u}_{\lambda }} \bigl({{\xi }^{0}} \bigr)-{{u}_{\lambda }}(\eta ) \bigr)-G \bigl(u \bigl({{ \xi }^{0}} \bigr)-u(\eta ) \bigr) \bigr)\,d\eta \\ &\quad\quad{} +{{C}_{Q,s,p}}\mathit{PV} \int _{{{\varSigma }_{\lambda }}} \bigl(G\bigl({{u}_{\lambda }}\bigl({{\xi }^{0}}\bigr)-{{u}_{\lambda }}(\eta )\bigr) -G\bigl(u\bigl({{\xi }^{0}}\bigr)-u(\eta )\bigr)+ G\bigl({{u}_{\lambda }}\bigl({{\xi }^{0}}\bigr)-{{u}}(\eta )\bigr) \\ &\quad\quad{} -G\bigl(u\bigl({{\xi }^{0}}\bigr)-u_{\lambda }(\eta )\bigr) \bigr) \bigl({{ \big\vert {{\bigl({{\eta }^{\lambda }}\bigr)}^{-1}}\circ {{\xi }^{0}} \big\vert }_{\mathbb{H}}}^{Q+ sp} \bigr)^{-1} \,d \eta \\ &\quad :={{C}_{Q,s,p}}({{I}_{1}}+{{I}_{2}}). \end{aligned}$$
(3.10)

Obviously,

$$ {{I}_{2}}\le 0. $$
(3.11)

Similar to (3.6), we know

$$ I_{1}< 0. $$

Denote

$$ {{\delta }_{{{\xi }^{0}}}}=\operatorname{dist} \bigl({{\xi }^{0}}, \partial {{\varSigma }_{ \lambda }} \bigr)= \bigl\vert \lambda -{{t}^{0}} \bigr\vert . $$

Combining (3.10), (3.11), and \(I_{1}<0\), we have

$$ \frac{(-{{\Delta }_{\mathbb{H}}})_{p}^{s}{{u}_{\lambda }}({{\xi }^{0}})-(-{{\Delta }_{\mathbb{H}}})_{p}^{s}u({{\xi }^{0}})}{{{\delta }_{{{\xi }^{0}}}}}< 0. $$
(3.12)

Noting that \({{\xi }^{0}}\) is a negative minimum of \({{w}_{\lambda }}\), we infer \(\nabla {{w}_{{{\lambda }_{0}}}}({{\xi }^{0}})=0\), and so

$$ \frac{\partial {{w}_{\lambda }}}{\partial t} \bigl({{\xi }^{0}} \bigr)= \lim _{{{\delta }_{k}}\to 0} \frac{{{w}_{\lambda }}({{\xi }^{0}})}{{{\delta }_{{{\xi }^{0}}}}}=0, $$

i.e.,

$$ \frac{c({{\xi }^{0}}){{w}_{\lambda }}({{\xi }^{0}})}{{{\delta }_{{{\xi }^{0}}}}} \le o(1). $$
(3.13)

Now (3.12) and (3.13) contradict (3.8), and then (3.9) is proved. □

4 Proof of the main results

Following the idea in [6], we first use Lemma 3.1, Lemma 3.2, and Lemma 3.3 to prove Theorem 1.1.

Proof of Theorem 1.1

First we check that for λ sufficiently negative it holds

$$ {{w}_{\lambda }}(\xi )\ge 0,\quad \xi \in {{\varSigma }_{\lambda }}. $$
(4.1)

Indeed, suppose that (4.1) is violated, then by (1.8) there exists a point \(\xi ^{0} \in {{\varSigma }_{\lambda }}\) such that

$$ {{w}_{\lambda }} \bigl({{\xi }^{0}} \bigr)= \min _{{{\varSigma }_{\lambda }}} {{w}_{\lambda }}< 0, $$

i.e., \({{u}_{\lambda }}({{\xi }^{0}})\le {{\varsigma }_{\lambda }}({{\xi }^{0}}) \le u({{\xi }^{0}})\). Note by (1.2) that

$$ (-{{\Delta }_{\mathbb{H}}})_{p}^{s}{{u}_{\lambda }}( \xi )-(-{{\Delta }_{ \mathbb{H}}})_{p}^{s}u(\xi )={f}' \bigl({{\varsigma }_{\lambda }}(\xi ) \bigr){{w}_{ \lambda }}( \xi ), $$
(4.2)

where \({{\varsigma }_{\lambda }}(\xi )\) between \({{u}_{\lambda }}(\xi )\) and \(u(\xi )\). For sufficiently negative λ, \(u({{\xi }^{0}})\) is small by (1.8), hence so is \({{\varsigma }_{\lambda }}({{\xi }^{0}})\). Due to the condition of \({f}'\), we have \({f}'({{\varsigma }_{\lambda }}({{\xi }^{0}}))\le 0\). From (4.2),

$$ (-{{\Delta }_{\mathbb{H}}})_{p}^{s}{{u}_{\lambda }} \bigl({{\xi }^{0}} \bigr)-(-{{ \Delta }_{\mathbb{H}}})_{p}^{s}u \bigl({{\xi }^{0}} \bigr)\ge 0. $$
(4.3)

On the other hand, it follows by the proof of Lemma 3.2 that

$$ (-{{\Delta }_{\mathbb{H}}})_{p}^{s}{{u}_{\lambda }} \bigl({{\xi }^{0}} \bigr)-(-{{ \Delta }_{\mathbb{H}}})_{p}^{s}u \bigl({{\xi }^{0}} \bigr)< 0. $$
(4.4)

This contradicts (4.3), and hence (4.1) is proved.

The above result provides the starting point of moving planes. Let us move the plane \(T_{\lambda }\) to the right as long as (4.1) holds to its limiting position

$$ {{\lambda }_{0}}=\sup \bigl\{ \lambda\mid {{w}_{\mu }}(\xi ) \ge 0, \forall \xi \in {{\varSigma }_{\mu }},\mu \le \lambda \bigr\} . $$

We will show that

$$ {{\lambda }_{0}}=0 $$
(4.5)

i.e.,

$$ {{w}_{{{\lambda }_{0}}}}(\xi )=0,\quad \xi \in {{\varSigma }_{{{\lambda }_{0}}}}. $$
(4.6)

In fact, suppose that (4.6) is false, we have by Lemma 3.2 that

$$ {{w}_{{{\lambda }_{0}}}}(\xi )>0,\quad \xi \in {{\varSigma }_{{{\lambda }_{0}}}}. $$
(4.7)

From the definition of \({{\lambda }_{0}}\), there exist a sequence \({{\lambda }_{k}}\to {{\lambda }_{0}}\) and a point \({{\xi }^{k}}\in {{\varSigma }_{{{\lambda }_{k}}}}\) such that

$$ {{w}_{{{\lambda }_{k}}}} \bigl({{\xi }^{k}} \bigr)= \min _{{{\varSigma }_{{{\lambda }_{k}}}}} {{w}_{{{ \lambda }_{k}}}}< 0, \quad\quad \nabla {{w}_{{{\lambda }_{k}}}} \bigl({{\xi }^{k}} \bigr)=0. $$
(4.8)

Note that

$$ (-{{\Delta }_{\mathbb{H}}})_{p}^{s}{{u}_{{{\lambda }_{k}}}} \bigl({{\xi }^{k}} \bigr)-(-{{ \Delta }_{\mathbb{H}}})_{p}^{s}u \bigl({{\xi }^{k}} \bigr)={f}' \bigl({{\varsigma }_{{{ \lambda }_{k}}}} \bigl({{\xi }^{k}} \bigr) \bigr){{w}_{{{\lambda }_{k}}}} \bigl({{\xi }^{k}} \bigr). $$
(4.9)

If \({{ \vert {{\xi }^{k}} \vert }_{\mathbb{H}}}\) is sufficiently large, then \(u({{\xi }^{k}})\) is small and so \({{\varsigma }_{{{\lambda }_{k}}}}({{\xi }^{k}})\) is also small, this implies \({f}'({{\varsigma }_{{{\lambda }_{k}}}}({{\xi }^{k}}))\le 0\) (because \({f}'(a)\le 0\) for the sufficiently small a). It follows

$$ (-{{\Delta }_{\mathbb{H}}})_{p}^{s}{{u}_{{{\lambda }_{k}}}} \bigl({{\xi }^{k}} \bigr)-(-{{ \Delta }_{\mathbb{H}}})_{p}^{s}u \bigl({{\xi }^{k}} \bigr)\ge 0. $$

But this contradicts the fact that \({{\xi }^{k}}\) is a negative minimum of \({{w}_{{{\lambda }_{k}}}}\) (see Lemma 3.2). Hence, \(\{ {{\xi }^{k}} \}\) is bounded, i.e., the sequence \(\{ {{\xi }^{k}} \}\) is bounded.

It follows that the subsequence of \(\{ {{\xi }^{k}} \} \) converges to some point \({{\xi }^{0}}\). Then (4.8) means that, for \({{\xi }^{0}}\in \partial {{\varSigma }_{{{\lambda }_{0}}}}\),

$$ {{w}_{{{\lambda }_{0}}}} \bigl({{\xi }^{0}} \bigr)\le 0, \quad \quad \nabla {{w}_{{{\lambda }_{0}}}} \bigl({{\xi}^{0}} \bigr)=0. $$

Particularly,

$$ \frac{\partial {{w}_{{{\lambda }_{0}}}}}{\partial t} \bigl({{\xi }^{0}} \bigr)= \lim _{{{\delta }_{k}}\to 0} \frac{{{w}_{{{\lambda }_{k}}}}({{\xi }^{k}})}{{{\delta }_{k}}}=0. $$

Applying (4.9), we have

$$ \lim_{{{\delta }_{k}}\to 0} \frac{1}{{{\delta }_{k}}} \bigl((-{{\Delta }_{\mathbb{H}}})_{p}^{s}{{u}_{{{ \lambda }_{k}}}} \bigl({{\xi }^{k}} \bigr)-(-{{\Delta }_{\mathbb{H}}})_{p}^{s}u \bigl({{ \xi }^{k}} \bigr) \bigr)=\lim_{{{\delta }_{k}}\to 0} \frac{1}{{{\delta }_{k}}}{f}' \bigl({{\varsigma }_{{{\lambda }_{k}}}} \bigl({{ \xi }^{k}} \bigr) \bigr){{w}_{{{\lambda }_{k}}}} \bigl({{\xi }^{k}} \bigr)=0, $$

which is a contradiction with Lemma 3.3. Therefore,

$$ {{w}_{{{\lambda }_{0}}}}(\xi )\ge 0,\quad \xi \in {{\varSigma }_{{{\lambda }_{0}}}}. $$
(4.10)

Similarly, we can move the plane from +∞ to the left to get

$$ {{w}_{{{\lambda }_{0}}}}(\xi )\le 0,\quad \xi \in {{\varSigma }_{{{\lambda }_{0}}}}. $$
(4.11)

Then (4.6) follows by combining (4.10) and (4.11). Finally, we see that u must be symmetric and monotone with respect to t about some point. □

Next, we give the proof of Theorem 1.2.

Proof of Theorem 1.2

By condition (1.8) and \(f(0)=0\), we claim

$$ u(\xi )>0 \quad \text{or} \quad u(\xi )\equiv 0 \quad \text{for any } \xi \in \mathbb{H}_{+}^{n}. $$

In fact, suppose that the conclusion \(u(\xi )>0\) is not correct, we will verify \(u(\xi )\equiv 0\). The lower semicontinuity of u on \(\mathbb{H}_{+}^{n}\) implies that there exists \({{\xi }^{0}}\in \bar{\mathbb{H}}_{+}^{n}\) such that

$$ u \bigl({{\xi }^{0}} \bigr)=\min_{\bar{\mathbb{H}}_{+}^{n}} u=0, $$

and then

$$\begin{aligned} (-{{\Delta }_{\mathbb{H}}})_{p}^{s}u \bigl({{\xi }^{0}} \bigr)&={{C}_{Q,s,p}}\mathit{PV} \int _{{{\mathbb{H}}^{n}}}{ \frac{{{ \vert u({{\xi }^{0}})-u(\eta ) \vert }^{p-2}}(u({{\xi }^{0}})-u(\eta ))}{{{ \vert {{\eta }^{-1}}\circ {{\xi }^{0}} \vert }_{\mathbb{H}}}^{Q+sp}}}\,d \eta \\ &={{C}_{Q,s,p}}\mathit{PV} \int _{\mathbb{H}_{+}^{n}}{ \frac{{{ \vert u({{\xi }^{0}})-u(\eta ) \vert }^{p-2}}(u({{\xi }^{0}})-u(\eta ))}{{{ \vert {{\eta }^{-1}}\circ {{\xi }^{0}} \vert }_{\mathbb{H}}}^{Q+sp}}}\,d \eta \\ &={{C}_{Q,s,p}}\mathit{PV} \int _{\mathbb{H}_{+}^{n}}{ \frac{-u(\eta ){{ \vert u(\eta ) \vert }^{p-2}}}{{{ \vert {{\eta }^{-1}}\circ {{\xi }^{0}} \vert }_{\mathbb{H}}}^{Q+sp}}}\,d \eta \\ &=f \bigl(u \bigl({{\xi }^{0}} \bigr) \bigr)=0. \end{aligned}$$

Hence \(\int _{\mathbb{H}_{+}^{n}}{ \frac{-u(\eta ){{ \vert u(\eta ) \vert }^{p-2}}}{{{ \vert {{\eta }^{-1}}\circ {{\xi }^{0}} \vert }_{\mathbb{H}}}^{Q+sp}}}\,d \eta =0\), and then \(u(\xi )\equiv 0\), \(\xi \in \mathbb{H}_{+}^{n}\).

In the sequel, we only need to treat the case \(u>0\) on \(\mathbb{H}_{+}^{n}\). Let us employ the method of moving planes to u along the t direction and denote

$$ T_{\lambda }^{+}= \bigl\{ \xi \in {{\mathbb{H}}_{+}^{n}} \mid t=\lambda ,\lambda \in {{\mathbb{R}}^{+}} \bigr\} $$

and

$$ \varSigma _{\lambda }^{+}= \bigl\{ \xi \in {{\mathbb{H}}_{+}^{n}} \mid 0< t< \lambda \bigr\} . $$

The H-reflection of \(\xi =(x,y,t)\) about \(T_{\lambda }^{+}\) is \({{\xi }^{\lambda }}=(y,x,2\lambda -t)\), and let

$$ {{w}_{\lambda }}(\xi )={{u}_{\lambda }}(\xi )-u(\xi ). $$

If \(\lambda >0\) is sufficiently small, we deduce from Lemma 3.3 with \(\varOmega =\varSigma _{\lambda }^{+}\) and \({{\varSigma }_{\lambda }}=\varSigma _{\lambda }^{+}\cup ({{\mathbb{H}}^{n}} \setminus \mathbb{H}_{+}^{n})\) that on the narrow region \(\varSigma _{\lambda }^{+}\),

$$ {{w}_{\lambda }}(\xi )\ge 0. $$
(4.12)

This provides the starting point of moving planes. Now we will explain that the plane \(T_{\lambda }^{+}\) can be moved to the infinity so that (4.12) holds. Let

$$ {{\lambda }_{0}}=\sup \bigl\{ \lambda >0\mid {{w}_{\mu }}(\xi )\ge 0, \forall \xi \in \varSigma _{\lambda }^{+},\mu \le \lambda \bigr\} , $$

and we will prove

$$ {{\lambda }_{0}}=\infty . $$
(4.13)

In fact, if \({{\lambda }_{0}}<\infty \), then we claim that \(T_{\lambda }^{+}\) can be moved further to the right, that is, there exists \(\sigma >0\) such that, for any \(\lambda \in ({{\lambda }_{0}},{{\lambda }_{0}}+\sigma )\),

$$ {{w}_{\lambda }}(\xi )\ge 0,\quad \xi \in \varSigma _{\lambda }^{+}. $$
(4.14)

This will contradict the definition of \({{\lambda }_{0}}\), and hence (4.13) holds.

At present, we prove (4.14). If \({{ \vert \xi \vert }_{\mathbb{H}}}\) is sufficiently large, then (4.14) is true by using the similar proof to (4.1) in Theorem 1.1. This implies that there exists some \({{R}_{0}}>0\) such that (4.14) holds true on \(\mathbb{H}_{+}^{n}\setminus {{B}_{\mathbb{H}}}(0,{{R}_{0}})\). Next we point out that (4.14) is also true on \({{B}_{\mathbb{H}}}(0,{{R}_{0}})\). Noting \({{\lambda }_{0}}<\infty \) and using Lemma 3.2, we find that on \(\xi \in \varSigma _{\lambda _{0} }^{+}\cap {{B}_{\mathbb{H}}}(0,{{R}_{0}})\),

$$ {{w}_{\lambda _{0} }}(\xi )>0 $$
(4.15)

or

$$ {{w}_{{{\lambda }_{0}}}}(\xi )\equiv 0. $$

In the case \({{w}_{{{\lambda }_{0}}}}(\xi )\equiv 0\), we observe by the boundary conditions of u that \(u(\xi )\equiv 0\). On the other hand, (4.15) implies that there exists small \(\delta >0\) such that

$$ {{w}_{{{\lambda }_{0}}}}(\xi )\ge c>0,\quad \xi \in \overline{ \varSigma _{{{\lambda }_{0}}-\delta }^{+}\cap {{B}_{\mathbb{H}}}(0,{{R}_{0}})}. $$
(4.16)

Since \({{w}_{\lambda }}\) relies continuously on λ, there exists \(\sigma >\varepsilon >0\) such that

$$ {{w}_{{{\lambda }_{0}}+\varepsilon }}(\xi )\ge 0,\quad \xi \in \overline{\varSigma _{{{\lambda }_{0}}-\delta }^{+}\cap {{B}_{\mathbb{H}}}(0,{{R}_{0}})}. $$
(4.17)

Since \((\varSigma _{{{\lambda }_{0}}+\varepsilon }^{+}\setminus \varSigma _{{{ \lambda }_{0}}-\delta }^{+})\cap {{B}_{\mathbb{H}}}(0,{{R}_{0}})\) is a narrow region, we have by Lemma 3.3

$$ {{w}_{{{\lambda }_{0}}+\varepsilon }}(\xi )\ge 0,\quad \xi \in \overline{\varSigma _{{{\lambda }_{0}}+\varepsilon }^{+}\cap {{B}_{\mathbb{H}}}(0,{{R}_{0}})}, $$

and therefore (4.14) is proved.

Using (4.13) and Lemma 3.2 once again, it follows that, for any \(\xi \in \varSigma _{\lambda }^{+}\) (here \(0\le \lambda \le \infty \)),

$$ {{w}_{\lambda }}(\xi )>0, $$
(4.18)

or

$$ {{w}_{\lambda }}(\xi )\equiv 0. $$
(4.19)

For (4.19), we can use the boundary condition of u to obtain \(u(\xi )\equiv 0\). In addition, it follows from (4.18) that \(u(\xi )\) is strictly increasing, which contradicts the boundary condition of u and (1.8). □