Riesz–Jacobi Transforms as Principal Value Integrals

Abstract

We establish an integral representation for the Riesz transforms naturally associated with classical Jacobi expansions. We prove that the Riesz–Jacobi transforms of odd orders express as principal value integrals against kernels having non-integrable singularities on the diagonal. On the other hand, we show that the Riesz–Jacobi transforms of even orders are not singular operators. In fact they are given as usual integrals against integrable kernels plus or minus, depending on the order, the identity operator. Our analysis indicates that similar results, existing in the literature and corresponding to several other settings related to classical discrete and continuous orthogonal expansions, should be reinvestigated so as to be refined and in some cases also corrected.

Appendix: Proof of Proposition 2.5

To begin with, we reduce the task to proving boundedness properties for simpler operators. Observe that for $f \in C_c^\infty (0,\pi )$

$$\begin{aligned} R_{2}^{\alpha ,\beta } f (\theta )&= - f(\theta ) - (\alpha +1/2)\cot \frac{\theta }{2} \, \partial _{\theta } \big (\mathcal J^{\alpha ,\beta }\big )^{-1} f(\theta )\\&\quad +(\beta +1/2)\tan \frac{\theta }{2} \, \partial _{\theta } \big (\mathcal J^{\alpha ,\beta }\big )^{-1} f(\theta ) \\&\quad + \tau _{\alpha ,\beta }^2 \big (\mathcal J^{\alpha ,\beta }\big )^{-1} f(\theta ), \quad \theta \in (0,\pi ), \end{aligned}$$

where one should replace f on the right-hand side by $\Pi _0 f$ when $\tau _{\alpha ,\beta } = 0$. Since $(\mathcal J^{\alpha ,\beta })^{-1}$ is bounded on $L^1(d\mu _{\alpha ,\beta })$, it suffices to consider

$$\begin{aligned} T_1^{\alpha ,\beta } f (\theta )&= \cot \frac{\theta }{2} \, \partial _{\theta } \big (\mathcal J^{\alpha ,\beta }\big )^{-1} f(\theta ),\quad \theta \in (0,\pi ),\\ T_2^{\alpha ,\beta } f (\theta )&= \tan \frac{\theta }{2} \, \partial _{\theta } \big (\mathcal J^{\alpha ,\beta }\big )^{-1} f(\theta ), \quad \theta \in (0,\pi ), \end{aligned}$$

with appropriate modification when $\tau _{\alpha ,\beta } = 0$. For symmetry reasons, we have $T_1^{\beta ,\alpha } f (\theta ) = - T_2^{\alpha ,\beta } \widetilde{f} (\pi - \theta )$, where $\widetilde{f} (\theta ) = f (\pi - \theta )$. Therefore proving Proposition 2.5 reduces to showing the following.

Lemma 7.1

Let $\alpha ,\beta >-1$. Then $T_1^{\alpha ,\beta }$ is a bounded operator from $L^1(d\mu _{\alpha ,\beta })$ to $L^1((3\pi /4,\pi ),d\mu _{\alpha ,\beta })$ and unbounded from $L^1(d\mu _{\alpha ,\beta })$ to $L^1((0,\pi /4),d\mu _{\alpha ,\beta })$.

The key tool which allows us to obtain this result is the well-known Schur criterion.

Lemma 7.2

Let $(X,\mu )$, $(Y,\nu )$ be $\sigma$-finite measure spaces and let K(x, y) be a measurable complex-valued kernel defined on $X \times Y$. If there exists a constant $C>0$ such that

$$\begin{aligned} \int _{X} |K(x,y)| \, d\mu (x) \le C, \quad \text {a.a. } y \in Y, \end{aligned}$$

(52)

then the integral operator $Tf(x) = \int _Y K(x,y) f(y) \, d\nu (y)$ is bounded from $L^1(Y,\nu )$ to $L^1(X,\mu )$. Moreover, when K is non-negative, the converse is true: boundedness of T from $L^1(Y,\nu )$ to $L^1(X,\mu )$ implies (52).

In the proof of Lemma 7.1 we will need also several technical results, which are gathered below. We begin with the following modification of Lemma 5.5 (corresponding to $K=(0,\pi /4)$, which is not admitted there).

Lemma 7.3

Let $\alpha ,\beta > -1$ be fixed. Assume that $\xi , \xi _1,\xi _2,\kappa _1,\kappa _2 \ge 0$ are fixed and such that $\alpha +\xi _1+\kappa _1, \, \beta +\xi _2+\kappa _2 \ge -1/2$. Then

$$\begin{aligned}&\iint \frac{d\Pi _{\alpha +\xi _1+\kappa _1}(u) \, d\Pi _{\beta +\xi _2+\kappa _2}(v) }{ \mathfrak {q}^{\alpha +\beta +\xi _1+\xi _2+\xi +1\slash 2}}\\&\quad \lesssim 1 + \chi _{\{ \alpha + \xi _1 + \xi = 0 \}} \log ^+ \frac{1}{|\theta - \varphi |}\\&\qquad + \bigg ( \frac{1}{\theta + \varphi } \bigg )^{ 2(\alpha + \xi _1 + \xi )} \bigg [ 1 + \chi _{\{\xi =1/2\}} \log \frac{\theta + \varphi }{|\theta - \varphi |} + \chi _{\{\xi >1/2\}} \bigg ( \frac{\theta + \varphi }{|\theta - \varphi |} \bigg )^{2\xi - 1} \bigg ], \end{aligned}$$

uniformly in $\theta \in (0,\pi /4)$, $\varphi \in (0,\pi )$, $\theta \ne \varphi$.

Proof

Observe that without any loss of generality we may and do assume that $\xi _1 = \xi _2 = 0$. Further, since $\mathfrak {q}$ is bounded and the measures $d\Pi _{\nu }$, $\nu \ge -1/2$, are finite, we have

$$\begin{aligned} \iint&\frac{d\Pi _{\alpha +\kappa _1}(u) \, d\Pi _{\beta +\kappa _2}(v) }{ \mathfrak {q}^{\alpha +\beta +\xi +1\slash 2}}\\&\lesssim 1 + \chi _{\{ \alpha + \beta + \xi + 1/2 >0 \}} \iint \frac{d\Pi _{\alpha +\kappa _1}(u) \, d\Pi _{\beta +\kappa _2}(v) }{ \mathfrak {q}^{\alpha + \beta + \xi + 1/2}}. \end{aligned}$$

Assuming that $\alpha + \beta + \xi + 1/2 >0$ and applying Corollary 5.4 to the integral against $d\Pi _{\beta +\kappa _2}(v)$ with the parameters $\nu = \beta +\kappa _2$, $\gamma = \alpha + \beta +\xi +1/2$, $A=1- u \sin \frac{\theta }{2}\sin \frac{\varphi }{2}\simeq 1$, $B= \cos \frac{\theta }{2}\cos \frac{\varphi }{2}$, we obtain

$$\begin{aligned}&\iint \frac{d\Pi _{\alpha +\kappa _1}(u) \, d\Pi _{\beta +\kappa _2}(v) }{ \mathfrak {q}^{\alpha +\beta +\xi +1\slash 2} }\\&\quad \lesssim 1 + \int \bigg ( \chi _{ \{ \alpha + \xi > \kappa _2 \} } \frac{1}{(1 - \cos \frac{\theta }{2}\cos \frac{\varphi }{2}- u\sin \frac{\theta }{2}\sin \frac{\varphi }{2})^{\alpha + \xi - \kappa _2}} \\&\quad \qquad \qquad + \chi _{ \{ \alpha + \xi = \kappa _2 \} } \log ^+ \frac{1}{1 - \cos \frac{\theta }{2}\cos \frac{\varphi }{2}- u\sin \frac{\theta }{2}\sin \frac{\varphi }{2}} \bigg )\, d\Pi _{\alpha +\kappa _1}(u) \\&\quad =: 1 + I_1 + I_2. \end{aligned}$$

We now analyze $I_1$ and $I_2$ separately. To treat $I_1$ we apply again Corollary 5.4 specified to $\nu =\alpha +\kappa _1$, $\gamma =\alpha +\xi -\kappa _2$, $A=1-\cos \frac{\theta }{2}\cos \frac{\varphi }{2}\simeq (\theta + \varphi )^2$, $B= \sin \frac{\theta }{2}\sin \frac{\varphi }{2}$, which leads to

$$\begin{aligned} I_1&\lesssim \chi _{\{\xi - 1/2 < \kappa _1 + \kappa _2\}} \bigg ( \frac{1}{\theta + \varphi } \bigg )^{ 2(\alpha + \xi - \kappa _2)}\\&\quad \ + \chi _{\{\xi - 1/2 = \kappa _1 + \kappa _2\}} \bigg ( \frac{1}{\theta + \varphi } \bigg )^{ 2(\alpha + \kappa _1 + 1/2)} \bigg ( 1 + \log ^+ \frac{\theta \varphi }{|\theta - \varphi |^2} \bigg ) \\&\quad \ + \chi _{\{\xi - 1/2 > \kappa _1 + \kappa _2\}} \bigg ( \frac{1}{\theta + \varphi } \bigg )^{ 2(\alpha + \kappa _1 + 1/2)} \frac{1}{|\theta - \varphi |^{2(\xi - 1/2 - \kappa _1 - \kappa _2)}} \\&=\,:\, J_1 + J_2 + J_3. \end{aligned}$$

The required bound for $J_1$ is straightforward, so let us pass to $J_2$. Since the constraint $\xi - 1/2 = \kappa _1 + \kappa _2$ implies $\xi \ge 1/2$ and $\kappa _1 + 1/2 = \xi - \kappa _2$, one can easily check that the conclusion follows (when $\xi > 1/2$ it is convenient to use (45)). Considering $J_3$, in this case $\xi > 1/2$ and we get

$$\begin{aligned} J_3 \le \bigg ( \frac{ |\theta - \varphi | }{\theta + \varphi } \bigg )^{2 \kappa _1} |\theta - \varphi |^{2 \kappa _2} \bigg ( \frac{1}{\theta + \varphi } \bigg )^{ 2(\alpha + \xi )} \bigg ( \frac{\theta + \varphi }{|\theta - \varphi |} \bigg )^{2\xi - 1}, \end{aligned}$$

which leads to the desired estimate. This finishes the analysis related to $I_1$.

Finally, we deal with $I_2$. The case $\alpha + \xi = 0$ is straightforward, so from now on we assume that $\alpha + \xi = \kappa _2 > 0$. To proceed it is convenient to distinguish two cases.

Case 1: $\alpha + \kappa _1 = -1/2$. Then in $I_2$ we have $\xi = 1/2 + \kappa _1 + \kappa _2 > 1/2$ and using (45) with any $\rho$ satisfying $0 < \rho \le (2\xi - 1) \wedge (2\alpha + 2\xi )$ we infer that

$$\begin{aligned} I_2 \lesssim 1 + \log ^+ \frac{1}{|\theta - \varphi |} \lesssim \frac{1}{|\theta - \varphi |^{\rho }} \lesssim \bigg ( \frac{1}{\theta + \varphi } \bigg )^{ 2(\alpha + \xi )} \bigg ( \frac{\theta + \varphi }{|\theta - \varphi |} \bigg )^{2\xi - 1}. \end{aligned}$$

Case 2: $\alpha + \kappa _1 > -1/2$. Applying (45) with a certain $\rho > 0$ satisfying $\rho < (\alpha + \kappa _1 + 1/2) \wedge (\alpha + \xi )$ and then Corollary 5.4 specified to $\nu =\alpha +\kappa _1$, $\gamma =\rho$, $A=1-\cos \frac{\theta }{2}\cos \frac{\varphi }{2}\simeq (\theta + \varphi )^2$, $B= \sin \frac{\theta }{2}\sin \frac{\varphi }{2}$, we get

$$\begin{aligned} I_2 \lesssim \int \frac{d\Pi _{\alpha +\kappa _1}(u)}{(1 - \cos \frac{\theta }{2}\cos \frac{\varphi }{2}- u\sin \frac{\theta }{2}\sin \frac{\varphi }{2})^{\rho }} \simeq \bigg ( \frac{1}{\theta + \varphi } \bigg )^{ 2\rho } \lesssim \bigg ( \frac{1}{\theta + \varphi } \bigg )^{ 2(\alpha + \xi )}. \end{aligned}$$

This finishes the proof of Lemma 7.3. $\square$

Lemma 7.4

Let $\nu , \lambda \in \mathbb {R}$, $\kappa < 1$, $\gamma > -1$ be fixed and such that $\nu + \lambda \ge 0$ and $\gamma + 1 + \nu - \kappa \ge 0$. Then, excluding the case when $\nu + \lambda = \gamma + 1 + \nu - \kappa = 0$, we have

$$\begin{aligned} \varphi ^{\nu } \int _0^{\pi /4} \bigg ( \frac{\varphi }{\theta + \varphi } \bigg )^{\lambda } \frac{\theta ^{\gamma } \, d\theta }{|\theta - \varphi |^{\kappa }} \lesssim 1,\quad \varphi \in (0,\pi ). \end{aligned}$$

Proof

Changing the variable of integration $\theta = \varphi s$ and keeping in mind that $\kappa < 1$ and $\gamma > -1$, we get

$$\begin{aligned} \varphi ^{\nu } \int _0^{\pi /4} \bigg ( \frac{\varphi }{\theta + \varphi } \bigg )^{\lambda } \frac{\theta ^{\gamma } \, d\theta }{|\theta - \varphi |^{\kappa }}&= \varphi ^{\gamma +1+\nu -\kappa } \int _{0}^{\pi /(4\varphi )} \bigg ( \frac{1}{1+s}\bigg )^{\lambda } \frac{s^{\gamma }\, ds}{|1-s|^{\kappa }} \\&\simeq \varphi ^{\gamma +1+\nu -\kappa } \bigg ( 1 + \chi _{\{\varphi < \pi /8\}}\int _2^{\pi /(4\varphi )} s^{\gamma - \lambda - \kappa }\, ds \bigg ) \\&\lesssim \varphi ^{\gamma +1+\nu -\kappa } \Big ( 1 + \chi _{\{\gamma -\lambda -\kappa +1=0\}} \log \frac{\pi }{\varphi } + \varphi ^{-\gamma + \lambda + \kappa -1} \Big ). \end{aligned}$$

Clearly, the last expression is bounded uniformly in $\varphi \in (0,\pi )$, in view of the assumptions imposed on the parameters. $\square$

The next result will be needed when dealing with the case $\alpha \ge -1/2$.

Lemma 7.5

Let $\alpha \ge -1/2$ and $\beta > -1$ be fixed. Consider the kernel $K(\theta ,\varphi )$ defined on $(0,\pi /4) \times (0,\pi )$ in the following way.

• (a) For $\beta \ge -1/2$,

  $$\begin{aligned} K(\theta ,\varphi ) = \theta ^{-1} \int _0^1 t \sinh \frac{t}{2}\iint \frac{\sin \frac{\theta - \varphi }{2} + (1-u) \cos \frac{\theta }{2}\sin \frac{\varphi }{2}}{(\cosh \frac{t}{2}- 1 + \mathfrak {q})^{\alpha + \beta + 3}} \, d\Pi _{\alpha }(u)\, d\Pi _{\beta }(v)\, dt. \end{aligned}$$

• (b) For $-1 < \beta < -1/2$,

  $$\begin{aligned} K(\theta ,\varphi ) = \theta ^{-1} \int _0^1 t \sinh \frac{t}{2}\iint \frac{\sin \frac{\theta - \varphi }{2} + (1-u) \cos \frac{\theta }{2}\sin \frac{\varphi }{2}}{(\cosh \frac{t}{2}- 1 + \mathfrak {q})^{\alpha + \beta + 4}} \, d\Pi _{\alpha }(u)\, \Pi _{\beta }(v)\,dv\, dt. \end{aligned}$$

Then we have

$$\begin{aligned} \int _0^{(\pi /4)\wedge (2\varphi )} |K(\theta ,\varphi )| \, d\mu _{\alpha ,\beta }(\theta ) \lesssim 1, \quad \varphi \in (0,\pi ). \end{aligned}$$

In the proofs of Lemmas 7.5 and 7.1 we will use the fact that for each fixed $\nu > -1/2$ we have

$$\begin{aligned} (1-u) \, d\Pi _{\nu } (u) \simeq d\Pi _{\nu + 1} (u), \quad u \in (0,1). \end{aligned}$$

(53)

Proof of Lemma 7.5

In the reasoning below we assume that $\theta \le (\pi /4) \wedge (2 \varphi )$, if not stated otherwise. Further, we define an auxiliary constant $\sigma = \sigma (\beta )$ which is equal to 0 if $\beta \ge -1/2$ and 1 if $-1 < \beta < -1/2$. We deal with items (a) and (b) simultaneously, but we consider the cases of $\alpha > -1/2$ and $\alpha = -1/2$ separately.

Case 1: $\alpha > -1/2$. By (36) we obtain

$$\begin{aligned} |K(\theta ,\varphi )| \lesssim \theta ^{-1} \int _0^1 t^2 \iint \frac{|\theta - \varphi | + (1-u) \varphi }{(t^2 + \mathfrak {q})^{\alpha + \beta + 3 + \sigma }} \, d\Pi _{\alpha }(u)\, d\Pi _{\beta + \sigma }(v) \, dt. \end{aligned}$$

Now applying Lemma 5.3 specified to $\nu = 5/2$, $\gamma = 2(\alpha + \beta + 3 + \sigma )$, $a=\sqrt{\mathfrak {q}}$, $B = 1$, and then Corollary 5.4 to the integral against $d\Pi _{\beta + \sigma }(v)$ with $\nu = \beta + \sigma$, $\gamma = \alpha + \beta + 3/2 + \sigma$, $A = 1- u \sin \frac{\theta }{2}\sin \frac{\varphi }{2}\simeq 1$, $B=\cos \frac{\theta }{2}\cos \frac{\varphi }{2}$, we see that

$$\begin{aligned} |K(\theta ,\varphi )|&\lesssim \theta ^{-1} \iint \frac{|\theta - \varphi | + (1-u) \varphi }{\mathfrak {q}^{\alpha +\beta +3/2 + \sigma }} \, d\Pi _{\beta + \sigma }(v) \, d\Pi _{\alpha }(u)\\&\simeq \theta ^{-1} \int \frac{|\theta - \varphi | + (1-u) \varphi }{(1 - \cos \frac{\theta }{2}\cos \frac{\varphi }{2}- u \sin \frac{\theta }{2}\sin \frac{\varphi }{2})^{\alpha + 1}} \, d\Pi _{\alpha }(u). \end{aligned}$$

To proceed, we split the region of integration in the last integral into the intervals $[-1,0]$ and [0, 1], and denote the corresponding expressions by $I_{-1}$ and $I_1$, respectively. In order to finish the proof of Case 1 it suffices to show that

$$\begin{aligned} I_{-1} + I_{1} \lesssim \theta ^{-1} \varphi ^{-2\alpha - 1}. \end{aligned}$$

Since for $u \in [-1,0]$ and $\theta ,\varphi \in (0,\pi )$ we have

$$\begin{aligned}&1 - \cos \frac{\theta }{2}\cos \frac{\varphi }{2}- u \sin \frac{\theta }{2}\sin \frac{\varphi }{2}\ge 1 - \cos \frac{\theta }{2}\cos \frac{\varphi }{2}\simeq (\theta + \varphi )^2, \end{aligned}$$

the conclusion for $I_{-1}$ is trivial. Using (53) and then Corollary 5.4 twice (with $\nu = \alpha$ or $\nu = \alpha + 1$ and $\gamma = \alpha + 1$, $A = 1- \cos \frac{\theta }{2}\cos \frac{\varphi }{2}\simeq (\theta + \varphi )^2$, $B=\sin \frac{\theta }{2}\sin \frac{\varphi }{2}$) we get the required estimate for $I_1$.

Case 2: $\alpha = -1/2$. Computing the integral against $d\Pi _{-1\slash 2}(u)$, applying the triangle inequality and then (43), we see that

$$\begin{aligned}&\bigg | \int \frac{\sin \frac{\theta - \varphi }{2} + (1-u) \cos \frac{\theta }{2}\sin \frac{\varphi }{2}}{(\cosh \frac{t}{2}- 1 + \mathfrak {q})^{\beta + 5/2 + \sigma }} \, d\Pi _{-1\slash 2}(u)\bigg | \\&\quad \simeq \bigg | \frac{ \sin \frac{\theta + \varphi }{2} }{(\cosh \frac{t}{2}-1 + q(\theta ,\varphi ,-1,v))^{\beta + 5/2 + \sigma }} + \frac{ \sin \frac{\theta - \varphi }{2} }{(\cosh \frac{t}{2}-1 + q(\theta ,\varphi ,1,v))^{\beta + 5/2 + \sigma }} \bigg | \\&\quad \le \Big |\sin \frac{\theta - \varphi }{2} \Big | \bigg | \frac{1}{(\cosh \frac{t}{2}-1 + q(\theta ,\varphi ,1,v))^{\beta + 5/2 + \sigma }} - \frac{1}{(\cosh \frac{t}{2}-1 + q(\theta ,\varphi ,-1,v))^{\beta + 5/2 + \sigma }} \bigg | \\&\quad \qquad + \frac{1}{(\cosh \frac{t}{2}-1 + q(\theta ,\varphi ,-1,v))^{\beta + 5/2 + \sigma }} \Big | \sin \frac{\theta - \varphi }{2} + \sin \frac{\theta + \varphi }{2} \Big | \\&\quad \lesssim \frac{ |\theta - \varphi | \theta \varphi ^{-1}}{ (t^2 + q(\theta ,\varphi ,1,v) )^{\beta + 5/2 + \sigma }} + \frac{\theta }{ (t^2 + q(\theta ,\varphi ,-1,v) )^{\beta + 5/2 + \sigma }}. \end{aligned}$$

Combining this with (36), Lemma 5.3 (applied with $\nu = 5/2$, $\gamma = 2(\beta + 5/2 + \sigma )$, $a=\sqrt{q(\theta ,\varphi ,\pm 1,v)}$, $B = 1$) and Corollary 5.4 (with $\nu = \beta + \sigma$, $\gamma = \beta + 1 + \sigma$, $A = 1 \mp \sin \frac{\theta }{2}\sin \frac{\varphi }{2}\simeq 1$, $B=\cos \frac{\theta }{2}\cos \frac{\varphi }{2}$) we obtain

$$\begin{aligned} |K(\theta ,\varphi )| \lesssim \int \bigg ( \frac{ |\theta - \varphi | \, \varphi ^{-1}}{ q(\theta ,\varphi ,1,v)^{\beta +1 + \sigma }} + \frac{1}{ q(\theta ,\varphi ,-1,v)^{\beta +1 + \sigma }} \bigg ) \, d\Pi _{\beta + \sigma }(v) \simeq \varphi ^{-1}, \end{aligned}$$

which concludes Case 2, and thus the proof of Lemma 7.5. $\square$

Lemma 7.6

Let $\alpha > -1$ and $\gamma > 0$ be fixed.

• (a) If $\alpha \ge -1/2$, then

  $$\begin{aligned}&\bigg | \int \frac{u \, d\Pi _{\alpha }(u)}{(\cosh \frac{t}{2}- 1 + \mathfrak {q})^{\gamma }} \bigg | \lesssim \frac{\theta \varphi }{(\theta + \varphi )^2} \int \frac{d\Pi _{\alpha }(u)}{(t^2 + \mathfrak {q})^{\gamma }}, \end{aligned}$$

  uniformly in $0 < t \le 1$, $\theta ,\varphi \in (0,\pi )$ and $v \in [-1,1]$.

• (b) If $-1 < \alpha < -1/2$, then

  $$\begin{aligned}&\bigg | \int \frac{ \Pi _{\alpha }(u)\,du}{(\cosh \frac{t}{2}- 1 + \mathfrak {q})^{\gamma }} \bigg | \lesssim \frac{\theta \varphi }{(\theta + \varphi )^2} \int \frac{d\Pi _{\alpha +1}(u)}{(t^2 + \mathfrak {q})^{\gamma }}, \end{aligned}$$

  uniformly in $0 < t \le 1$, $\theta ,\varphi \in (0,\pi )$ and $v \in [-1,1]$.

Proof

We will treat both items simultaneously. Since the measures $u \, d\Pi _{\alpha }(u)$, $\alpha \ge -1/2$, and $\Pi _{\alpha }(u)\,du$, $-1 < \alpha < -1/2$, are odd in $[-1,1]$, do not possess any atom at 0 and have finite total variation, we get

$$\begin{aligned}&\bigg | \int \frac{u \, d\Pi _{\alpha }(u)}{(\cosh \frac{t}{2}- 1 + \mathfrak {q})^{\gamma }} \bigg |\\&\le \int _{[0,1]} \bigg | \frac{1}{(\cosh \frac{t}{2}- 1 + \mathfrak {q})^{\gamma }} - \frac{1}{(\cosh \frac{t}{2}- 1 + q(\theta ,\varphi ,-u,v) )^{\gamma }} \bigg | \, d\Pi _{\alpha }(u), \\&\bigg | \int \frac{\Pi _{\alpha }(u)\,du}{(\cosh \frac{t}{2}- 1 + \mathfrak {q})^{\gamma }} \bigg |\\&\lesssim \int _{[0,1]} \bigg | \frac{1}{(\cosh \frac{t}{2}- 1 + \mathfrak {q})^{\gamma }} - \frac{1}{(\cosh \frac{t}{2}- 1 + q(\theta ,\varphi ,-u,v) )^{\gamma }} \bigg | \, d\Pi _{\alpha +1}(u); \end{aligned}$$

to obtain the second estimate we used also (36). Now the conclusion is an immediate consequence of (43) and the relations

$$\begin{aligned} q(\theta ,\varphi ,-u,v) \ge 1 - \cos \frac{\theta }{2}\cos \frac{\varphi }{2}\simeq (\theta + \varphi )^2, \quad \theta ,\varphi \in (0,\pi ), \quad u \in [0,1], \quad v \in [-1,1]. \end{aligned}$$

$\square$

Finally, in the proof of Lemma 7.1 we will frequently use the estimates, see (39) and (35),

$$\begin{aligned} |\partial _{\theta } \mathfrak {q}| \lesssim \sqrt{\mathfrak {q}}, \quad |\partial _u \mathfrak {q}| \lesssim \theta \varphi , \quad |\partial _v \mathfrak {q}| \lesssim 1, \quad |\partial _{\theta } \partial _u \mathfrak {q}| \lesssim \varphi , \quad |\partial _{\theta } \partial _v \mathfrak {q}| \lesssim \theta , \end{aligned}$$

(54)

holding uniformly in $\theta ,\varphi \in (0,\pi )$ and $u, v \in [-1,1]$.

Proof of Lemma 7.1

By Lemmas 3.3 and 3.1 we see that $T_1^{\alpha ,\beta }$ is an integral operator with the kernel

$$\begin{aligned} K(\theta ,\varphi ) = \cot \frac{\theta }{2} \int _0^\infty \partial _{\theta } H_t^{\alpha ,\beta } (\theta ,\varphi ) \, tdt, \quad \theta ,\varphi \in (0,\pi ), \quad \theta \ne \varphi ; \end{aligned}$$

note that $\tau _{\alpha ,\beta } = 0$ is also included. We first focus on the positive part of the lemma. Using sequently Lemma 3.4, Lemma 5.3 (applied with $\nu = 5/2$, $\gamma = 3$, $a=|\theta -\varphi |$ and $B=1$) and (45) with $\rho =1/2$ we obtain

$$\begin{aligned} |K(\theta ,\varphi )|&\lesssim 1 + (\pi - \theta ) \int _0^1 \frac{t^2\,dt}{(t + \pi - \theta + \pi - \varphi )^{2\beta +1} (t + |\theta - \varphi |)^3 } \\&\lesssim 1 + (\pi - \theta )^{-2\beta -1} \int _0^1 \frac{t^2\,dt}{(t + |\theta - \varphi |)^3 } \\&\lesssim 1 + (\pi - \theta )^{-2\beta -1} |\theta -\varphi |^{-1/2}, \quad \theta \in (3\pi /4, \pi ), \quad \varphi \in (0,\pi ). \end{aligned}$$

Now the desired conclusion is a direct consequence of Lemma 7.2.

We pass to proving the negative part. We split the region of integration in the definition of $K(\theta ,\varphi )$ into (0, 1) and $(1,\infty )$ denoting the resulting expressions by $K_0 (\theta ,\varphi )$ and $K_{\infty }(\theta ,\varphi )$, respectively. We first show that the operator $T_\infty$ associated with the kernel $K_{\infty }(\theta ,\varphi )$ is $L^1(d\mu _{\alpha ,\beta })$-bounded. Using the series definition of $H_t^{\alpha ,\beta } (\theta ,\varphi )$, see (5), and proceeding as in the proof of [16, Lemma 3.8], we obtain

$$\begin{aligned} |\partial _{\theta } H_t^{\alpha ,\beta } (\theta ,\varphi )| \lesssim \sin \theta \, e^{-ct}, \quad t \ge 1, \quad \theta ,\varphi \in (0,\pi ), \end{aligned}$$

for some $c=c_{\alpha ,\beta }>0$. This combined with Lemma 7.2 gives us the desired property for $T_\infty$.

It remains to deal with the operator $T_0$ associated to the kernel $K_0(\theta ,\varphi )$. We will show that $T_0$ is not bounded from $L^1(d\mu _{\alpha ,\beta })$ to $L^1((0,\pi /4),d\mu _{\alpha ,\beta })$. This will finish the proof. It is convenient to distinguish four cases depending on whether each of the parameters of type $\alpha ,\beta$ is less than $-1/2$ or not.

Case 1: $\alpha ,\beta \ge -1/2$. Using (49) and the decomposition (41) we arrive at

$$\begin{aligned} K_0 (\theta ,\varphi )&= c \cot \frac{\theta }{2} \int _0^1 t \sinh \frac{t}{2}\\&\quad \times \iint \frac{\big [ \sin \frac{\theta -\varphi }{2} + (1-u) \cos \frac{\theta }{2}\sin \frac{\varphi }{2}\big ] - (1-v) \sin \frac{\theta }{2}\cos \frac{\varphi }{2}}{ (\cosh \frac{t}{2}- 1 + \mathfrak {q})^{\alpha + \beta + 3} } \, d\Pi _{\alpha }(u)\, d\Pi _{\beta }(v)\, dt \\&=\,:\, c L_1(\theta ,\varphi ) - c L_2(\theta ,\varphi ), \quad \theta ,\varphi \in (0,\pi ), \quad \theta \ne \varphi , \end{aligned}$$

with some non-zero constant $c=c_{\alpha ,\beta }$. We claim that $L_2 (\theta ,\varphi )$ produces a bounded operator from $L^1(d\mu _{\alpha ,\beta })$ to $L^1((0,\pi /4),d\mu _{\alpha ,\beta })$. Indeed, applying Lemma 5.3 (specified to $\nu = 5/2$, $\gamma = 2(\alpha + \beta + 3)$, $a=\sqrt{\mathfrak {q}}$, $B = 1$), splitting the integration in v into $[-1,0]$ and [0, 1], and then using the estimates

$$\begin{aligned} 1- u \sin \frac{\theta }{2}\sin \frac{\varphi }{2}\simeq 1, \quad u\in [-1,1], \quad \theta \in (0,\pi /4), \quad \varphi \in (0,\pi ), \end{aligned}$$

(55)

and (53), we see that

$$\begin{aligned} |L_2(\theta ,\varphi )|&\lesssim \int _0^1 t^2 \iint \frac{(1-v)\, d\Pi _{\alpha }(u)\, d\Pi _{\beta }(v)}{(t^2 + \mathfrak {q})^{\alpha +\beta +3}} \, dt \simeq \iint \frac{(1-v)\, d\Pi _{\alpha }(u)\, d\Pi _{\beta }(v)}{\mathfrak {q}^{\alpha +\beta +3/2}} \\&\lesssim 1 + \int _{[0,1]} \int \frac{ d\Pi _{\alpha }(u)\, d\Pi _{\beta +1}(v)}{\mathfrak {q}^{\alpha +\beta +3/2}}, \quad \theta \in (0,\pi /4), \quad \varphi \in (0,\pi ), \quad \theta \ne \varphi ; \end{aligned}$$

note that here $\beta = -1/2$ is also included. Now Lemma 7.3 (specified to $\xi _1 = \kappa _1 = \kappa _2 = \xi = 0$, $\xi _2 = 1$) leads to

$$\begin{aligned} |L_2(\theta ,\varphi )|\lesssim & {} (\theta + \varphi )^{-2\alpha } + |\theta -\varphi |^{-1/2}\\\lesssim & {} (\theta + \varphi )^{-2\alpha - 1} |\theta -\varphi |^{-1/2}, \quad \theta \in (0,\pi /4), \quad \varphi \in (0,\pi ), \quad \theta \ne \varphi . \end{aligned}$$

This, in view of Lemmas 7.2 and 7.4 (applied with $\gamma = \lambda = - \nu = 2\alpha + 1$, $\kappa = 1/2$), gives the asserted property for the operator connected with $L_2 (\theta ,\varphi )$.

We now focus on $L_1(\theta ,\varphi )$ and show that it produces an unbounded operator from $L^1(d\mu _{\alpha ,\beta })$ to $L^1((0,\pi /4),d\mu _{\alpha ,\beta })$. By combining Lemma 7.5(a) with Lemma 7.2 we know that $\chi _{ \{ \theta \le 2\varphi \} } L_1(\theta ,\varphi )$ produces a bounded operator from $L^1(d\mu _{\alpha ,\beta })$ to $L^1((0,\pi /4),d\mu _{\alpha ,\beta })$. Since $\chi _{ \{ \theta > 2\varphi \} } L_1(\theta ,\varphi ) \ge 0$, in order to finish Case 1 it suffices to show that, see Lemma 7.2,

$$\begin{aligned} \mathop {{{\mathrm{ess\,sup}}}}\limits _{\varphi \in (0,\pi /8)} \int _{2\varphi }^{\pi /4} L_1(\theta ,\varphi ) \, \theta ^{2\alpha + 1} \, d\theta = \infty . \end{aligned}$$

(56)

Using Lemma 5.3 (taken with $\nu = 5/2$, $\gamma = 2( \alpha + \beta + 3)$, $a=\sqrt{\mathfrak {q}}$, $B = 1$) and Corollary 5.4 twice (first to the integral against $d\Pi _{\beta }(v)$ with $\nu = \beta$, $\gamma = \alpha + \beta + 3/2$, $A = 1 - u \sin \frac{\theta }{2}\sin \frac{\varphi }{2}\simeq 1$, $B=\cos \frac{\theta }{2}\cos \frac{\varphi }{2}$, and then to the resulting integral against $d\Pi _{\alpha }(u)$ with $\nu = \alpha$, $\gamma = \alpha + 1$, $A = 1 - \cos \frac{\theta }{2}\cos \frac{\varphi }{2}\simeq (\theta + \varphi )^2$, $B=\sin \frac{\theta }{2}\sin \frac{\varphi }{2}$) we obtain

$$\begin{aligned} \chi _{ \{ \pi /4 \ge \theta > 2\varphi \} } L_1(\theta ,\varphi )&\gtrsim \chi _{ \{ \pi /4 \ge \theta > 2\varphi \} } \, \theta ^{-1} \int _0^1 t^2 \iint \frac{ \theta - \varphi }{ (t^2 + \mathfrak {q})^{\alpha + \beta + 3}} \, d\Pi _{\alpha }(u)\, d\Pi _{\beta }(v)\, dt \\&\simeq \chi _{ \{ \pi /4 \ge \theta > 2\varphi \} } \, \iint \frac{d\Pi _{\beta }(v)\, d\Pi _{\alpha }(u)}{ \mathfrak {q}^{\alpha + \beta + 3/2}} \\&\simeq \chi _{ \{ \pi /4 \ge \theta > 2\varphi \} } \, \int \frac{d\Pi _{\alpha }(u)}{ (1 - \cos \frac{\theta }{2}\cos \frac{\varphi }{2}- u \sin \frac{\theta }{2}\sin \frac{\varphi }{2})^{\alpha + 1}}\\&\simeq \chi _{ \{ \pi /4 \ge \theta > 2\varphi \} } \, \theta ^{-2\alpha - 2}. \end{aligned}$$

This confirms (56) and finishes the reasoning justifying Lemma 7.1 for $\alpha ,\beta \ge -1/2$.

Case 2: $-1 < \alpha < -1/2 \le \beta$. Decompose $K_0(\theta ,\varphi ) = L_1(\theta ,\varphi ) + L_2(\theta ,\varphi ) + L_3(\theta ,\varphi )$, where $L_j(\theta ,\varphi )$ corresponds to the term with constant $C^j_{\alpha ,\beta }$ in (50), $j=1,2,3$ (observe that $C^j_{\alpha ,\beta } \ne 0$). We first ensure that $L_1(\theta ,\varphi )$ and $L_2(\theta ,\varphi )$ are associated with bounded operators from $L^1(d\mu _{\alpha ,\beta })$ to $L^1((0,\pi /4),d\mu _{\alpha ,\beta })$. Using Lemma 7.6(b) (to $L_2(\theta ,\varphi )$ with $\gamma = \alpha + \beta + 3$) and then (54) and (36), we get

$$\begin{aligned}&|L_1(\theta ,\varphi )| + |L_2(\theta ,\varphi )| \lesssim \int _0^1 t^2 \\&\quad \times \iint \bigg [ \frac{\varphi \, \sqrt{ \mathfrak {q}} }{(t^2 + \mathfrak {q})^{\alpha +\beta +4}} + \frac{\varphi ^2}{(\theta + \varphi )^2} \frac{1 }{(t^2 + \mathfrak {q})^{\alpha +\beta +3}} \bigg ] \, d\Pi _{\alpha +1}(u)\, d\Pi _{\beta }(v)\, dt. \end{aligned}$$

Then an application of Lemma 5.3 (specified to $\nu = 5/2$, $a=\sqrt{\mathfrak {q}}$, $B = 1$ and $\gamma = 2( \alpha + \beta + 4)$ or $\gamma = 2( \alpha + \beta + 3)$) and then Lemma 7.3 (choosing $\xi _1 = 1$, $\kappa _1 = \xi _2 = \kappa _2 = 0$ and $\xi = 1/2$ or $\xi = 0$) leads to

$$\begin{aligned} |L_1(\theta ,\varphi )| + |L_2(\theta ,\varphi )|&\lesssim \iint \bigg [ \frac{\varphi }{ \mathfrak {q}^{\alpha +\beta +2}} + \frac{\varphi ^2}{(\theta + \varphi )^2} \frac{1 }{\mathfrak {q}^{\alpha +\beta +3/2}} \bigg ] \, d\Pi _{\alpha +1}(u)\, d\Pi _{\beta }(v)\\&\lesssim \frac{\varphi }{(\theta + \varphi )^{2\alpha + 3}} \bigg ( \frac{\theta + \varphi }{|\theta - \varphi |} \bigg )^{1/2} + \frac{\varphi ^2}{ (\theta + \varphi )^{2\alpha + 4} }\\&\lesssim \frac{\varphi }{(\theta + \varphi )^{2\alpha + 5/2}} \frac{1}{|\theta - \varphi |^{1/2}}, \end{aligned}$$

provided that $\theta \in (0,\pi /4)$, $\varphi \in (0,\pi )$, $\theta \ne \varphi$. This, with the aid of Lemmas 7.2 and 7.4 (taken with $\gamma = 2\alpha + 1$, $\lambda = 2\alpha + 5/2$, $\nu = -2\alpha - 3/2$, $\kappa = 1/2$), finishes the analysis concerning $L_1(\theta ,\varphi )$ and $L_2(\theta ,\varphi )$.

It remains to check that $L_3(\theta ,\varphi )$ defines an unbounded operator from $L^1(d\mu _{\alpha ,\beta })$ to $L^1((0,\pi /4),d\mu _{\alpha ,\beta })$. Since

$$\begin{aligned} \partial _{\theta } \mathfrak {q}= -\frac{1}{2} u \cos \frac{\theta }{2}\sin \frac{\varphi }{2}+ \frac{1}{2} v \sin \frac{\theta }{2}\cos \frac{\varphi }{2}, \end{aligned}$$

(57)

we consider the kernels $J_k(\theta ,\varphi )$, $k=-1,0,1$, given by

$$\begin{aligned}&J_0(\theta ,\varphi ) = \cot \frac{\theta }{2} \cos \frac{\theta }{2}\sin \frac{\varphi }{2}\int _0^1 t \sinh \frac{t}{2}\iint \frac{u \, d\Pi _{-1\slash 2}(u)\, d\Pi _{\beta }(v)}{ (\cosh \frac{t}{2}- 1 + \mathfrak {q})^{\alpha + \beta + 3} } \, dt, \\&\cos \frac{\theta }{2}\cos \frac{\varphi }{2}\int _0^1 t \sinh \frac{t}{2}\iint \frac{v \, d\Pi _{-1\slash 2}(u)\, d\Pi _{\beta }(v)}{ (\cosh \frac{t}{2}- 1 + \mathfrak {q})^{\alpha + \beta + 3} } \, dt =\,:\, J_{-1}(\theta ,\varphi ) + J_1(\theta ,\varphi ), \end{aligned}$$

where $J_{-1}(\theta ,\varphi )$ and $J_1(\theta ,\varphi )$ correspond to the integration in v restricted to $[-1,0]$ and [0, 1], respectively. We will show that the operators associated with $J_0(\theta ,\varphi )$ and $J_{-1}(\theta ,\varphi )$ are bounded from $L^1(d\mu _{\alpha ,\beta })$ to $L^1((0,\pi /4),d\mu _{\alpha ,\beta })$, whereas the one connected with $J_1(\theta ,\varphi )$ is unbounded.

By (55) the required property for $J_{-1}(\theta ,\varphi )$ is straightforward. Next we focus on $J_0(\theta ,\varphi )$. Combining Lemma 7.6(a) (specified to $\gamma = \alpha + \beta + 3$) with Lemma 5.3 (applied with $\nu = 5/2$, $\gamma = 2(\alpha + \beta + 3)$, $a=\sqrt{\mathfrak {q}}$, $B = 1$) and Lemma 7.3 (with $\xi _1 = -\alpha - 1/2$, $\kappa _1 = \xi _2 = \kappa _2 = 0$, $\xi = \alpha + 3/2 > 1/2$) we infer that

$$\begin{aligned} |J_0(\theta ,\varphi )|&\lesssim \frac{\varphi ^2}{(\theta + \varphi )^2} \iint \frac{ d\Pi _{-1\slash 2}(u)\, d\Pi _{\beta }(v)}{ \mathfrak {q}^{\alpha + \beta + 3/2} } \\&\lesssim \frac{\varphi ^2}{(\theta + \varphi )^2} \frac{1}{|\theta - \varphi |^{2\alpha + 2}}, \quad \, \theta \in (0,\pi /4), \quad \varphi \in (0,\pi ), \quad \theta \ne \varphi . \end{aligned}$$

Now the conclusion for $J_0(\theta ,\varphi )$ is a direct consequence of Lemmas 7.2 and 7.4 (specified to $\gamma = 2\alpha + 1$, $\lambda = 2$, $\nu = 0$ and $\kappa = 2\alpha + 2$).

Finally, we deal with $J_{1}(\theta ,\varphi )$. Since the integrand in the definition of $J_{1}(\theta ,\varphi )$ is non-negative, in view of Lemma 7.2 it is sufficient to show that

$$\begin{aligned} \mathop {{{\mathrm{ess\,sup}}}}\limits _{\varphi \in (0,\pi /8)} \int _0^{\pi /4} J_1(\theta ,\varphi ) \, \theta ^{2\alpha + 1} \, d\theta = \infty . \end{aligned}$$

(58)

Restricting the integration in v to the interval [1 / 2, 1], using Lemma 5.3 (specified to $\nu = 5/2$, $\gamma = 2(\alpha + \beta + 3)$, $a=\sqrt{\mathfrak {q}}$, $B = 1$) and then integrating in u, we see that for $\theta ,\varphi \in (0,\pi /4)$, $\theta \ne \varphi$, one has

$$\begin{aligned} J_1(\theta ,\varphi )&\gtrsim \int _0^1 t^2 \int _{[1/2,1]} \int \frac{ d\Pi _{-1\slash 2}(u)\, d\Pi _{\beta }(v)}{(t^2 + \mathfrak {q})^{\alpha +\beta +3}} \, dt \\&\simeq \int _{[1/2,1]} \int \frac{ d\Pi _{-1\slash 2}(u)\, d\Pi _{\beta }(v)}{ \mathfrak {q}^{\alpha +\beta +3/2}} \\&\simeq \int _{[1/2,1]} \frac{ d\Pi _{\beta }(v)}{ (1 - \sin \frac{\theta }{2}\sin \frac{\varphi }{2}- v \cos \frac{\theta }{2}\cos \frac{\varphi }{2})^{\alpha +\beta +3/2}} \\&\simeq \int \frac{ d\Pi _{\beta }(v)}{ (1 - \sin \frac{\theta }{2}\sin \frac{\varphi }{2}- v \cos \frac{\theta }{2}\cos \frac{\varphi }{2})^{\alpha +\beta +3/2}}; \end{aligned}$$

here we also used the fact that the essential contribution to the last integral comes from integration over [1 / 2, 1]. Applying now Corollary 5.4 (with $\nu = \beta$, $\gamma = \alpha + \beta + 3/2$, $A = 1 - \sin \frac{\theta }{2}\sin \frac{\varphi }{2}\simeq 1$, $B=\cos \frac{\theta }{2}\cos \frac{\varphi }{2}$) we get

$$\begin{aligned} \int _0^{\pi /4} J_1(\theta ,\varphi ) \, \theta ^{2\alpha + 1} \, d\theta \gtrsim \int _0^{\pi /4} \frac{\theta ^{2\alpha + 1} \, d\theta }{|\theta - \varphi |^{2\alpha + 2}} \gtrsim \int _{2\varphi }^{\pi /4} \theta ^{-1} \, d\theta = \log \frac{\pi }{8 \varphi }, \quad \varphi \in (0,\pi /8), \end{aligned}$$

which confirms (58) and completes the case $-1 < \alpha < -1/2 \le \beta$.

Case 3: $-1 < \beta < -1/2 \le \alpha$. From [16, Proposition 2.3(iii)] we get

$$\begin{aligned} \partial _{\theta } {H}_t^{\alpha ,\beta }(\theta ,\varphi ) =&\; C_{\alpha ,\beta }^1 \sinh \frac{t}{2}\iint \frac{\partial _\theta \mathfrak {q}\, \partial _v \mathfrak {q}}{(\cosh \frac{t}{2}- 1 + \mathfrak {q})^{\alpha +\beta +4}} \, d\Pi _{\alpha }(u)\, \Pi _{\beta }(v)\,dv\\&+ C_{\alpha ,\beta }^2 \sinh \frac{t}{2}\iint \frac{\partial _\theta \partial _v \mathfrak {q}}{(\cosh \frac{t}{2}- 1 + \mathfrak {q})^{\alpha +\beta +3}} \, d\Pi _{\alpha }(u)\, \Pi _{\beta }(v)\,dv\\&+ C_{\alpha ,\beta }^3 \sinh \frac{t}{2}\iint \frac{\partial _\theta \mathfrak {q}}{(\cosh \frac{t}{2}- 1 + \mathfrak {q})^{\alpha +\beta +3}} \, d\Pi _{\alpha }(u)\, d\Pi _{-1\slash 2}(v), \end{aligned}$$

where $C_{\alpha ,\beta }^j \ne 0$, $j=1,2,3$. We denote by $L_j(\theta ,\varphi )$, $j=1,2,3$, the corresponding components of $K_0(\theta ,\varphi )$.

We first show that the operators emerging from $L_2(\theta ,\varphi )$ and $L_3(\theta ,\varphi )$ are bounded from $L^1(d\mu _{\alpha ,\beta })$ to $L^1((0,\pi /4),d\mu _{\alpha ,\beta })$. Indeed, applying Lemma 7.6(a) (to the component of $L_3(\theta ,\varphi )$ connected with the first term in the decomposition (57) of $\partial _{\theta } \mathfrak {q}$) and also (54) together with (36), we obtain

$$\begin{aligned} |L_2(\theta ,\varphi )| + |L_3(\theta ,\varphi )|&\lesssim \int _0^1 t^2 \iint \frac{d\Pi _{\alpha }(u)\, d\Pi _{\beta +1}(v)}{(t^2 + \mathfrak {q})^{\alpha +\beta +3}} \, dt \\&\quad \, + \bigg ( \frac{\varphi ^2}{(\theta + \varphi )^2} + 1 \bigg )\int _0^1 t^2 \iint \frac{d\Pi _{\alpha }(u)\, d\Pi _{-1\slash 2}(v)}{(t^2 + \mathfrak {q})^{\alpha +\beta +3}} \, dt \\&\simeq \sum _{R=0,1} \int _0^1 t^2 \iint \frac{d\Pi _{\alpha }(u)\, d\Pi _{\beta , R}(v)}{(t^2 + \mathfrak {q})^{\alpha +\beta +3}} \, dt. \end{aligned}$$

Making use of Lemma 5.3 (specified to $\nu = 5/2$, $\gamma = 2(\alpha + \beta + 3)$, $a=\sqrt{\mathfrak {q}}$, $B = 1$) and then Lemma 7.3 (with $\xi _1 = \kappa _1 = \kappa _2 = 0$ and $\xi _2 = - \beta -1/2$, $\xi = \beta + 3/2 > 1/2$ if $R=0$, and $\xi _2 = 1$, $\xi = 0$ if $R=1$) yields

$$\begin{aligned} |L_2(\theta ,\varphi )| + |L_3(\theta ,\varphi )| \lesssim \sum _{R=0,1} \iint \frac{d\Pi _{\alpha }(u)\, d\Pi _{\beta , R}(v)}{\mathfrak {q}^{\alpha +\beta +3/2}} \lesssim \frac{1}{(\theta + \varphi )^{2\alpha + 1}} \, \frac{1}{|\theta - \varphi |^{2\beta + 2}}, \end{aligned}$$

for $\theta \in (0,\pi /4)$, $\varphi \in (0,\pi )$, $\theta \ne \varphi$. This, in view of Lemmas 7.2 and 7.4 (specified to $\gamma = \lambda = - \nu = 2\alpha + 1$ and $\kappa = 2\beta + 2$), gives the asserted property for $L_2(\theta ,\varphi )$ and $L_3(\theta ,\varphi )$.

It remains to investigate $L_1(\theta ,\varphi )$ and prove that it defines an unbounded operator from $L^1(d\mu _{\alpha ,\beta })$ to $L^1((0,\pi /4),d\mu _{\alpha ,\beta })$. Observe that, up to a non-zero multiplicative constant, $L_1(\theta ,\varphi )$ is equal to, see (41),

$$\begin{aligned}&\cot \frac{\theta }{2} \cos \frac{\theta }{2}\cos \frac{\varphi }{2}\int _0^1 t \sinh \frac{t}{2}\iint \frac{\big [ \sin \frac{\theta -\varphi }{2} + (1-u) \cos \frac{\theta }{2}\sin \frac{\varphi }{2}\big ] - (1-v) \sin \frac{\theta }{2}\cos \frac{\varphi }{2}}{ (\cosh \frac{t}{2}- 1 + \mathfrak {q})^{\alpha + \beta + 4} }\\&\qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \, \times d\Pi _{\alpha }(u)\, \Pi _{\beta }(v)\,dv\, dt. \end{aligned}$$

This expression splits into two terms according to the main difference in the numerator of the fraction under the double integral. We denote by $J_{-1}(\theta ,\varphi )$ and $J_1(\theta ,\varphi )$ the first of these terms with the integration in v restricted to $[-1,0]$ and [0, 1], respectively. Further, let $J_2(\theta ,\varphi )$ stand for the second term. We will prove that $\chi _{ \{ \theta \le 2\varphi \} } ( J_{-1}(\theta ,\varphi ) + J_1(\theta ,\varphi ) )$, $\chi _{ \{ \theta > 2\varphi \} } J_{-1}(\theta ,\varphi )$ and $J_2(\theta ,\varphi )$ define bounded operators from $L^1(d\mu _{\alpha ,\beta })$ to $L^1((0,\pi /4),d\mu _{\alpha ,\beta })$, whereas $\chi _{ \{ \theta > 2\varphi \} } J_1(\theta ,\varphi )$ corresponds to an unbounded operator between those spaces.

Using (b) of Lemma 7.5 and (55), respectively, the asserted property for the first two kernels follows. We now focus on $J_2(\theta ,\varphi )$. Splitting the integration in v into intervals $[-1,0]$, [0, 1], and then using (55) to the first term and the estimates (36), (53) to the second one, we obtain

$$\begin{aligned} |J_2(\theta ,\varphi )|&\lesssim 1 + \int _0^1 t^2 \iint \frac{ d\Pi _{\alpha }(u)\, d\Pi _{\beta + 2}(v) }{(t^2 + \mathfrak {q})^{\alpha +\beta +4}} \, dt. \end{aligned}$$

This together with Lemma 5.3 (specified to $\nu = 5/2$, $\gamma = 2(\alpha + \beta + 4)$, $a=\sqrt{\mathfrak {q}}$, $B = 1$) and Lemma 7.3 (choosing $\xi _1 = \kappa _1 = \kappa _2 = 0$, $\xi _2 = 2$ and $\xi = 0$) leads to

$$\begin{aligned} |J_2(\theta ,\varphi )|\lesssim & {} (\theta + \varphi )^{-2 \alpha } + |\theta - \varphi |^{-1/2}\\\lesssim & {} (\theta + \varphi )^{-2 \alpha - 1} |\theta - \varphi |^{-1/2}, \quad \theta \in (0,\pi /4), \quad \varphi \in (0,\pi ), \quad \theta \ne \varphi . \end{aligned}$$

Now the conclusion follows from Lemmas 7.2 and 7.4 (with $\gamma = \lambda = - \nu = 2\alpha + 1$ and $\kappa = 1/2$).

Finally, we consider $\chi _{ \{ \theta > 2\varphi \} } J_1(\theta ,\varphi )$. Since the integrand in the definition of $\chi _{ \{ \theta > 2\varphi \} } J_1(\theta ,\varphi )$ is non-positive, in view of Lemma 7.2 it is enough to ensure that

$$\begin{aligned} \mathop {{{\mathrm{ess\,sup}}}}\limits _{\varphi \in (0,\pi /8)} \int _{2\varphi }^{\pi /4} |J_1(\theta ,\varphi )| \, \theta ^{2\alpha + 1} \, d\theta = \infty . \end{aligned}$$

(59)

Restricting the integration in v to [1 / 2, 1] and taking into account (36) we obtain

$$\begin{aligned}&\chi _{ \{ \pi /4 \ge \theta > 2\varphi \} } | J_1(\theta ,\varphi ) |\\&\quad \gtrsim \chi _{ \{ \pi /4 \ge \theta > 2\varphi \} } \, \theta ^{-1} \int _0^1 t^2 \int _{[1/2,1]} \int \frac{ \theta - \varphi }{ (t^2 + \mathfrak {q})^{\alpha + \beta + 4}} \, d\Pi _{\alpha }(u)\, d\Pi _{\beta +1}(v)\, dt . \end{aligned}$$

Since the last expression is comparable with a similar one with no restriction in v, an application of Lemma 5.3 (specified to $\nu = 5/2$, $\gamma = 2( \alpha + \beta + 4)$, $a=\sqrt{\mathfrak {q}}$, $B = 1$) and Corollary 5.4 twice (first to the integral with respect to $d\Pi _{\beta +1}(v)$ with parameters $\nu = \beta + 1$, $\gamma = \alpha + \beta + 5/2$, $A = 1 - u \sin \frac{\theta }{2}\sin \frac{\varphi }{2}\simeq 1$, $B=\cos \frac{\theta }{2}\cos \frac{\varphi }{2}$ and then to the resulting integral with respect to $d\Pi _{\alpha }(u)$ with $\nu = \alpha$, $\gamma = \alpha + 1$, $A = 1 - \cos \frac{\theta }{2}\cos \frac{\varphi }{2}\simeq (\theta + \varphi )^2$, $B=\sin \frac{\theta }{2}\sin \frac{\varphi }{2}$) gives

$$\begin{aligned} \chi _{ \{ \pi /4 \ge \theta > 2\varphi \} } | J_1(\theta ,\varphi ) |&\gtrsim \chi _{ \{ \pi /4 \ge \theta > 2\varphi \} } \, \iint \frac{d\Pi _{\beta +1}(v)\, d\Pi _{\alpha }(u)}{ \mathfrak {q}^{\alpha + \beta + 5/2}} \\&\simeq \chi _{ \{ \pi /4 \ge \theta > 2\varphi \} } \, \int \frac{d\Pi _{\alpha }(u)}{ (1 - \cos \frac{\theta }{2}\cos \frac{\varphi }{2}- u \sin \frac{\theta }{2}\sin \frac{\varphi }{2})^{\alpha + 1}} \\&\simeq \chi _{ \{ \pi /4 \ge \theta > 2\varphi \} } \, \theta ^{-2\alpha - 2}, \end{aligned}$$

which implies (59). This completes the reasoning justifying Lemma 7.1 for $-1 < \beta < -1/2 \le \alpha$.

Case 4: $-1 < \alpha ,\beta < -1/2$. We decompose $K_0(\theta ,\varphi )$ into 7 parts according to the formula (51) (note that the constants $C^j_{\alpha ,\beta }$, $j=1,\ldots ,7$, appearing there are non-zero) and denote them by $L_j(\theta ,\varphi )$, $j=1,\ldots ,7$, respectively. We first deal with $L_j(\theta ,\varphi )$, $j\ne 5,7$. We will show that these kernels produce bounded operators from $L^1(d\mu _{\alpha ,\beta })$ to $L^1((0,\pi /4),d\mu _{\alpha ,\beta })$. Using Lemma 7.6(b) (when it comes to $L_2(\theta ,\varphi )$ and $L_4(\theta ,\varphi )$) and (54) and (36), we see that they can be estimated simultaneously by

$$\begin{aligned} \theta ^{-1} \int _0^1 t^2 \sum _{\begin{array}{c} K,R = 0,1 \\ K+R > 0 \end{array}}&\iint \bigg ( \frac{ \chi _{ \{ K=1 \} } \, \theta \varphi \, \sqrt{\mathfrak {q}} }{(t^2 + \mathfrak {q})^{\alpha +\beta +4 + R}} + \frac{(\theta \varphi )^K}{ (\theta + \varphi )^{2K} } \frac{ \chi _{ \{ K=1 \} } \varphi + \chi _{ \{ R=1 \} } \theta ^{1+K} \varphi ^K }{(t^2 + \mathfrak {q})^{\alpha +\beta +2 + K + R}}\, \bigg )\\&\times d\Pi _{\alpha , K}(u)\, d\Pi _{\beta , R}(v)\, dt. \end{aligned}$$

Since $\theta ^K( \chi _{ \{ K=1 \} } \varphi + \chi _{ \{ R=1 \} } \theta ^{1+K} \varphi ^K ) \lesssim \theta \varphi ^K$, $K=0,1$, an application of Lemma 5.3 (with $\nu = 5/2$, $a=\sqrt{\mathfrak {q}}$, $B = 1$ and $\gamma = 2(\alpha + \beta + 4 + R)$ or $\gamma = 2(\alpha + \beta + 2 + K + R)$) gives further bound by

$$\begin{aligned} \frac{1}{|\theta - \varphi |^{\alpha + 1}} + \sum _{\begin{array}{c} K,R = 0,1 \\ K+R > 0 \end{array}}&\iint \bigg ( \frac{ \chi _{ \{ K=1 \} } \, \varphi }{ \mathfrak {q}^{\alpha +\beta +2 + R}} + \frac{ \varphi ^{2K} }{ (\theta + \varphi )^{2K} } \frac{ \chi _{ \{ \alpha + \beta + 1/2 + K + R > 0 \} } }{ \mathfrak {q}^{\alpha +\beta + 1/2 + K + R} } \bigg ) \\&\times d\Pi _{\alpha , K}(u)\, d\Pi _{\beta , R}(v); \end{aligned}$$

here we used also the relation $\mathfrak {q}\gtrsim (\theta - \varphi )^2$ and (45) with $\rho = \alpha + 1$. Next, using Lemma 7.3 to the first term under the sum above (specified to $\xi _1 = 1$, $\kappa _1 = \kappa _2 = 0$ and $\xi _2 = 1$, $\xi = 1/2$ if $R=1$, and $\xi _2 = - \beta -1/2$, $\xi = \beta + 1 < 1/2$ if $R=0$) and also to the second term (taken with $\xi _1 = \xi _2 = 1$, $\kappa _1 = \kappa _2 = \xi = 0$ if $K = R = 1$ and $\xi _1 = \kappa _1 = 1/2$, $\xi _2 = - \beta -1/2$, $\kappa _2 = 0$, $\xi = \beta + 1 < 1/2$ if $K=1$, $R=0$ and finally $\xi _1 = -\alpha - 1/2$, $\kappa _1 = 0$, $\xi _2 = \kappa _2 = 1/2$, $\xi = \alpha + 1 < 1/2$ if $K=0$, $R=1$), we infer that the expression in question is controlled by

$$\begin{aligned}&\frac{1}{|\theta - \varphi |^{\alpha + 1}} + \frac{ \varphi }{ (\theta + \varphi )^{2\alpha + 3} } \bigg ( \frac{ \theta + \varphi }{ |\theta - \varphi | } \bigg )^{1/2} + \frac{ \varphi }{ (\theta + \varphi )^{2\alpha + 2\beta + 4} } + \frac{ \varphi ^2 }{ (\theta + \varphi )^{2\alpha + 4} } \\&\quad \lesssim \frac{1}{|\theta - \varphi |^{\alpha + 1}} + \frac{ \varphi }{ (\theta + \varphi )^{2\alpha + 5/2} } \frac{ 1 }{ |\theta - \varphi |^{1/2} } , \quad \theta \in (0,\pi /4), \quad \varphi \in (0,\pi ), \quad \theta \ne \varphi ; \end{aligned}$$

here we used also (45). This, in view of Lemma 7.2 and Lemma 7.4 (applied with $\gamma = 2\alpha + 1$ and $\lambda = \nu = 0$, $\kappa = \alpha + 1$ or $\lambda = 2\alpha + 5/2$, $\nu = -2\alpha - 3/2$, $\kappa = 1/2$), gives the desired conclusion for $L_j(\theta ,\varphi )$, $j \ne 5,7$.

Next we show that $L_7(\theta ,\varphi )$ also produces a bounded operator from $L^1(d\mu _{\alpha ,\beta })$ to $L^1((0,\pi /4),d\mu _{\alpha ,\beta })$. Taking into account (57) and applying Lemma 7.6(a) (to the component of $L_7(\theta ,\varphi )$ connected with the first term in (57)) we obtain

$$\begin{aligned} |L_7(\theta ,\varphi )| \lesssim \bigg ( \frac{ \, \varphi ^2 }{(\theta + \varphi )^2} + 1 \bigg ) \int _0^1 t^2 \iint \frac{d\Pi _{-1\slash 2}(u)\, d\Pi _{-1\slash 2}(v)}{(t^2 + \mathfrak {q})^{\alpha +\beta +3 }} \, dt. \end{aligned}$$

Using now Lemma 5.3 (with $\nu = 5/2$, $\gamma = 2(\alpha + \beta + 3)$, $a=\sqrt{\mathfrak {q}}$, $B = 1$) and then integrating in u and v we get, uniformly in $\theta \in (0,\pi /4)$, $\varphi \in (0,\pi )$, $\theta \ne \varphi$,

$$\begin{aligned}&|L_7(\theta ,\varphi )| \lesssim \frac{1}{|\theta - \varphi |^{\alpha + 1}}+ \chi _{ \{ \alpha + \beta + 3/2 > 0 \} } \frac{1}{|\theta - \varphi |^{2\alpha + 2\beta + 3}}, \end{aligned}$$

which with the aid of Lemma 7.2 and Lemma 7.4 (specified to $\gamma = 2\alpha + 1$, $\lambda = \nu = 0$ and $\kappa = \alpha + 1$ or $\kappa = 2\alpha + 2\beta + 3$) leads to the desired property of $L_7(\theta ,\varphi )$.

Finally, we consider $L_5(\theta ,\varphi )$. We denote by $J(\theta ,\varphi )$ the component of $L_5(\theta ,\varphi )$ corresponding to the first term in (57) and by $J_{-1}(\theta ,\varphi )$ and $J_1(\theta ,\varphi )$ the remaining parts of $L_5(\theta ,\varphi )$ with integration in v restricted to $[-1,0]$ and [0, 1], respectively. We aim at showing that $J_{-1}(\theta ,\varphi )$ and $J(\theta ,\varphi )$, in contrast with $J_1(\theta ,\varphi )$, stand behind bounded operators from $L^1(d\mu _{\alpha ,\beta })$ to $L^1((0,\pi /4),d\mu _{\alpha ,\beta })$. The conclusion for $J_{-1}(\theta ,\varphi )$ is a straightforward consequence of (55). We pass to analyzing $J(\theta ,\varphi )$. Using sequently Lemma 7.6(a), (36), Lemma 5.3 (specified to $\nu = 5/2$, $\gamma = 2(\alpha + \beta + 4)$, $a=\sqrt{\mathfrak {q}}$, $B = 1$) and Lemma 7.3 (with $\xi _1 = -\alpha -1/2$, $\xi _2 = 1$, $\kappa _1 = \kappa _2 = 0$, $\xi = \alpha + 3/2 > 1/2$) we get

$$\begin{aligned} |J(\theta ,\varphi )|&\lesssim \frac{ \varphi ^2 }{(\theta + \varphi )^2} \int _0^1 t^2 \iint \frac{d\Pi _{-1\slash 2}(u)\, d\Pi _{\beta +1}(v)}{(t^2 + \mathfrak {q})^{\alpha +\beta + 4}} \, dt \\&\simeq \frac{ \varphi ^2 }{(\theta + \varphi )^2} \iint \frac{d\Pi _{-1\slash 2}(u)\, d\Pi _{\beta +1}(v)}{ \mathfrak {q}^{\alpha +\beta + 5/2}} \\&\lesssim \frac{ \varphi ^2 }{(\theta + \varphi )^2} \frac{ 1 }{|\theta - \varphi |^{2\alpha + 2}}, \quad \theta \in (0,\pi /4), \quad \varphi \in (0,\pi ), \quad \theta \ne \varphi . \end{aligned}$$

Now Lemmas 7.2 and 7.4 (with $\gamma = 2\alpha + 1$, $\lambda = 2$, $\nu = 0$ and $\kappa = 2\alpha + 2$) come into play, and the conclusion follows.

It remains to check that $J_1(\theta ,\varphi )$ gives rise to an operator that is not bounded from $L^1(d\mu _{\alpha ,\beta })$ to $L^1((0,\pi /4),d\mu _{\alpha ,\beta })$. Since the integrand related to $J_1(\theta ,\varphi )$ is non-positive, it suffices to prove that

$$\begin{aligned} \mathop {{{\mathrm{ess\,sup}}}}\limits _{\varphi \in (0,\pi /8)} \int _{0}^{\pi /4} |J_1(\theta ,\varphi )| \, \theta ^{2\alpha + 1} \, d\theta = \infty . \end{aligned}$$

(60)

Restricting the integration in v to the interval [1 / 2, 1] and applying (36) we obtain

$$\begin{aligned} | J_1(\theta ,\varphi ) | \gtrsim \int _0^1 t^2 \int _{[1/2,1]} \int \frac{ d\Pi _{-1\slash 2}(u)\, d\Pi _{\beta +1}(v)}{ (t^2 + \mathfrak {q})^{\alpha + \beta + 4}} \, dt, \quad \theta ,\varphi \in (0,\pi /4), \quad \theta \ne \varphi . \end{aligned}$$

Observe that the last expression is comparable to a similar one with integration in v over the whole interval $[-1,1]$. Using this, Lemma 5.3 (specified to $\nu = 5/2$, $\gamma = 2(\alpha + \beta + 4)$, $a=\sqrt{\mathfrak {q}}$, $B = 1$), Corollary 5.4 (applied to the integral against $d\Pi _{\beta +1}(v)$ with $\nu = \beta + 1$, $\gamma = \alpha + \beta + 5/2$, $A= 1 - u \sin \frac{\theta }{2}\sin \frac{\varphi }{2}\simeq 1$, $B = \cos \frac{\theta }{2}\cos \frac{\varphi }{2}$) and then integrating in u, we see that

$$\begin{aligned} | J_1(\theta ,\varphi ) |&\gtrsim \iint \frac{ d\Pi _{\beta +1}(v)\, d\Pi _{-1\slash 2}(u)}{ \mathfrak {q}^{\alpha + \beta + 5/2}} \\&\simeq \int \frac{ d\Pi _{-1\slash 2}(u)}{ (1 - \cos \frac{\theta }{2}\cos \frac{\varphi }{2}- u \sin \frac{\theta }{2}\sin \frac{\varphi }{2})^{\alpha + 1}} \simeq |\theta - \varphi |^{-2\alpha - 2}, \end{aligned}$$

for $\theta \in (0,\pi /4)$, $\varphi \in (0,\pi /8)$, $\theta \ne \varphi$. Consequently,

$$\begin{aligned} \int _0^{\pi /4} | J_1(\theta ,\varphi ) | \, \theta ^{2\alpha + 1} \, d\theta \gtrsim \int _{2\varphi }^{\pi /4} \theta ^{-1} \, d\theta = \log \frac{\pi }{8 \varphi }, \quad \varphi \in (0,\pi /8), \end{aligned}$$

which implies (60). This completes showing the case of $-1 < \alpha ,\beta < -1/2$ in Lemma 7.1.

The proof of Lemma 7.1 is at last finished. $\square$