Compact Hankel Operators Between Distinct Hardy Spaces and Commutators

Abstract

The paper is devoted to the study of compactness of Hankel operators acting between distinct Hardy spaces generated by Banach function lattices. We prove an analogue of Hartman’s theorem characterizing compact Hankel operators in terms of properties of their symbols. As a byproduct we give an estimation of the essential norm of such operators. Furthermore, compactness of commutators and semicommutators of Toeplitz operators for unbounded symbols is discussed.

1 Introduction

Let $H^2$ be the space of analytic and square-integrable functions on the unit circle $\mathbb {T}=\{e^{i\theta }:\theta \in [0,2\pi )\}$. For a measurable and bounded function $a$ on $\mathbb {T}$ (called a symbol), the Toeplitz operator $T_a$ is defined on $H^2$ as $T_af=P(af)$, $f\in H^2$. Here $P$ is the orthogonal projection from $L^2$ onto $H^2$ (i.e., the Riesz projection). Similarly, the Hankel operator $H_a$ is defined on $H^2$ by the formula $H_af = P(aJf)$, where $J$ is the flip operator.

The theory of Toeplitz and Hankel operators has been constantly developed since the beginning of the twentieth century and plays a pivotal role in functional and harmonic analysis as well as in the operator theory (see [3, 31]). Through intimate connection with Toeplitz and Hankel matrices, these operators are of great importance in many applications far beyond pure mathematics, for instance in signal and image processing, queueing theory or quantum mechanics.

It is worth mentioning that from the historical point of view, the main lines of research in the field of Toeplitz and Hankel operators are related to the case when a symbol of operators is bounded, that is the discussed operators act on (the same) Hardy space (mainly $H^2$). The case of Toeplitz and Hankel operators acting from one to another Hardy space (further we will refer to such a situation as nonalgebraic) was considered for the first time—up to our knowledge—in the papers [13, 35, 36]. Recently, this kind of study has been developed from a general perspective of Hardy spaces generated by Banach lattices in [20] (cf. [14]).

This paper is in a sense a follow-up to an article [20]. There, the first author studied Toeplitz and Hankel operators acting between distinct Hardy spaces. Among other things, he proved nonalgebraic versions of Brown–Halmos and Nehari’s theorems. The present work is devoted to the study of compact counterparts of the results obtained in [20]. Since there are no nontrivial compact Toeplitz operators even in the general approach (see [20, Section 4.2]), we focus on compactness of Hankel operators.

In the classical case of Hankel operators acting on $H^2$ their compactness has been completely described by Hartman [8]. Namely, $H_a$ is compact on $H^2$ if and only if $a\in C(\mathbb {T})+\overline{H^2}$. Exactly the same characterization holds for $H_a:H^p\rightarrow H^p$ in the case when $p\in (1,\infty )$, see [3, p. 80]. For $H_a:H^1\rightarrow H^1$ the situation is more involved. In [29] (cf. [13, 35]) the authors showed that the compactness of $H_a$ on $H^1$ is equivalent to $Pa\in VMOA _{{\text {log}}}$.

The main result of the paper is an analogue of the Hartman’s theorem, but fitted for nonalgebraic setting. Namely, we fully characterize compact operators among all bounded Hankel operators $H_a:H[X]\rightarrow H[Y]$, where $H[X],H[Y]$ are Hardy type spaces built upon r.i. spaces satisfying some general assumptions (actually, assumptions of the general Nehari theorem—see Sect. 4). The special cases of Hardy–Orlicz and Hardy–Lorentz spaces are also discussed.

Furthermore, although we consider an essentially nonalgebraic situation, that is Toeplitz and Hankel operators act between distinct spaces, we are able to study commutators $[T_a,T_b]$ and semicommutators $(T_a,T_b]$ of these operators (see Sect. 5). On one hand, such a nonalgebraic approach to an ideologically algebraic objects is not completely new in harmonic analysis. For example, commutators $[T,b]:L^p\rightarrow L^q$ of Calderón–Zygmund and multiplication operators have been investigated in the recent papers [11, 12] (see also references therein). On the other hand, the commutator $[T,b]$ considered in the mentioned papers is understood rather as a single operator acting directly from $L^p$ to $L^q$ without considering any intermediate space between $L^p$ and $L^q$. Our approach is different. Basing on Proposition 5.3 we find an intermediate space such that the respective component of the commutator $[T_a,T_b]$ factors through it. In consequence, we may treat components of $[T_a,T_b]$ as individual operators and thanks to that we are able to apply obtained results to give sufficient conditions for compactness of such commutator.

2 Preliminaries

In this part we present a necessary background for consideration in the following sections. When it comes to that we follow the classical monographs like [19, 23] or [2]. Nevertheless, for the convenience of the reader and due to the generality of the topic being considered, we recall the main definitions and notations below.

2.1 Quasi-Banach Function Spaces

Recall that $\mathbb {T}$ is the unit circle and $m$ is the normalized Lebesgue measure on $\mathbb {T}$, that is $d m(t)=(2\pi )^{-1}|d t|$. Let $L^0:=L^0(\mathbb {T})$ be the space of (equivalence classes of) all $m$-measurable complex-valued almost everywhere finite functions on $\mathbb {T}$.

A quasi-Banach space $X\subset L^0$ is called a quasi-Banach function space if it has the so-called ideal property (that is, $g\in X$ and $\Vert g\Vert _X \leqslant \Vert f\Vert _X$ for all $f\in X$ and $g\in L^0$ satisfying $|g| \leqslant |f|$ a.e. on $\mathbb {T}$) and $L^{\infty }\subset X$. If in addition $X$ is a Banach space, then we will call it a Banach function space (it is also common to refer to such a space a Köthe space or Banach function lattice).

A quasi-Banach function space $X$ has the Fatou property when given a sequence $(f_n)_{n\in \mathbb {N}}\subset X$ and $f\in L^0$ satisfying $0\le f_n\uparrow f$ a.e. as $n\rightarrow \infty$ and $\sup _{n\in \mathbb {N}}\Vert f_n\Vert _X<\infty$ we have $f\in X$ and $\Vert f\Vert _X = \sup \{\Vert f_n\Vert _X:n\in \mathbb {N}\}$. A quasi-Banach function space $X$ has the semi-Fatou property if for each $(f_n)_{n\in \mathbb {N}}\subset X$, $f\in X$ such that $0\le f_n\uparrow f$ a.e. as $n\rightarrow \infty$ there holds $\Vert f\Vert _X = \sup \{\Vert f_n\Vert _X:n\in \mathbb {N}\}$.

Recall that $f\in X$ is said to be an order continuous element if for each $(f_n)_{n\in \mathbb {N}}\subset X$ satisfying $0\le f_n\le |f|$ for all $n\in \mathbb {N}$ and $f_n\rightarrow 0$ a.e. as $n\rightarrow \infty$, there holds $\Vert f_n\Vert _X\rightarrow 0$ as $n\rightarrow \infty$. The subspace of order continuous elements of $X$ is denoted by ${X}_{o}$. Evidently, ${X}_{o}$ has the semi-Fatou property. We say that $X$ is order continuous, when $X={X}_{o}$. Notice that this is equivalent to the separability of $X$.

For a quasi-Banach function space $X$, its Köthe dual $X'$ is defined as the space of functions $g\in L^0$ satisfying $fg\in L^1$ for all $f\in X$ and equipped with a quasi-norm $\Vert g\Vert _{X'}=\sup \{\Vert fg\Vert _{L^1}:\Vert f\Vert _X\leqslant 1\}$. Notice that $X'$ is nontrivial and possesses the Fatou property if $X$ is a Banach function space. Note also that $X'$ may be trivial, that is $X'=\{0\}$, when $X$ is just a quasi-Banach function space. For example, $(L^p)'=\{0\}$ when $p\in (0,1)$. It is known that a Banach function space $X$ has the Fatou property if and only if $X''\equiv X$ (see [23, p. 30]).

We will work within an important class of quasi-Banach function spaces, that is with rearrangement invariant spaces. Recall that for $f\in L^0$ it’s distribution $\mu _f$ is given by $\mu _f(\lambda )=m\bigl \{t\in \mathbb {T}:|f(t)|>\lambda \bigr \},\ \lambda \geqslant 0$. Two functions $f,g\in L^0$ are said to be equimeasurable if $\mu _f(\lambda )=\mu _g(\lambda )$ for all $\lambda \geqslant 0$. The non-increasing rearrangement ${f}^{*}$ of $f\in L^0$ is defined by

$$\begin{aligned} {f}^{*}(t)=\inf \{\lambda :\mu _f(\lambda )\le t\},\quad t\in [0,1]. \end{aligned}$$

A quasi-Banach function space $X$ is called rearrangement invariant (r.i. for short) if for every pair of equimeasurable functions $f,g \in L^0$, $f\in X$ implies that $g\in X$ and $\Vert f\Vert _X=\Vert g\Vert _X$. We point out that many important examples of quasi-Banach function spaces are rearrangement-invariant, for example Lebesgue, Orlicz and Lorentz spaces. We refer the reader to [19, 23] and [2] for more information on non-increasing rearrangements and r.i. spaces.

Let $X$ be a r.i. quasi-Banach function space. For each $s\in \mathbb {R}_+$ the dilation operator $D_s$ is defined by

$$\begin{aligned} (D_s f)(e^{i\theta })= {\left\{ \begin{array}{ll} f(e^{i\theta s}), &{} \theta s\in [0,2\pi ),\\ 0, &{}\theta s\not \in [0,2\pi ), \end{array}\right. } \quad \theta \in [0,2\pi ). \end{aligned}$$

It is well-known (see, for example, [19] for the case of Banach spaces, or [5, 28] for the case of quasi-Banach spaces on $\mathbb {R}$ and $\mathbb {R}_+$, respectively) that $D_s$ is bounded on $X$ for each $s>0$ and the following limits exist

$$\begin{aligned} \alpha _X=\lim _{s\rightarrow 0^+}\frac{\log \Vert D_{1/s}\Vert _{X\rightarrow X}}{\log s}, \quad \beta _X=\lim _{s\rightarrow \infty }\frac{\log \Vert D_{1/s}\Vert _{X\rightarrow X}}{\log s}. \end{aligned}$$

The numbers $\alpha _X$ and $\beta _X$ are called the lower and the upper Boyd index of $X$, respectively. For an arbitrary r.i. Banach function space $X$, its Boyd indices belong to $[0,1]$ and $\alpha _X\le \beta _X$. We say that Boyd indices are nontrivial, if $\alpha _X,\beta _X\in (0,1)$.

2.2 Pointwise Multipliers and Products

Let $X$ and $Y$ be Banach function spaces. The space of pointwise multipliers $M(X,Y)$ from $X$ to $Y$ is defined as

$$\begin{aligned} M(X,Y)=\bigl \{ f\in L^0 :fg\in Y\text { for all } g\in X \bigr \}, \end{aligned}$$

equipped with the operator norm

$$\begin{aligned} \Vert f\Vert _{M(X,Y)}=\sup \bigl \{ \Vert fg\Vert _Y:\Vert g\Vert _X\le 1\bigr \}. \end{aligned}$$

It is clear, that each $f\in M(X,Y)$ corresponds to a bounded multiplication operator $M_f:X\rightarrow Y$ defined by $M_f:g\mapsto fg$ and there holds $\Vert f\Vert _{M(X,Y)}=\Vert M_f\Vert _{X\rightarrow Y}$. Function f defining the operator $M_f$ will be called its symbol.

Let us collect some known facts about spaces of pointwise multipliers.

• If $1\leqslant p<q\leqslant \infty$, then $M(L^p,L^q)=\{0\}$ (see [25, Theorem 2]), while for $1\leqslant q\leqslant p\leqslant \infty$ there holds $M(L^p,L^q)\equiv L^r$, where $1/r=1/q-1/p$, (see [25, Proposition 3]).

• For an arbitrary Banach function space $X$, $M(X,X)\equiv L^\infty$ (see [25, Theorem 3]).

• For two rearrangement invariant Banach function spaces $X$ and $Y$, $M(X,Y)\not =\{0\}$ if and only if $X\subset Y$ (see [25, the remark after Corollary 1], or [16, Theorem 2.2 (i)]).

• If $X$ and $Y$ are rearrangement invariant Banach function spaces and $X\subset Y$, then $M(X,Y)$ is also rearrangement invariant (see  [16, Theorem 2.2]).

The notion of multipliers is associated with a concept of products of spaces. Given two quasi-Banach spaces $X$ and $Y$, the pointwise product $X\odot Y$ is defined as

$$\begin{aligned} X\odot Y=\bigl \{gh:g\in X,\ h\in Y\bigr \}. \end{aligned}$$

It should be noted that even if $X,Y$ are Banach function spaces, then $X\odot Y$ need not to be a Banach space. Indeed, the product space is, in general, a quasi-Banach space when equipped with the quasi-norm

$$\begin{aligned} \Vert f\Vert _{X\odot Y} = \inf \bigl \{\Vert g\Vert _X\,\Vert h\Vert _Y :f=gh,\ g\in X,\ h\in Y\bigr \}. \end{aligned}$$

For example, if $p,q\in (0,\infty ]$, then $L^p\odot L^q\equiv L^r$, where $\frac{1}{r}=\frac{1}{p}+\frac{1}{q}$. For further reading we refer to the papers [17, 34] and references included therein.

3 Hardy Spaces, Toeplitz and Hankel Operators

For $n\in \mathbb {Z}$ and $t\in \mathbb {T}$, let $\chi _n(t):=t^n$. The Fourier coefficients of a function $f\in L^1$ are given by

$$\begin{aligned} \widehat{f}(n) = \langle f,\chi _n\rangle := \int _\mathbb {T}f(t) t^{-n}\,d m(t), \quad n\in \mathbb {Z}. \end{aligned}$$

Let $X$ be a quasi-Banach function space such that $X\subset L^1$ (note that $X\subset L^1$ for each r.i. Banach function space). The Hardy space $H[X]$ is defined by

$$\begin{aligned} H[X]=\bigl \{f\in X:\widehat{f}(n)=0 \text { for all } n<0\bigr \}, \end{aligned}$$

with the quasi-norm inherited from $X$ (see for example [14, 26, 27, 38]). If $p\in [1,\infty ]$, then $H^p:=H[L^p]$ is the classical Hardy space (see for example [6, 9, 15]). We shall also use the following variants of Hardy spaces:

$$\begin{aligned} \overline{H[X]}=\bigl \{\overline{f}:f\in H[X]\bigr \}. \end{aligned}$$

The Riesz projection $P$ is for $f\in L^1$ given by the formula

$$\begin{aligned} P:f\mapsto \frac{1}{2}\bigl (f+\widetilde{f}+\widehat{f}(0)\bigr ), \end{aligned}$$

where $\widetilde{f}$ is the conjugate function to $f$ (see for example [15, Chapter III] for the definition of $\widetilde{f}$). We recall that if a quasi-Banach function space $X$ has nontrivial Boyd indices, then from the Boyd interpolation theorem it follows that $P:X\rightarrow X$ is a bounded operator (see [5, 23]). A little more can be said in the case of r.i. Banach function spaces. Indeed, in [7] it was proved that $P:X\rightarrow X$ is bounded if and only if $X$ has nontrivial Boyd indices. In such the case, the following equivalent formula for P is meaningful

$$\begin{aligned} P:f\mapsto \sum _{n\geqslant 0}\hat{f}(n)\chi _n,\quad f\in X. \end{aligned}$$

Below we state definitions of Toeplitz and Hankel operators. Roughly speaking we follow [20], however, notice that we allow also quasi-Banach spaces, which will be used as a tool in the last section. Let $X$ and $Y$ be r.i. quasi-Banach function spaces such that $X\subset Y\subset L^1$ and assume that $P$ is bounded on $Y$. For $a\in M(X,Y)$, the Toeplitz operator $T_a$ is defined by the formula

$$\begin{aligned} T_a f = P(af),\quad f\in H[X]. \end{aligned}$$

Analogously, the Hankel operator is given by

$$\begin{aligned} H_a f = P(aJf),\quad f\in H[X],\end{aligned}$$

where J is the so-called flip operator, that is

$$\begin{aligned} Jf(t)=t^{-1}f\bigl (t^{-1}\bigr ),~~t\in \mathbb {T}. \end{aligned}$$

It is well known that a Toeplitz operator $T_a:H[X]\rightarrow H[Y]$ admits the matrix representations with a Toeplitz matrix, that is $\langle T_a\chi _j,\chi _k\rangle =a(k-j)$ for all $k,j\geqslant 0$. The converse statement is known as the Brown–Halmos theorem and holds for a wide class of spaces (see for example [3], or [14, 20] for nonalgebraic setting). An analogous role, but for Hankel operators is played by the Nehari theorem. We will need in the sequel the following general version of it (see [20, Theorem 5.2]).

Theorem 3.1

(The general Nehari theorem) Let $X,Y$ be two r.i. Banach function spaces, such that $X$ is separable, $X\subset Y$, $Y$ has nontrivial Boyd indices and one of the following conditions holds:

• (i) $X\odot M(X,Y)=Y$ and $X,Y$ have the Fatou property,

• (ii) $\beta _X<\alpha _Y$ and $Y$ has the semi-Fatou property.

Then a continuous linear operator $A:H[X]\rightarrow H[Y]$ satisfies

$$\begin{aligned} \langle A\chi _j,\chi _k\rangle =a_{k+j+1},\quad j,k\geqslant 0, \end{aligned}$$

for some sequence $(a_k)_{k>0}$ if and only if there exists $a\in M(X,Y)$ such that $\widehat{a}(n)=a_n$ for each $n>0$ and $A=H_a$, i.e., $A:f\mapsto P(aJf)$. Moreover,

$$\begin{aligned} c\, {\mathrm {dist}}_{M(X,Y)}\bigl (a,\overline{ H[M(X,Y)]}\bigr )&\leqslant \Vert H_a\Vert _{H[X]\rightarrow H[Y]}\\ {}&\leqslant \Vert P\Vert _{Y\rightarrow Y}{\mathrm {dist}}_{M(X,Y)}\bigl (a,\overline{ H[M(X,Y)]}\bigr ), \end{aligned}$$

where the constant $c>0$ depends only on the spaces $X,Y$.

4 Compact Hankel Operators

Before we state the main result we need to prove an analogue of Lemma 5.4 from [31]. Recall that a Banach space $Y$ has the approximation property, if for every Banach space $X$, the set of finite rank operators from $X$ into $Y$ is dense in the subspace $K(X,Y)$ of compact operators from $X$ into $Y$ (cf. [23, Theorem 1.e.4, p. 32]).

In what follows $S:X\rightarrow X$ denotes the shift operator given by

$$\begin{aligned} Sf(t)=tf(t), \ \ t\in \mathbb {T}. \end{aligned}$$

Lemma 4.1

Let $X$ and $Y$ be r.i. Banach function spaces such that $Y$ has nontrivial Boyd indices and assume that $X$ and $X'$ are order continuous. If $K:H[X]\rightarrow H[Y]$ is compact, then

$$\begin{aligned} \Vert KS^n\Vert \rightarrow 0\quad \text {as } n\rightarrow \infty . \end{aligned}$$

Proof

Notice that from [32, Theorem 4.6] (cf. [33]) it follows that every r.i. Banach function space has the approximation property. Since $Y$ has nontrivial Boyd indices, then by the boundedness of the Riesz projection, $H[Y]$ is complemented in $Y$ and thus also $H[Y]$ has the approximation property. In consequence, any compact operator $K:H[X]\rightarrow H[Y]$ may be approximated in the norm topology by finite rank operators.

Let $\varepsilon >0$ be arbitrary and $K:H[X]\rightarrow H[Y]$ be compact. Then there exists a finite rank operator $F$ such that $\Vert K-F\Vert \leqslant \frac{\varepsilon }{4}$. Representing $F$ in a canonical form there exist (not necessarily unique) functions $\varphi _k\in X'$ (since $H[X]^*\simeq X'/ H[X]^{\perp }$) and $\eta _k\in H[Y]$, where $k\in \{1,\dots ,m\}$ and $m>0$, such that

$$\begin{aligned} \biggl \Vert K-\sum _{k=1}^m \eta _k\otimes \varphi _k \biggr \Vert \leqslant \frac{\varepsilon }{4}. \end{aligned}$$

Since $X'$ is order continuous, then polynomials are dense in $X'$. Thus there exist polynomials $p_k\in X'$ such that

$$\begin{aligned} \biggl \Vert K-\sum _{k=1}^m \eta _k\otimes p_k \biggr \Vert \leqslant \frac{\varepsilon }{2}. \end{aligned}$$

Further observe that

$$\begin{aligned} \Vert KS^n\Vert&\leqslant \biggl \Vert KS^n-\biggl (\sum _{k=1}^m \eta _k\otimes p_k\biggr )S^n\biggr \Vert + \biggl \Vert \biggl (\sum _{k=1}^m \eta _k\otimes p_k\biggr )S^n\biggr \Vert \\&\leqslant \biggl \Vert K-\biggl (\sum _{k=1}^m \eta _k\otimes p_k\biggr )\biggr \Vert \,\bigl \Vert S^n\bigr \Vert +\biggl \Vert \biggl (\sum _{k=1}^m \eta _k\otimes p_k\biggr )S^n\biggr \Vert . \end{aligned}$$

Notice that for a fixed $k$ and arbitrary $f\in H[X]$ we have

$$\begin{aligned} (\eta _k\otimes p_k)S^nf=\langle S^nf,p_k\rangle \eta _k=0, \end{aligned}$$

whenever $n>\deg p_k$. As a consequence, for $n>\max \{\deg p_k:k\in \{1,\dots ,m\}\}$ there holds

$$\begin{aligned} \Vert KS^n\Vert \leqslant \varepsilon \end{aligned}$$

and the proof is finished. $\square$

Recall that the essential norm $\Vert T\Vert _{{\mathrm{ess}}}$ of an operator $T\in L(X,Y)$, where $X,Y$ are Banach spaces, is the distance of $T$ from the subspace of compact operators $K(X,Y)$, that is

$$\begin{aligned} \Vert T\Vert _{{\mathrm{ess}}} = \inf \bigl \{ \Vert T-K\Vert :K\in K(X,Y) \bigr \}. \end{aligned}$$

In 1958, Hartman [8] characterized the compactness of Hankel operators $H_a:H^2\rightarrow H^2$ and showed that $H_a$ is compact if and only if there exists a continuous function $g$ on $\mathbb {T}$ such that $\widehat{g}(n) = a_n$ for $n\in \mathbb {N}$. Using the notion of the essential norm even more can be said. The following result attributed to Hartman, Adamyan, Arov and Krein, contains estimates on the essential norm of Hankel operators on $H^p$ spaces.

Theorem 4.2

([3, Theorem 2.54]) Let $a\in L^\infty$ and $p\in (1,\infty )$. Then

$$\begin{aligned} {\mathrm{dist}}_{L^\infty } \bigl (a, C + \overline{H^\infty }) \leqslant \Vert H(a)\Vert _{{\mathrm{ess}}} \leqslant c_p {\mathrm{dist}}_{L ^\infty } \bigl (a, C + \overline{H^\infty }\bigr ), \end{aligned}$$

where $c_p$ is the norm of the Riesz projection $P:L^p\rightarrow H^p$.

Here the space $C+ \overline{H^\infty }$ is a closed subalgebra of $L^\infty$ (this statement is usually referred to as Sarason’s theorem) consisting of functions $\phi \in L^\infty$ such that $\phi$ admits a representation $\phi = g + f$, where $g\in C$ and $f\in \overline{H^\infty }$ (see [31, p. 25]).

Our description of compact Hankel operators will be of similar fashion, but $C$ will be replaced by $M (X,Y)_{o}$, while $M(X,Y)$ and $\overline{H[M(X,Y)]}$ will play the role of $L^\infty$ and $H^\infty$, respectively. At first, we need, however, one more auxiliary result, i.e., an analogue of Sarason’s theorem.

Theorem 4.3

Let $X$ be r.i. Banach function spaces with the Fatou property. Then ${X}_{o}+\overline{H[X]}$ is a closed subspace of $X$. In particular,

$$\begin{aligned} {X}_{o}+\overline{H[X]}={\mathrm{clos}}_X\biggl \{\bigcup _{n>0}\chi _n\overline{H[X]}\biggr \}. \end{aligned}$$

Proof

First notice that if $X={X}_{o}$ or $X$ has nontrivial Boyd indices, then there is nothing to prove. Suppose that $X\not ={X}_{o}$. We will follow ideas of [31, Theorem 5.1]. At first we will show that for each $f\in {X}_{o}$

$$\begin{aligned} {\mathrm{dist}}_{X}\bigl (f, \overline{H[X]}\bigr )={\mathrm{dist}}_{X}\bigl (f, \overline{H[{X}_{o}]}\bigr ). \end{aligned}$$

(4.1)

We need to prove only the inequality ${\mathrm{dist}}_{X}(f, \overline{H[X]})\geqslant {\mathrm{dist}}_{X}(f, \overline{H[{X}_{o}]})$. Denote for $f\in L^1$ and $r\in (0,1)$

$$\begin{aligned} f_r(t)=f*P_r(t),\quad t\in \mathbb {T}, \end{aligned}$$

where $P_r$ is the Poisson kernel for the unit disc $\mathbb {D}$ (see [6, 9, 15]). Let now $f\in {X}_{o}$ and $g\in \overline{H[X]}$. Then

$$\begin{aligned} \bigl \Vert (f-g)_r\bigr \Vert _X=\bigl \Vert (f-g)*P_r\bigr \Vert _X\leqslant \Vert f-g\Vert _X, \end{aligned}$$

where the last inequality follows from the Calderón–Mitjagin theorem (cf. [2]) and the fact that $\Vert P_r\Vert _{H^1}=1$ for each $r\in (0,1)$. As a consequence,

$$\begin{aligned} \bigl \Vert f-g\bigr \Vert _X\geqslant \lim _{r\rightarrow 1^-}\bigl \Vert (f-g)_r\bigr \Vert _X\geqslant \lim _{r\rightarrow 1^-}\bigl (\Vert f-g_r\Vert _X-\Vert f-f_r\Vert _X\bigr ). \end{aligned}$$

However, $f,f_r\in {X}_{o}$ and so $\Vert f-f_r\Vert _{{X}_{o}}=\Vert f-f_r\Vert _X\rightarrow 0$ as $r\rightarrow \infty$, because $(P_r)$ is the approximation kernel and ${X}_{o}$ is a homogeneous Banach space in the sense of [15, Theorem 2.11]. Hence,

$$\begin{aligned} \bigl \Vert f-g\bigr \Vert _X\geqslant \lim _{r\rightarrow 1^-}\Vert f-g_r\Vert _X\geqslant {\mathrm{dist}}_{X}(f,{X}_{o}), \end{aligned}$$

since also $g_r\in {X}_{o}$ as a continuous function. Thus the equality (4.1) is proved.

The proved claim means that ${X}_{o}/\overline{H[{X}_{o}]}$ may be regarded as a closed subspace of $X/\overline{H[X]}$ since the natural embedding is isometric by the formula (4.1). Then the argument is exactly the same as in [31, Theorem 5.1] with $C$, $C_A$, $L^{\infty }$, and $H^{\infty }$ replaced by ${X}_{o}$, $\overline{H[{X}_{o}]}$, $X$, and $\overline{H[X]}$, respectively. $\square$

The following theorem describes compactness of Hankel operators in a general nonalgebraic setting. Notice that the point (i) in case of Hardy spaces $H[X]=H^p,H[Y]=H^q$, $1<q<p$, trivially holds, which was already known to Tolokonnikov [35].

Theorem 4.4

Let $X$ and $Y$ be r.i. Banach function spaces such that $L^{\infty }\varsubsetneq M(X,Y)$ and assume that $Y$ has nontrivial Boyd indices.

• (i) For each $a\in M(X,Y)$ and the Hankel operator $H_a:H[X]\rightarrow H[Y]$ there holds

  $$\begin{aligned} \Vert H_{a}\Vert _{{\mathrm{ess}}}\leqslant \Vert P\Vert _{X\rightarrow Y}{\mathrm{dist}}_{M(X,Y)}\bigl (a,{M(X,Y)}_{o}+\overline{H[M(X,Y)]}\bigr ). \end{aligned}$$

• (ii) If in addition spaces $X$ and $Y$ satisfy assumptions of Theorem 3.1, Y has the Fatou property and $X,X'$ are order continuous, then also

  $$\begin{aligned} \Vert H_{a}\Vert _{{\mathrm {ess}}}\geqslant c\, {\mathrm {dist}}_{M(X,Y)}\bigl (a,{M(X,Y)}_{o}+\overline{H[M(X,Y)]}\bigr ), \end{aligned}$$

  for some constant $c>0$ depending only on spaces $X$ and $Y$.

Proof

(i). At first assume that $a\in {M(X,Y)}_{o}$. Thanks to the assumption $L^{\infty }\varsubsetneq M(X,Y)$, the set of all polynomials is included in ${M(X,Y)}_{o}$ and is dense therein (see [20, Lemma 3.1 (a) and Lemma 3.4]). Note also that any Hankel operator induced by a polynomial has a finite rank (cf. Kronecker’s theorem [31]). Let $(p_n)$ be a sequence of polynomials such that $\Vert a-p_n\Vert _{M(X,Y)}\rightarrow 0$ as $n\rightarrow \infty$. By the definition of the Hankel operator we have

$$\begin{aligned} \Vert H_a-H_{p_n}\Vert = \Vert H_{a-p_n}\Vert \leqslant \Vert P\Vert _{Y\rightarrow Y}\Vert a-p_n\Vert _{M(X,Y)}\rightarrow 0\quad \text {as } n\rightarrow \infty . \end{aligned}$$

Hence $H_a$ is compact.

Assume now that $a\in M(X,Y)$. Then evidently

$$\begin{aligned} \Vert H_a\Vert _{{\mathrm{ess}}}\leqslant \inf \bigl \{ \Vert H_a-H_b\Vert :b\in {M(X,Y)}_{o}\bigr \}. \end{aligned}$$

Further we have

$$\begin{aligned}&\inf \{\Vert H_a-H_b\Vert :b\in {M(X,Y)}_{o}\bigr \} = \inf \bigl \{\Vert H_{a-b}\Vert :b\in {M(X,Y)}_{o}\bigr \} \\&\quad \leqslant \Vert P\Vert _{Y\rightarrow Y}\inf \bigl \{ {\mathrm{dist}}_{M(X,Y)}(a-b,\overline{H[M(X,Y)]}):b\in {M(X,Y)}_{o}\bigr \}. \end{aligned}$$

where the inequality comes from the definition and properties of Hankel operator (cf. [20, Theorem 5.2]). Besides, we get

$$\begin{aligned}&\inf \bigl \{ {\mathrm{dist}}_{M(X,Y)}(a-b,\overline{H[M(X,Y)]}):b\in {M(X,Y)}_{o}\bigr \} \\&\quad = \inf \bigl \{\Vert a-b-d\Vert _{M(X,Y)} :b\in {M(X,Y)}_{o},\ d\in \overline{H[M(X,Y)]}\bigr \} \\&\quad = \inf \bigl \{ \Vert a-f\Vert _{M(X,Y)}:f\in {M(X,Y)}_{o}+\overline{H[M(X,Y)]}\bigr \} \\&\quad = {\mathrm{dist}}_{M(X,Y)}\bigl (a,{M(X,Y)}_{o}+\overline{H[M(X,Y)])}\bigr ). \end{aligned}$$

Eventually, the calculations above lead to the conclusion that

$$\begin{aligned} \Vert H_{a}\Vert _{{\mathrm{ess}}}\leqslant \Vert P\Vert _{Y\rightarrow H[Y]}{\mathrm{dist}}_{M(X,Y)}\bigl (a,{M(X,Y)}_{o}+\overline{H[M(X,Y)])}\bigr ). \end{aligned}$$

(ii). Let $a\in M(X,Y)$ and assume that $K:H[X]\rightarrow H[Y]$ is compact. From Theorem 3.1 it follows

$$\begin{aligned} \Vert H_a-K\Vert&= \Vert S^n\Vert \,\Vert H_a-K\Vert \geqslant \Vert H_aS^n-KS^n\Vert \geqslant \Vert H_aS^n\Vert -\Vert KS^n\Vert \\ {}&\geqslant c\, {\mathrm {dist}}_{M(X,Y)}\bigl (\overline{\chi _n}a,\overline{H[M(X,Y)])}\bigr )-\Vert KS^n\Vert . \end{aligned}$$

From Lemma 4.1 we know that $\Vert KS^n\Vert \rightarrow 0$ as $n\rightarrow \infty$. Thus, to conclude the proof it suffices to see that

$$\begin{aligned} {\mathrm{dist}}_{M(X,Y)}\bigl (\overline{\chi _n}a,\overline{H[M(X,Y)])}\bigr )\rightarrow {\mathrm{dist}}_{M(X,Y)}\bigl (a,{M(X,Y)}_{o}+\overline{H[M(X,Y)])}\bigr ) \end{aligned}$$

as $n\rightarrow \infty$. But this statement easily follows from the observation that

$$\begin{aligned} {\mathrm{dist}}_{M(X,Y)}\bigl (\overline{\chi _n}a,\overline{H[M(X,Y)])}\bigr )= {\mathrm{dist}}_{M(X,Y)}\bigl (a,\chi _n\overline{H[M(X,Y)]}\bigr ) \end{aligned}$$

and Theorem 4.3. $\square$

Before we proceed with further results let us comment on assumptions of the above theorem and discuss some special cases excluded from it. First of all, we have assumed $L^{\infty }\varsubsetneq M(X,Y)$. Nevertheless, this assumption is essentially not necessary to prove the point (i). The reason to assume it is that in case $L^{\infty }= M(X,Y)$ we can not use the notion of $M (X,Y)_{o}=\{0\}$. However, in such circumstances the role of $M (X,Y)_{o}$ is played by the space of continuous functions $C=C(\mathbb {T})$. Thus, when $X\subset Y$, but $M(X,Y)=L^{\infty }$, point (i) holds (with exactly the same proof, under the assumption that $Y$ has nontrivial Boyd indices) and takes the form

$$\begin{aligned} \Vert H(a)\Vert _{{\mathrm{ess}}} \leqslant \Vert P\Vert _{X\rightarrow Y} {\mathrm{dist}}_{L ^\infty } \bigl (a, C + \overline{H^\infty }\bigr ), \end{aligned}$$

for each $a\in L^{\infty }$, where $H_a:H[X]\rightarrow H[Y]$. In particular, $H_a$ is compact, when $a\in C+ \overline{H^\infty }$.

For the point (ii) much more is needed. Namely, we have two possibilities. If the assumption of Theorem 3.1(ii) is satisfied, it implies that $L^{\infty }\varsubsetneq M(X,Y)$. Notice however that the assumption from Theorem 3.1(ii) is much stronger than $L^{\infty }\varsubsetneq M(X,Y)$. On the other hand, the assumption of Theorem 3.1(i) is satisfied when $X=Y$ and X has the Fatou property. In this case Theorem 4.4 has the following form.

Theorem 4.5

Let $X$ be a reflexive r.i. Banach function space with nontrivial Boyd indices. For each $a\in L^{\infty }$, $H_a:H[X]\rightarrow H[X]$ is bounded and

$$\begin{aligned} c\, {\mathrm {dist}}_{L ^\infty } \bigl (a, C + \overline{H^\infty }\bigr )\leqslant \Vert H(a)\Vert _{{\mathrm {ess}}} \leqslant \Vert P\Vert _{X\rightarrow X}{\mathrm {dist}}_{L ^\infty } \bigl (a, C + \overline{H^\infty }\bigr ), \end{aligned}$$

where constant $c>0$ depends only on the space $X$.

Proof

As was mentioned, in case $X=Y$ assumptions of Theorem 4.4 are satisfied and $L^{\infty }= M(X,Y)$. Therefore, it is enough to repeat the proof of Theorem 4.4 with $M (X,Y)_{o}$ replaced by $C$. $\square$

Finally, notice that it may happen that $L^{\infty }= M(X,Y)$ even for $X\varsubsetneq Y$ (for example, $M(L^{p,q},L^{p,r})=L^{\infty }$ when $p\in [1,\infty )$ and $1\leqslant q<r\leqslant \infty$, see the definition of the Lorentz space $L^{p,q}$ below). In such a case, however, neither assumption (i), nor (ii) of Theorem 3.1 is satisfied, and thus estimation from Theorem 4.4(ii) is problematic—with current knowledge we can not decide whether it holds.

Let us state one more conclusion of the above Theorem 4.4.

Corollary 4.6

Let $X$ and $Y$, $X\subset Y$ be r.i. Banach function spaces such that $Y$ has nontrivial Boyd indices. If $M(X,Y)$ is separable, then for each $a\in M(X,Y)$ the Hankel operator $H_a:H[X]\rightarrow H[Y]$ is compact.

Proof

If $M(X,Y)$ is separable, then $M(X,Y)={M(X,Y)}_{o}$ and by the fact that $\overline{H[M(X,Y)]}$ is the subspace of $M(X,Y)$, we have

$$\begin{aligned} {\mathrm{dist}}_{M(X,Y)}\bigl (a,{M(X,Y)}_{o}+\overline{H[M(X,Y)])}\bigr ) ={\mathrm{dist}}_{M(X,Y)}\bigl (a,M(X,Y)\bigr )=0. \end{aligned}$$

Thus Theorem 4.4(i) implies the claim. $\square$

From Theorem 4.4 the following results on compact Hankel operators between special classes of Hardy type spaces follow. Let us recall that for $p\in (0,\infty )$ and $q\in (0,\infty ]$ the Lorentz space $L^{p,q}$ is defined by the (quasi-) norm

$$\begin{aligned} \Vert f\Vert _{L^{p,q}}=\biggl (\int _{0}^1 \bigl [{f}^{*}(s)s^{1/p}\bigr ]^q\tfrac{d s}{s}\biggr )^{1/q}\!\!, \end{aligned}$$

with the standard modification when $q=\infty$. A function $\varphi :[0,\infty )\rightarrow [0,\infty ]$ which is convex, nondecreasing and $\varphi (0)=0$ is called the Young function, or the Orlicz function, when additionally $\varphi :[0,\infty )\rightarrow [0,\infty )$. The Orlicz space $L^{\varphi }$ is defined by the norm

$$\begin{aligned} \Vert f\Vert _{\varphi }=\inf \biggl \{\lambda >0:\int _{\mathbb {T}}\varphi \bigl (|f(t)|/\lambda \bigr )\,d m(t)\leqslant 1\biggr \}. \end{aligned}$$

Then the Hardy–Lorentz space $H^{p,q}$ is defined as $H^{p,q}:=H[L^{p,q}]$, for $p,q\geqslant 1$, while the Hardy–Orlicz space $H^\varphi$ is given by $H^{\varphi }=H[L^{\varphi }]$.

Corollary 4.7

• (i) If $1<q<p<\infty$, then each Hankel operator $H_a:H^p\rightarrow H^q$ is compact (see [35]).

• (ii) Let $1<p_2<p_1<\infty$.

  • (a) If $1<q_1<q_2<\infty$, then any Hankel operator $H_a:H^{p_1,q_1}\rightarrow H^{p_2,q_2}$ between Hardy–Lorentz spaces is compact.

  • (b) If $1<q_2\leqslant q_1<\infty$, then the Hankel operator $H_a:H^{p_1,q_1}\rightarrow H^{p_2,q_2}$ between Hardy–Lorentz spaces is compact if and only if $Pa\in H[{L^{p,\infty }_{o}}]$ where $\frac{1}{p}=\frac{1}{p_2}-\frac{1}{p_1}$.

• (iii) Let $\varphi _1,\varphi$ be Orlicz functions such that $L^{\varphi }$ has nontrivial Boyd indices, i.e., $1<a_{\varphi }\leqslant b_{\varphi }<\infty$, where $a_{\varphi },b_{\varphi }$ are Matuszewska–Orlicz indices of $\varphi$ (cf. [20]). Let $\varphi \ominus \varphi _1$ be a Young function given by $\varphi \ominus \varphi _1 (t) = \sup \{ \varphi (st) - \varphi _1(s):s>0\}$ (see [22]) and assume that either $\varphi ^{-1}\approx \varphi _1^{-1}(\varphi \ominus \varphi _1)^{-1}$, or $b_{\varphi }<a_{\varphi _1}$. If $a\in L^{\varphi \ominus \varphi _1}$, then the Hankel operator $H_a:H^{\varphi _1}\rightarrow H^{\varphi }$ is compact if and only if $a\in {L^{\varphi \ominus \varphi _1}_{o}}+\overline{H^{\varphi \ominus \varphi _1}}$.

5 Commutators and Semicommutators of Toeplitz and Hankel Operators

For two operators $T,S:X\rightarrow X$ their commutator is defined as

$$\begin{aligned} {[}T,S]=TS-ST. \end{aligned}$$

The commutator inspects to what extent operators $T$ and $S$ fail to be commutative, which constitutes an important question in the field of operator algebras.

Seemingly, the definition of commutator requires that both operators belong to the same operator algebra and in general can not be directly extended to nonalgebraic settings. Nonetheless, it is quite interesting that we may define commutators and semicommutators for Toeplitz and Hankel operators acting between distinct Hardy spaces.

The following example serves as an inspiration for considerations in this part of the paper.

Example 5.1

Let $1<q<r<p<\infty$ and suppose that $a\in M(L^p,L^r)=L^{s_1}$ and $b\in M(L^r,L^q)=L^{s_2}$, where $1/s_1=1/r-1/p$ and $1/s_2=1/q-1/r$. Observe that if $s$ satisfies $\frac{1}{s} = \frac{1}{p}+\frac{1}{q}-\frac{1}{r}$, then $a\in M(L^{s},L^q)$ and $b\in M(L^p,L^{s})$. Moreover, $ab\in M(L^p,L^q)$. On the diagram it looks like

Thus all three operators $T_aT_b$, $T_bT_a$ and $T_{ba}$ map $H^p$ into $H^q$. Therefore, commutators $[T_a,T_b]:=T_aT_b-T_bT_a$ and semicommutators $(T_b,T_a] :=T_bT_a-T_{ba}$ make sense. Note also that essentially the same reasoning applies to Hankel operators and their commutators and semicommutators.

We will study when commutators and semicommutators of Toeplitz operators are compact. For this to be done we need the following basic relations, which are in fact the same as in the classical (that is, algebraic) settings (see [3, 2.14 Proposition, p. 57]). However, we will present the proof for the sake of convenience, since, on one hand, the notion of Hankel operators varies through the literature and, on the other hand, sometimes proofs of the formulas below goes through bases (while we do not assume separability of spaces $X,Y,Z$). Below S denotes the shift operator, as in Sect. 4.

Proposition 5.2

Let $X$, $Y$, $Z$ be r.i. quasi-Banach function spaces such that $X\subset Y\subset Z$ and both $Y,Z$ have nontrivial Boyd indices. Assume that $a\in M(X,Y)$ and $b\in M(Y,Z)$. Then $ba\in M(X,Z)$ and

• (i) $T_{ba} = T_bT_a+H_bH_{SJa}$, where $T_{ba}, T_bT_a, H_bH_{SJa}:H[X]\rightarrow H[Z]$,

• (ii) $H_{ba} = T_bH_a+H_bT_{SJa}$, where $H_{ba},T_bH_a,H_bT_{SJa}:H[X]\rightarrow H[Z]$.

Proof

(i). It is easy to see that $PJ=J(1-P)$ and also from the definition of the flip operator it follows that for any $f,g\in L^0$ one has $J(fg)=(SJf)\cdot (Jg)$. Since $P$ is a projection and $J$ is an involution (that is, $J^2=\mathrm {id}$, we get

$$\begin{aligned} T_{ba}&= PM_{ba} = PM_bM_a = PM_b\bigl (P+(1-P)\bigr )M_a\\&= PM_bPM_a + PM_bJ^2(1-P)M_a = T_bT_a +H_bJ(1-P)M_a\\&= T_bT_a + H_b PJM_a = T_bT_a+H_bPM_{SJa}J = T_bT_a+H_bH_{SJa}. \end{aligned}$$

(ii). Proceeding as in the proof of (i), we get

$$\begin{aligned} H_{ba}&= PM_{ba}J = PM_bM_a J = PM_b\bigl (P+(1-P)\bigr )M_aJ\\&= PM_bPM_aJ + PM_b(1-P)M_aJ = T_bH_a+PM_bJJ(1-P)M_aJ\\&= T_bH_a + H_bPJM_aJ = T_bH_a+H_bT_{SJa}. \end{aligned}$$

$\square$

Observe that compactness of semicommutators $T_bT_a-T_{ba}$ can be directly derived from Theorem 4.4 and Proposition 5.2. However, both results are not sufficient when studying the same property of commutators. Indeed, we need to work with an intermediate space which plays the role of $H^{s}$ on the diagram in Example 5.1 and which replaces $Y$ after changing an order of operators. Let $X$, $Y$ and $Z$ satisfy assumption of the above theorem. The diagram that answers this quest looks like

where the formula for W is given in the lemma below.

Proposition 5.3

Let $X$, $Y$, $Z$ be r.i. Banach function spaces such that $X\subset Y\subset Z$. If $a\in M(X,Y)$, $b\in M(Y,Z)$, and $W=X\odot M(Y,Z)$, then the following diagram commutes

Proof

Let $W=X\odot M(Y,Z)$. It should be noted that in general $W$ is not a Banach space, but only a quasi-Banach space. Regardless of that we may proceed in the following way. From the definition of the space of pointwise multipliers we get that

$$\begin{aligned} Y\odot M(Y,Z)\subset Z. \end{aligned}$$

Now, by the cancellation law from [17, Theorem 4] we get that

$$\begin{aligned} M(X,Y)=M\bigl (X\odot M(Y,Z), Y\odot M(Y,Z)\bigr )\subset M(W,Z). \end{aligned}$$

(5.1)

Similarly

$$\begin{aligned} M(X,W)=M\bigl (X,X\odot M(Y,Z)\bigr )=M\bigl (L^{\infty },M(Y,Z)\bigr )=M(Y,Z). \end{aligned}$$

(5.2)

However, since $M_a:X\rightarrow Y$ and $M_b:Y\rightarrow Z$ are equivalent to $a\in M(X,Y)$ and $b\in M(Y,Z)$, then inclusion (5.1) and equality (5.2) yield $M_a:W\rightarrow Z$ and $M_b:X\rightarrow W$, which finishes the proof. $\square$

We are now ready to state the main result of this section.

Theorem 5.4

Let $X\subset Y\subset Z$ be r.i. Banach function spaces such that $Y$ and $Z$ have nontrivial Boyd indices. Let $a\in M(X,Y)$ and $b\in M(Y,Z)$.

• (i) Assume that $W=X\odot M(Y,Z)$ has nontrivial Boyd indices. If $a\in M (X,Y)_{o}+H[M(X,Y)]$ or $b\in M {(Y,Z)}_{o}+\overline{H[M(Y,Z)]}$ and $a\in M (X,Y)_{o}+\overline{H[M(X,Y)]}$ or $b\in M {(Y,Z)}_{o}+H[M(Y,Z)]$, then the commutator $[T_a,T_b]$ is compact. In particular, when $a\in M (X,Y)_{o}$ or $b\in M {(Y,Z)}_{o}$ then the commutator $[T_a,T_b]$ is compact.

• (ii) If $a\in M (X,Y)_{o}+H[M(X,Y)]$ or $b\in {M(Y,Z)}_{o}+\overline{H[M(Y,Z)]}$, then the semicommutator $(T_b,T_a]$ is compact.

Proof

(i) Notice that Proposition 5.3 implies

$$\begin{aligned} a\in M(W,Z) \quad \text {and}\quad b\in M(X,W), \end{aligned}$$

which gives

$$\begin{aligned} T_a,H_a,H_{SJa}:H[W]\rightarrow H[Z]\quad \text {and}\quad T_b,H_b,H_{SJb}:H[X]\rightarrow H[W]. \end{aligned}$$

Applying Proposition 5.2(i) once to $T_bT_a$ and secondly to $T_aT_b$ with the space Y replaced by W, we get the formula

$$\begin{aligned} {[}T_a,T_b]=H_bH_{SJa}-H_aH_{SJb}. \end{aligned}$$

Thus the point (i) will be proved once we explain that under our assumptions both $H_bH_{SJa}$ and $H_aH_{SJb}$ are compact.

At first suppose that $a\in M (X,Y)_{o}+H[M(X,Y)]$. Then

$$\begin{aligned} SJa\in M (X,Y)_{o}+\overline{H[M(X,Y)]} \end{aligned}$$

and Theorem 4.4 implies that $H_{SJa}:H[X]\rightarrow H[Y]$ is compact. Alternatively, $H_{b}:H[Y]\rightarrow H[Z]$ is compact, so is $H_bH_{SJa}$.

Consider now the second assumption, that is

$$\begin{aligned} a\in M (X,Y)_{o}+\overline{H[M(X,Y)]}\quad \text {or}\quad b\in M(Y,Z)+H[M(Y,Z)]. \end{aligned}$$

If $a\in M (X,Y)_{o}+\overline{H[M(X,Y)]}$ then $a\in M{(W,Z)}_{o}+\overline{H[M(W,Z)]}$ by the formula (5.1). Moreover, $M(W,Z)$ is a Banach space, even if $W$ is just quasi-Banach. Thus explaining exactly as in the proof of Theorem 4.4(i) we conclude that $H_a:H[W]\rightarrow H[Z]$ is compact. Otherwise, when $b\in M(Y,Z)+H[M(Y,Z)]$, then the explanation is the same because of equalities (5.2). Consequently, also $H_aH_{SJb}$ is compact.

(ii) The proof follows from the formula $(T_b,T_a]=-H_bH_{SJa}$ and Theorem 4.4. $\square$

Recall that the question about compactness of the commutator $[T_a,T_b]$ or the semicommutator $(T_b,T_a]$ in the classical setting of $H^2$ was considered by many authors (see for example [1, 4] and references therein). Especially, it was a famous problem to find necessary and sufficient condition for compactness of $(T_b,T_a]$, which arose from Fredholm theory for Toeplitz operators. Finally it was solved by Axler, Chang, Sarason [1] and Vol’berg [37]. They showed that $(T_b,T_a]$ is compact if and only if $H^{\infty }[\overline{f}]\cap H^{\infty }[g]\subset H^{\infty }+C$, where $H^{\infty }[h]$ denotes the Douglas algebra generated by $H^{\infty }$ and $h$. In particular, this condition is far weaker than condition that follows from Hartman’s compactness theorem for Hankel operators by the formula $(T_b,T_a]=-H_bH_{SJa}$. This suggests that also conditions of Theorem 5.4 are far from being necessary. However, in essentially nonalgebraic setting, it is even difficult to decide whether these conditions are in fact unnecessary. The reason is that, on one hand, we do not dispose any tools of Banach algebras as in the classical setting and, on the other hand, one can not look for such examples among classical Hardy spaces, since $M(L^p,L^q)$ is order continuous for every $1\leqslant q<p\leqslant \infty$ and thus each Hankel operator from $H^p$ to $H^q$ is compact.

The following corollary explains that one may look for such an example among the class of Hardy–Lorentz spaces.

Corollary 5.5

Let $1<p_3<p_2<p_1<\infty$ and suppose $a\in M(L^{p_1,q_1},L^{p_2,q_2})$, $b\in M(L^{p_2,q_2},L^{p_3,q_3})$.

• (i) If $q_1>q_2$ or $q_2>q_3$, then $[T_a,T_b],(T_b,T_a]:H^{p_1,q_1}\rightarrow H^{p_3,q_3}$ are compact.

• (ii) If $q_1\leqslant q_2\leqslant q_3$, then $M(L^{p_1,q_1},L^{p_2,q_2})=L^{r_1,\infty }$ and $M(L^{p_2,q_2},L^{p_3,q_3})=L^{r_2,\infty }$ for $\frac{1}{r_i}=\frac{1}{p_{i+1}}-\frac{1}{p_i}$, $i=1,2$. Furthermore, $[T_a,T_b]:H^{p_1,q_1}\rightarrow H^{p_3,q_3}$ is compact, whenever

  $$\begin{aligned}&\bigl [Pa\in {L^{r_1,\infty }_{o}} \text{ or } (1-P)b\in {L^{r_2,\infty }_{o}}\bigr ] \\ \text{ and }&\bigl [(1-P)a\in {L^{r_1,\infty }_{o}} \text{ or } Pb\in {L^{r_2,\infty }_{o}}\bigr ]. \end{aligned}$$

  Moreover, $(T_b,T_a]:H^{p_1,q_1}\rightarrow H^{p_3,q_3}$ is compact, whenever

  $$\begin{aligned} Pa\in {L^{r_1,\infty }_{o}}\quad \text{ or }\quad (1-P)b\in {L^{r_2,\infty }_{o}}. \end{aligned}$$

Proof

The proof is an immediate consequence of Proposition 5.4, [18, Theorem 4] and boundedness of $P$ on $L^{p,q}$ spaces for each $p\in (1,\infty )$. $\square$

Therefore, finalizing the previous discussion, we see that in the case (ii) one can select $a$ and $b$ such that

$$\begin{aligned} Pa,(1-P)a\in L^{r_1,\infty }\setminus {L^{r_1,\infty }_{o}} \quad \text{ and }\quad Pb,(1-P)b\in L^{r_2,\infty }\setminus {L^{r_2,\infty }_{o}} . \end{aligned}$$

Then the natural question is whether for some $a,b$ as above

$$\begin{aligned} {[}T_a,T_b],(T_b,T_a]:H^{p_1,q_1}\rightarrow H^{p_3,q_3} \end{aligned}$$

may still be compact?