Remarks on the Levi core

Abstract

We investigate a few aspects of the notion of Levi core, recently introduced by the authors in Dall’Ara, Mongodi (J l’École Polytech Math 10:1047-1095, 2023): a basic finiteness question, the connection with Kohn’s algorithm, and with Catlin’s property (P).

1 Introduction

This note complements the recent paper "The core of the Levi distribution" [8], where the authors introduced a new geometric invariant associated to CR manifolds of hypersurface type, called the Levi core. We refer to the paper for motivations and applications to the regularity theory of the ${{\bar{\partial }}}$-Neumann problem. Here, we want to address certain basic questions that arise quite naturally when considering the Levi core, and indeed have been raised by several people in private communications to the authors.

The Levi core is a special case of a more general construction that attaches to any distribution ${\mathcal {D}}$ of subspaces of the tangent bundle of a manifold a smaller distribution ${\mathfrak {C}}({\mathcal {D}})$, its core. The core is defined starting with ${\mathcal {D}}$ and iterating a "derived distribution" construction until it eventually stabilizes. This works in great generality, namely for any smooth manifold and any distribution, under no regularity assumption beyond the basic requirement that ${\mathcal {D}}$ be a closed subset of the tangent bundle. The price to be paid for that generality is that the stabilization may require a (countably) infinite number of iterations, a fact that is in the nature of things, as the notion of derived distribution collapses in the one-dimensional case to that of derived set (and this is indeed the origin of the terminology) in set theory. We refer again to [8] for a more detailed discussion.

The transfinite nature of the core makes it difficult to extract some usable geometric structure from it. Since such a structure would be highly desirable in applications, for example, in the case of the Levi core in connection with the ${{\bar{\partial }}}$-Neumann problem, a couple of basic natural questions arise: is the core ${\mathfrak {C}}({\mathcal {D}})$ "reached" after finitely many iterations, under the assumption that the underlying manifold and the distribution ${\mathcal {D}}$ lie in an appropriate category smaller than ${\mathcal {C}}^\infty$? In that case, do the core and its support lie in the same category? After reviewing the terminology and definitions from [8] in Sect. 2, in Sect. 3 we observe that this is indeed the case in the complex algebraic and complex analytic categories, and also in the real analytic category under an additional coherence assumption. We also point out the difficulties that present themselves if one drops the coherence hypothesis, or ventures beyond the analytic category.

The algebraic outlook of Sect. 3 leads then to the consideration of the relation between the notion of Levi core (that is, the core of the Levi null distribution of a CR manifold) and the well-known Kohn’s algorithm. In Sect. 4, we discuss this relation, the theme being that the two are indeed manifestations of the same idea, at least when restricting to nice enough categories.

In Sect. 5, we discuss a couple of specializations of the notion of core to complex analysis, other than the Levi core.

Finally, in Sect. 6 we show that there are bounded smooth pseudoconvex domains satisfying Catlin’s property (P) and having nontrivial Levi core. This theme has been explored in more depth by Treuer [22].

2 Basic definitions

Let M be a smooth manifold. We begin by recalling some terminology and definitions from [8].

A real distribution on M is a subset ${\mathcal {D}}$ of the tangent bundle TM such that the fiber ${\mathcal {D}}_p:={\mathcal {D}}\cap T_pM$ is a vector subspace of $T_pM$ for every $p\in M$. A complex distribution on M is a subset ${\mathcal {D}}$ of the complexified tangent bundle ${\mathbb {C}}TM$ such that ${\mathcal {D}}_p:={\mathcal {D}}\cap {\mathbb {C}}T_pM$ is a complex vector subspace of ${\mathbb {C}}T_pM$ for every $p\in M$.

The support $S_{{\mathcal {D}}}$ of a distribution ${\mathcal {D}}$ is the set of points $p\in M$ such that ${\mathcal {D}}_p\ne \{0\}$. We call a distribution closed if it is a closed subset of TM (or ${\mathbb {C}}TM$); for a closed distribution the function $p\mapsto \dim {\mathcal {D}}_p$ is upper semicontinuous, and in particular the support $S_{{\mathcal {D}}}$ is closed.

Closed distributions naturally occur as common kernels of subsets ${\mathfrak {R}}$ of $\Omega ^1(M)$, i.e., of smooth differential 1-forms. More precisely, given such an ${\mathfrak {R}}$ one defines

$$\begin{aligned} {\mathcal {D}}_p=\{X_p\in T_pM\,:\ \langle \omega _{p}, X_p\rangle =0\quad \forall \omega \in {\mathfrak {R}} \}, \end{aligned}$$

and ${\mathcal {D}}=\cup _p{\mathcal {D}}_p=:\ker {\mathfrak {R}}$. There is no loss of generality in assuming that ${\mathfrak {R}}$ is a $C^\infty (M)$-submodule of $\Omega ^1(M)$.

As a special case of the above construction, given a set $S\subseteq M$, consider the associated ideal of smooth functions

$$\begin{aligned} {\mathcal {I}}(S)=\{f\in {\mathcal {C}}^\infty (M)\,:\ f\vert _S\equiv 0\} \end{aligned}$$

and the $C^\infty (M)$-module of 1-forms ${\mathfrak {R}}(S)$ generated by $\{df\,:\ f\in {\mathcal {I}}(S)\}$. We call the associated distribution $\ker {\mathfrak {R}}(S)$ the tangent distribution to S, and we denote it by TS (cf. [8, Def. 2.5]). Notice that, according to this definition, TS is a subset of TM with zero fiber at every point outside the closure of S (and also at every isolated point of S).

We now recall from [8] the two definitions that are most important for our discussion.

Definition 2.1

(Derived distribution) Let M be a smooth manifold and let ${\mathcal {D}}\subseteq TM$ be a real distribution. The distribution

$$\begin{aligned} {\mathcal {D}}':={\mathcal {D}}\cap TS_{{\mathcal {D}}} \end{aligned}$$

will be called the derived distribution of ${\mathcal {D}}$. Analogously, if ${\mathcal {D}}\subseteq {\mathbb {C}}TM$ is a complex distribution on M, its derived distribution is defined as

$$\begin{aligned} {\mathcal {D}}':={\mathcal {D}}\cap {\mathbb {C}}TS_{{\mathcal {D}}}. \end{aligned}$$

Here, ${\mathbb {C}}TS_{{\mathcal {D}}}$ is obtained complexifying pointwise $TS_{{\mathcal {D}}}$.

A distribution that equals its derived distribution is said to be perfect.

It is possible to iterate the operation of "taking the derived distribution." Given an ordinal $\alpha$, we define ${\mathcal {D}}^{(\alpha )}$ by transfinite recursion (see [6], Theorem 4.3.1):

• (1) if $\alpha =0$, we set ${\mathcal {D}}^{(0)}:={\mathcal {D}}$;

• (2) if $\alpha +1$ is a successor ordinal, we set ${\mathcal {D}}^{(\alpha +1)}:=({\mathcal {D}}^{(\alpha )})'$;

• (3) if $\alpha$ is a limit ordinal, we set ${\mathcal {D}}^{(\alpha )}:=\bigcap _{\beta <\alpha }{\mathcal {D}}^{(\beta )}$.

If ${\mathcal {D}}$ is closed, then every ${\mathcal {D}}^{(\alpha )}$ is closed (this is easily proved by transfinite induction) and we have the following Cantor–Bendixson-type theorem (cf. [8, Thm. 2.9]).

Theorem 2.2

If ${\mathcal {D}}$ is a closed (real or complex) distribution, then there exists a countable ordinal $\alpha$ such that ${\mathcal {D}}^{(\alpha )}$ is perfect.

Thanks to Theorem 2.2, we can formulate our second key definition.

Definition 2.3

(Core of a distribution) Let ${\mathcal {D}}$ be a (real or complex) closed distribution. Then, the core of ${\mathcal {D}}$ is the distribution ${\mathfrak {C}}({\mathcal {D}})={\mathcal {D}}^{(\alpha )}$, where $\alpha$ is the minimal ordinal such that $({\mathcal {D}}^{(\alpha )})'={\mathcal {D}}^{(\alpha )}$.

Notice that both Definition 2.1 and Definition 2.3 are local in nature: distributions can be restricted to open sets in the obvious way

$$\begin{aligned} {\mathcal {D}}_{|U}=\cup _{p\in U}{\mathcal {D}}_p\qquad \forall U\subseteq M\text { open}, \end{aligned}$$

and then ${\mathcal {D}}'_{|U}=({\mathcal {D}}_{|U})'$ and ${\mathfrak {C}}({\mathcal {D}}_{|U})={\mathfrak {C}}({\mathcal {D}})_{|U}$.

3 The core in the algebraic and analytic categories

In the construction of the core, we require that the distribution ${\mathcal {D}}$ be closed; this property carries on to the derived distribution, because the support of a closed distribution is a closed set and the tangent distribution to a closed set is a closed distribution, and it is a key ingredient in proving that we need countably many steps to reach the core, which, in turn, is again a closed set.

An analogous situation happens in the category of complex algebraic or analytic varieties.

Proposition 3.1

Let X be a complex algebraic (or analytic) manifold and let ${\mathcal {D}}\subseteq TX$ be a distribution which is also an algebraic (or analytic) subvariety of TX; then:

• (1) $S_{\mathcal {D}}$ is an algebraic (or analytic) subvariety of X,

• (2) ${\mathcal {D}}'$ is an algebraic (or analytic) subvariety of TX,

• (3) ${\mathfrak {C}}_{{\mathcal {D}}}$ is locally reached in a finite number of steps; hence, it is an algebraic (or analytic) subvariety of TX and its support is an algebraic (or analytic) subvariety of X.

Proof

We prove the three statements in order.

$S_{{\mathcal {D}}}$ is an algebraic (or analytic) subvariety of X. $X\times \{0\}$ is a subvariety of TX which is isomorphic to X and it is contained in ${\mathcal {D}}$. If $S_{{\mathcal {D}}}=X$, then $S_{{\mathcal {D}}}$ is trivially a subvariety of X, otherwise there are points $(x,0)\in TX$ where, locally, $X\times \{0\}$ and ${\mathcal {D}}$ coincide (i.e., they coincide in an open neighborhood of $(x,0)\in TX$); therefore, $X\times \{0\}$ is an irreducible component of ${\mathcal {D}}$ (by the decomposition in irreducible components of complex algebraic, or analytic, subvarieties). Let us denote by Y the union of all the other irreducible components of ${\mathcal {D}}$, which is again a subvariety; then, via the isomorphism between X and $X\times \{0\}\subseteq TX$, $S_{{\mathcal {D}}}$ corresponds to $Y\cap X\times \{0\}$, which is a subvariety.

${\mathcal {D}}'$ is an algebraic (or analytic) subvariety of TX. By [20], the tangent distribution to $S_{{\mathcal {D}}}$ as we defined it in the previous section coincides with the holomorphic Zariski tangent space (which coincides with the algebraic one for an algebraic subvariety), so it is a subvariety of TX. Therefore, ${\mathcal {D}}'={\mathcal {D}}\cap TS_{{\mathcal {D}}}$ is an intersection of two subvarieties of TX, hence a subvariety.

${\mathfrak {C}}({\mathcal {D}})$ is reached (locally) in a finite number of steps. The ring of germs of polynomials and the ring of germs of holomorphic functions are Noetherian, hence descending chains stabilize in a finite number of steps. Therefore, around each point of TX, the sequence ${\mathcal {D}}\supset {\mathcal {D}}'\supset \ldots$ stabilizes in a finite number of steps, which implies that, around each point of TX, ${\mathfrak {C}}({\mathcal {D}})$ is a subvariety (as it is obtained, locally, as the intersection of finitely many subvarieties).

As a consequence, the support of the core is a complex algebraic (or analytic) subvariety of X. $\square$

Example 3.2

Consider a holomorphic map $F:{\mathbb {C}}^n\rightarrow {\mathbb {C}}^m$ and define

$$\begin{aligned} \Omega =\{(z',z_{n+1})\in {\mathbb {C}}^n\times {\mathbb {C}}\,:\ \Re z_{n+1}<\Vert F(z')\Vert ^2\}. \end{aligned}$$

The distribution given by the kernel of the Levi form of $b\Omega$ is diffeomorphic to (by projection) $\ker \textrm{Jac}F(x)$ as a distribution in ${\mathbb {C}}^n$; the latter is a complex analytic distribution and, as the construction of the core is invariant by diffeomorphisms, this shows that the core of the distribution given by the kernel of the Levi form of $b\Omega$ is reached in a finite number of steps.

We also have the following geometric results about complex algebraic (or analytic) distributions.

Lemma 3.3

Let X be a complex algebraic (or analytic) manifold and ${\mathcal {D}}\subseteq TX$ be a distribution which is also a complex algebraic (or analytic) subvariety of TX; denote by Y the union of the irreducible components of ${\mathcal {D}}$ different from $X\times \{0\}$, then for every $(x,v)\in TX$ we have

$$\begin{aligned} T_{(x,v)}Y=T_xS_{{\mathcal {D}}}\times {\mathcal {D}}_x. \end{aligned}$$

Proof

Take $x\in S_{{\mathcal {D}}}$, otherwise there is nothing to prove. It is obvious that $T_xS_{{\mathcal {D}}}\times {\mathcal {D}}_x$ is a subspace of $T_{(x,v)}Y$, because $S_{{\mathcal {D}}}\times \{0\}\cup \{x\}\times {\mathcal {D}}_x$ is contained in Y. Now, for the reverse inclusion, consider a complex algebraic (or analytic) manifold $M\subseteq X$ such that $S_{{\mathcal {D}}}\subseteq M$ and $T_x S_{{\mathcal {D}}}=T_xM$; in a local trivialization of TX, consider the product $M\times {\mathcal {D}}_x=D$. There are two cases:

• either Y is locally contained in D around (x, v), then $T_{(x,v)}Y\subseteq T_{(x,v)}M=T_xS_{{\mathcal {D}}}\times {\mathcal {D}}_x$

• or, locally around (x, v), $Y\cap D=\{x\}\times {\mathcal {D}}_x$;

in the second case, the fiber-wise projection $\pi :TX\vert _M\rightarrow D$ given by $\pi (y,v)=(y,\pi _y(v))$ where $\pi _y:T_yX\rightarrow \{y\}\times {\mathcal {D}}_x$ is a linear projection, is a holomorphic injective map. Hence, locally near (x, v), $\pi \vert _{Y}:Y\rightarrow \pi (Y)$ is a biholomorphism with its image; it is clear that $T_{(x,v)}\pi (Y)\subseteq T_{(x,v)}D=T_xS_{{\mathcal {D}}}\times {\mathcal {D}}_x$ and $D\pi$ (the Jacobian of $\pi$) is the identity along such subspace of T(TX). $\square$

Proposition 3.4

If X is a complex algebraic (or analytic) manifold and ${\mathcal {D}}\subseteq TX$ is a distribution which is also a complex algebraic (or analytic) subvariety of TX, then there exists ${\mathcal {R}}\subseteq \Omega ^1(X)$ a ${\mathcal {O}}$-module of 1-forms (with ${\mathcal {O}}$ the structural sheaf) such that ${\mathcal {D}}$ is described as the zero locus of ${\mathcal {R}}$.

Proof

Let ${\mathcal {O}}_{TX}$ be the structure sheaf of TX as a complex algebraic (or analytic) manifold and take a germ $f\in {\mathcal {O}}_{TX,(x,v)}$ such that $f\vert _{{\mathcal {D}}}\equiv 0$. In a local trivialization of TX, with coordinates (y, w), we can write

$$\begin{aligned} f=f_0+f_1+f_2+\ldots \end{aligned}$$

where each $f_j$ is a homogeneous polynomial of degree j in the variables w; then, as ${\mathcal {D}}_y$ is a linear subspace of $T_yX$ for every $y\in X$, it is easy to see that $f_0\equiv 0$ and ${\mathcal {D}}_y\subseteq \{w\in T_yX\,\ f_1(y,w)=0\}$.

On the other hand, given ${\tilde{w}}\not \in {\mathcal {D}}_x$, by the previous lemma $W=(0,{\tilde{w}})\not \in T_x{\mathcal {D}}$, therefore there is $f\in {\mathcal {O}}_{TX,(x,{\tilde{w}})}$, vanishing along ${\mathcal {D}}$, such that $\langle W, df\rangle \ne 0$ in $(x,{\tilde{w}})$. But then, as ${\mathcal {D}}_x$ is a linear subspace, $\langle W, df\rangle \ne 0$ at (x, 0) as well. This implies that $f_1(x,{\tilde{w}})\ne 0$.

Therefore, for any $f\in {\mathcal {O}}_{TX, (x,0)}$ which vanishes identically on ${\mathcal {D}}$, consider $f_1$ as defined above and write

$$\begin{aligned} f_1=\alpha _1(x)v_1+\ldots +\alpha _n(x)v_n. \end{aligned}$$

To this function, we associate the differential form $\alpha =\alpha _1(x)dx_1+\ldots +\alpha _n(x)dx_n\in \Omega ^1(X)$; the set of all these differential forms is a ${\mathcal {O}}_X$-module, that we denote by ${\mathcal {R}}$, such that

$$\begin{aligned} \{(x,v)\in TX\,:\ \langle \alpha (x),v\rangle =0\ \forall \;\alpha \in {\mathcal {R}}\}={\mathcal {D}}, \end{aligned}$$

by what we proved above. $\square$

In view of these results, we give an algebraic version of the construction of the derived distribution, applied to modules of 1-forms.

Let X be a manifold and ${\mathcal {A}}={\mathcal {A}}_X\subseteq {\mathcal {C}}^\infty _X$ be a sheaf of ${\mathbb {R}}$-algebras (or ${\mathbb {C}}$-algebras, in case we consider complex valued functions); we furthermore assume that ${\mathcal {A}}$ gives local coordinates and is closed under differentiation, that is, we require the following two-part property:

• A1) every $p\in X$ has an open neighborhood U and $x_1,\ldots , x_n\in {\mathcal {A}}(U)$ such that $(x_1,\ldots , x_n)$ is a system of local coordinates, and

• A2) if $f\in {\mathcal {A}}(U)$, then there exist $f_1,\ldots , f_n\in {\mathcal {A}}(U)$ such that $df=\sum _{j=1}^nf_jdx_j$.

These properties give a naturally induced sheaf of algebras ${\mathcal {A}}_{TX}$ on TX.

We denote by $\Omega ^\bullet _{\mathcal {A}}=\bigoplus _{k=0}^n\Omega ^k_{\mathcal {A}}$ the sheaf of smooth differential forms that can be locally represented as a finite sum of terms of the form $fdg_1\wedge \ldots \wedge dg_k$, where $f,g_j$ are local sections of ${\mathcal {A}}$. It is easily checked that $\Omega ^\bullet _{\mathcal {A}}$ is closed under wedge product and exterior differentiation.

Definition 3.5

Given $S\subseteq X$, we define ${\mathcal {I}}_S$ as the ideal of germs $f\in {\mathcal {A}}$ which vanish identically on S; moreover, we define the ${\mathcal {A}}$-cotangent module of S as the sheaf of ${\mathcal {A}}$-modules $T^*_{\mathcal {A}}S$ generated by

$$\begin{aligned} \{df\,:\ f\in {\mathcal {I}}_{S}\}\subseteq \Omega ^1_{{\mathcal {A}}}. \end{aligned}$$

Definition 3.6

Given a sheaf of ${\mathcal {A}}$-modules ${\mathcal {R}}\subseteq \Omega ^1_{\mathcal {A}}$, the support of ${\mathcal {R}}$ is

$$\begin{aligned} Z_{{\mathcal {R}}}=\{x\in X\,:\ \alpha _1(x)\wedge \ldots \wedge \alpha _n(x)=0\ \forall \;(\alpha _1,\ldots ,\alpha _n)\in {\mathcal {R}}^n\} \end{aligned}$$

hence it is the vanishing locus of the sheaf of modules generated by all the n-forms $\alpha _1\wedge \ldots \wedge \alpha _n$ where $\alpha _1,\ldots , \alpha _n\in {\mathcal {R}}$.

The derived module ${\mathcal {R}}'$ is defined as the sheaf of ${\mathcal {A}}$-modules generated by ${\mathcal {R}}$ and $T^*_{\mathcal {A}}Z_{{\mathcal {R}}}$.

Remark 3.7

Proceeding by transfinite induction, it is possible to define a concept analogous to the core, a core module; however, we do not pursue this direction, as in general there is no clear link between this module and the core of the starting distribution.

If ${\mathcal {A}}={\mathcal {C}}^\infty$, or if X is complex algebraic (or analytic) and ${\mathcal {A}}$ is the corresponding structure sheaf, we know that, defining ${\mathcal {D}}$ as the kernel of ${\mathcal {R}}$ (we write ${\mathcal {D}}=\ker {\mathcal {R}}$), we have that $Z_{{\mathcal {R}}}=S_{{\mathcal {D}}}$, $\ker T^*_{\mathcal {A}}Z_{{\mathcal {R}}}=TS_{{\mathcal {D}}}$, ${\mathcal {D}}'=\ker {\mathcal {R}}'$. In such cases, this construction is just an algebraic counterpart of the derived distribution construction; for complex algebraic (or analytic) functions, the algebraic properties of the structure sheaf allow us to derive further conclusions (like the local termination in a finite number of steps and the complex algebraicity—or analyticity—of the core).

In general, however, we incur in some problems:

• we do not know that $Z_{{\mathcal {R}}}$ is the zero set of an ideal of ${\mathcal {A}}$

• we do not know that $\ker T^*_{\mathcal {A}}Z=TZ$ (even if Z is the zero set of an ideal of ${\mathcal {A}}$)

• we do not have any Noetherianity assumption on ${\mathcal {A}}$.

In the case of real analytic functions, we have the Noetherianity of the local ring; to take care of the other two issues, we consider a coherent real analytic space (see Section 1.2 in [1] for details).

Proposition 3.8

Let X be a real analytic manifold and ${\mathcal {D}}$ be a distribution which is also a real analytic coherent subspace of TX, i.e., such that its ideal sheaf ${\mathcal {I}}_{{\mathcal {D}}, TX}\subseteq {\mathcal {O}}_{TX}$ is coherent. Then, $S_{{\mathcal {D}}}$ is a real analytic coherent subspace of X, ${\mathcal {D}}'$ and ${\mathfrak {C}}({\mathcal {D}})$ are again a real analytic coherent subspace of TX.

Proof

By coherence, we can repeat verbatim the proof of Proposition 3.1: ${\mathcal {D}}$ admits a decomposition in irreducible components and, if Z is an irreducible real analytic space such that $Z_{(x,v)}={\mathcal {D}}_{(x,v)}$ for some $(x,v)\in TX$, then Z is an irreducible component of ${\mathcal {D}}$. Therefore, $S_{{\mathcal {D}}}$ is defined as the intersection of real analytic coherent subspaces, and hence, it belongs to the same category; moreover, the quotient of coherent sheaves is coherent, as it is the dual of a coherent sheaf, so the tangent distribution to a real analytic coherent space is a real analytic coherent subspace of TX. Again by [20], the real analytic tangent distribution coincides with the smooth one and we reach the conclusion for ${\mathcal {D}}'$ in the same way. By Noetherianity, we have that the core is reached, locally at each point, in a finite number of steps, hence it is also a real analytic coherent subspace of TX, as it is its support. $\square$

Remark 3.9

In particular, given a pseudoconvex domain $\Omega$ with real analytic boundary in some complex manifold X, if the kernel of the Levi form gives a coherent distribution in $Tb\Omega$ (viewed as a real analytic manifold), then the Levi core of $\Omega$ is a coherent real analytic subset with positive holomorphic dimension. Hence, the results of [7] apply, giving the existence of at least a complex curve inside $b\Omega$ (and hence inside the Levi core).

Without the coherence hypothesis, things break down quite soon.

We first give an example of a noncoherent real analytic subspace of ${\mathbb {R}}^3$, which is nonetheless globally defined by one real analytic function; a coherent sheaf ${\mathcal {F}}$ is, in particular, of finite type, i.e., at every point $x\in X$ there exist an open set $U\ni x$ and a finite number of sections $s_1,\ldots , s_k\in \Gamma ({\mathcal {F}},U)$ such that $(s_1)_y,\ldots , (s_k)_y$ generate ${\mathcal {F}}_y$ for all $y\in U$.

Example 3.10

Let $M={\mathbb {R}}^3$ and ${\mathcal {A}}$ be the algebra of real analytic functions; consider ${\mathcal {I}}=(x^2-yz^2)$. Its zero set $W=Z({\mathcal {I}})$ is called Whitney umbrella. We have that ${\mathcal {I}}_W$ is not of finite type: pick any neighborhood U of (0, 0, 0), then ${\mathcal {I}}_W(U)\subseteq (x^2-yz^2)$ by real analyticity, but, for each $p=(0,t,0)$ with $t<0$, $({\mathcal {I}}_W)_p=(x,z)$.

We notice that, in the case of the Whitney’s umbrella, the problem is not the requirement for a finite number of sections: all the sections of ${\mathcal {I}}_W$ on a neighborhood of (0, 0, 0) do not generate ${\mathcal {I}}_W$ at any point of the form (0, t, 0) with $t<0$.

Next, we show that the support of a real analytic distribution may not be a real analytic space, never mentioning it being coherent.

Example 3.11

Consider the Whitney umbrella W and let ${\mathcal {R}}$ be the module of 1-forms given by the cotangent sheaf ${\mathcal {I}}_W/{\mathcal {I}}_W^2$ (as ${\mathcal {I}}_W$ is not of finite type, ${\mathcal {R}}$ is also not of finite type). Then, $\ker {\mathcal {R}}$ is the tangent distribution of W, in particular $\ker {\mathcal {R}}_p$ with $p=(0,t,0)$, $t<0$, is the distribution locally generated by $\partial _y$; now, set ${\mathcal {D}}=\ker \mod {\mathbb {R}}\cap \ker {\mathcal {A}}dy$. As $\dim ({\mathcal {I}}_W/{\mathcal {I}}_W^2)^*_p=2$ if $W\ni p\ne (0,t,0)$, $t<0$, we have that ${\mathcal {D}}_p\ne 0$ if $p\in W\cap \{y\ge 0\}$; however, ${\mathcal {D}}_{(0,t,0)}=\{0\}$ for $t<0$. So $S_{{\mathcal {D}}}=W\cap \{y\ge 0\}$ which is not an analytic set in ${\mathbb {R}}^3$.

Moreover, even if the distribution is defined by a finite type ideal in TX, it may happen that its support is not of finite type.

Example 3.12

Consider again $M={\mathbb {R}}^3$, ${\mathcal {A}}$ the algebra of real analytic functions and ${\mathcal {R}}$ the module of 1-forms generated by

$$\begin{aligned} xdx-ydz,\ zdx+dy,\ zdy+xdz. \end{aligned}$$

If we set ${\mathcal {D}}=\ker {\mathcal {R}}$, we have that $S_{\mathcal {D}}$ is the Whitney umbrella W, hence not of finite type, even if ${\mathcal {R}}$ is globally generated.

3.1 Comments on the general case

An algebraic approach to the reduction to the core finds the following problems, which seem to be interconnected:

• even if the module of 1-forms defining a distribution is locally generated at each point by its sections, this may not be true for the module defining the derived distribution

• even if the module of 1-forms defining a distribution is locally generated at each point by its sections, this may not be true for the ideal of germs vanishing on its support

• if the module of 1-forms defining a distribution is not locally generated by its sections, its support may not be the zero set of an ideal

• even if an ideal ${\mathcal {I}}$ is locally generated at each point by its sections, this may not be true for the radical of such an ideal or for ${\mathcal {I}}(Z({\mathcal {I}}))$.

All these issues remain even if we consider a finite number of generating local sections, i.e., sheaves of finite type.

Coherence, in the real analytic case, helped us to overcome such difficulties; in general, if we can find a good class of sheaves whose elements are locally generated at each point by their sections and which is stable under suitable operations, we may construct a sequence of ideals associated to the derived distributions and hence prove that the core of a distribution is (in such class) a zero set of an ideal.

We notice that, in constructing the module ${\mathcal {R}}'$, we had to consider the ideal ${\mathcal {I}}_{Z_{{\mathcal {R}}}})$; when we are in ${\mathbb {C}}^n$ or ${\mathbb {R}}^n$, $Z_{{\mathcal {R}}}$ can be described as the zero set of an ideal ${\mathcal {I}}_{\mathcal {R}}$, as all the wedge products $\alpha _1\wedge \ldots \wedge \alpha _n$ will have the form $fdx_1\wedge \ldots \wedge dx_n$. Therefore, we are reduced to consider an ideal of the form ${\mathcal {I}}_{Z_{{\mathcal {I}}_{\mathcal {R}}}}$.

In many situations, we can describe more precisely this ideal, thanks to Null stellen satz-type results. For example, if we are working with germs of complex polynomials (or holomorphic functions), ${\mathcal {I}}_{Z_{{\mathcal {I}}_{\mathcal {R}}}}=\root h \of {{\mathcal {I}}_{\mathcal {R}}}$, where $\root h \of {\cdot }$ is the radical appearing in Hilbert’s Nullstellensatz (the "usual one"):

$$\begin{aligned} \root h \of {{\mathcal {J}}}=\{f\in {\mathcal {A}}:\ g-f^k=0\ \text {for some }k\in {\mathbb {N}},\ g\in {\mathcal {J}}\}. \end{aligned}$$

In the real (or real analytic) case, the real radical $\root r \of {\cdot }$ plays the same role, where

$$\begin{aligned} \root r \of {{\mathcal {J}}}=\{f\in {\mathcal {A}}:\ g-f^{2m}=\sum h_i^2 ,\ \ m\in {\mathbb {N}},\ h_i\in {\mathcal {A}},\ g\in {\mathcal {J}}\}. \end{aligned}$$

Other kinds of radical can be defined, for example, for real germs, adding a convexity property (as in [9]), we obtain what is called the Lojasiewicz radical in [2],

$$\begin{aligned} \root c \of {{\mathcal {J}}}=\{f\in {\mathcal {A}}\,:\ \exists \;g\in {\mathcal {J}},\ m\in {\mathbb {N}}\ \text{ s.t. }\ g-f^{2m}\ge 0\}; \end{aligned}$$

this radical appears, for example, in a version of the Nullstellensatz for Denjoy-Carleman quasianalytic functions.

It is natural to give the following definition.

Definition 3.13

Given an ${\mathcal {A}}$-module of germs of differential 1-forms, we can define ${\mathcal {R}}^c$ as the module generated by ${\mathcal {R}}$ and

$$\begin{aligned} \{df\,:\ f\in \root c \of {{\mathcal {I}}_{{\mathcal {R}}})}\} \end{aligned}$$

We define the c-derivation as ${\mathcal {D}}'_c=\ker {\mathcal {R}}^c$.

Remark 3.14

Obviously, we can define ${\mathcal {R}}^*$ and ${\mathcal {D}}'_*$ with $*\in \{\text {h, r}\}$, in an analogous way; however, we will only need them for $*=c$ in what follows.

Remark 3.15

If a Nullstellensatz-type result holds, one can show the equality between ${\mathcal {R}}'$ and the corresponding ${\mathcal {R}}^*$ and also between the related cores.

4 Kohn’s algorithm

We discuss in this section the relation between the construction of the derived distributions and Kohn’s algorithm; we refer to [9] for the meaning of Kohn’s algorithm in relation to the subellipticity problem for ${\overline{\partial }}$ and ${\overline{\partial }}$-Neumann operators. We recall the definition of Kohn’s algorithm of multiplier ideal sheaves for (0, 1)-forms: given $\Omega$ a domain in ${\mathbb {C}}^n$, let $x_0\in b\Omega$ and take a germ $r\in {\mathcal {C}}^\infty _{x_0}$ such that $r\equiv 0$ on $b\Omega$, but dr does not vanish; define

$$\begin{aligned}{} & {} I_0(x_0)=(r,\star \partial r\wedge {\overline{\partial }}r\wedge (\partial {\overline{\partial }}r)^{n-1})\\{} & {} I_k(x_0)=\root c \of {(I_{k-1}(x_0), A_{k-1}(x_0))} \end{aligned}$$

where $\star$ is the Hodge $\star$-operator and $A_{k-1}(x_{0})$ is given by

$$\begin{aligned} \big \{\star \partial f_1\wedge {\overline{\partial }}f_1\wedge \ldots \wedge \partial f_j\wedge {\overline{\partial }}f_j\wedge \partial r\wedge {\overline{\partial }}r\wedge (\partial {\overline{\partial }}r)^{n-1-j}\,: \ j\in {\mathbb {N}}, f_1,\ldots , f_j\in I_{k-1}(x_0)\big \}. \end{aligned}$$

Theorem 4.1

If we consider the Levi null distribution ${\mathcal {N}}$ on $b\Omega$ and its sequence f derived distributions ${\mathcal {N}}={\mathcal {D}}^{(0)}\supseteq {\mathcal {D}}^{(1)}\supseteq \ldots \supseteq {\mathfrak {C}}({\mathcal {N}})$, we have that

$$\begin{aligned} I_k(x_0)\subseteq {\mathcal {I}}(Z(I_{{\mathcal {D}}^{(k)}, x_0})) \end{aligned}$$

for all $k\in {\mathbb {N}}$, where ${\mathcal {I}}_{{\mathcal {D}}^{(k)}}$ is the ideal sheaf of the functions that vanish identically on $S_{{\mathcal {D}}^{(k)}}$ (whose zero set may, in principle, be larger than the support of ${\mathcal {D}}^{(k)}$).

On the other hand, let ${\mathcal {D}}^{(\alpha )}_c$ be the sequence of null distributions given by the c-derivation ${\mathcal {R}}\mapsto {\mathcal {R}}^c$, starting from ${\mathcal {N}}$; then:

$$\begin{aligned} I_k(x_0)= \root c \of {{\mathcal {I}}_{{\mathcal {D}}^{(k)}_c, x_0}} \end{aligned}$$

for all $k>0$.

Proof

Let $z_1,\ldots , z_n$ be local coordinates around $x_0$, then the module of 1-forms ${\mathcal {R}}$ generated by $\partial r,{\overline{\partial }}r$ and the collections

$$\begin{aligned} \partial {\overline{\partial }}r(\cdot ,\partial _{{\bar{z}}_j})\ \text{ for } j=1,\ldots , m\qquad rdz_j\ \text{ for } j=1,\ldots , m. \end{aligned}$$

Then, ${\mathcal {N}}=\ker {\mathcal {R}}$ and $I_0(x_0)={\mathcal {I}}_{{\mathcal {N}},x_0}$.

Now, ${\mathcal {R}}^{(1)}$ is generated by ${\mathcal {R}}$ and by the set

$$\begin{aligned} \{df\,\ f\in {\mathcal {I}}(Z({\mathcal {I}}_{{\mathcal {N}}}))\}. \end{aligned}$$

Therefore, ${\mathcal {R}}^{(1)}_{x_0}$ contains ${\mathcal {R}}_{x_0}$ and $\{df\,\ f\in I_{0}(x_0)\}$; as ${\mathcal {D}}^{(1)}=\ker {\mathcal {R}}^{(1)}$, when we compute ${\mathcal {I}}_{{\mathcal {D}}^{(1)}}$, by definition we consider all the determinants of the matrices obtained using as rows the coefficients of elements of ${\mathcal {R}}^{(1)}$. It is clear that this includes all the expressions of the form

$$\begin{aligned} \star \partial f_1\wedge {\overline{\partial }}f_1\wedge \ldots \wedge \partial f_j\wedge {\overline{\partial }}f_j\wedge \partial r\wedge {\overline{\partial }}r\wedge (\partial {\overline{\partial }}r)^{n-1-j} \end{aligned}$$

with $j\in {\mathbb {N}}, f_1,\ldots , f_j\in I_{0}(x_0)$. Therefore,

$$\begin{aligned} (I_0(x_0), A_0(x_0))\subseteq {\mathcal {I}}_{{\mathcal {D}}^{(1)},x_0}, \end{aligned}$$

which implies the thesis for $k=1$.

To iterate the argument, it is enough to recall that $\root c \of {{\mathcal {I}}}\subseteq {\mathcal {I}}(Z({\mathcal {I}}))$. $\square$

Remark 4.2

• (1) If the defining function r can be chosen in a suitable subring ${\mathcal {A}}$ of ${\mathcal {C}}^\infty$, then the ideals $I_k(x_0)$ can be considered as ideals in ${\mathcal {A}}$.

• (2) In some cases, for example, when ${\mathcal {A}}$ is the ring of germs of holomorphic or real analytic functions, we know that $\root c \of {{\mathcal {I}}}={\mathcal {I}}(Z({\mathcal {I}}))$ and then the reduction to the core coincides with Kohn’s algorithm.

• (3) We recall that in the particular cases mentioned in the previous point, the ring ${\mathcal {A}}$ is Noetherian, hence, around each point, the core is reached in a finite number of steps.

We would like to give an example of a non-Noetherian ring of functions where nonetheless a Nullstellensatz-type result holds, allowing us to conclude, as in the previous remark, that the reduction to the core and Kohn’s algorithm are two instantiations of the same idea.

Definition 4.3

Let ${\mathcal {C}}$ be a sheaf such that, for every open set U,

• (1) ${\mathcal {C}}(U)$ is a ${\mathbb {R}}$-subalgebra of ${\mathcal {C}}^\infty (U)$.

• (2) ${\mathcal {C}}^\omega (U)\subseteq {\mathcal {C}}(U)$

• (3) ${\mathcal {C}}$ is closed under composition with mappings whose components are in ${\mathcal {C}}$

• (4) ${\mathcal {C}}$ is closed under differentiation

• (5) ${\mathcal {C}}$ is quasianalytic, i.e., if $f\in {\mathcal {C}}(U)$, $a\in U$ and the Taylor series of f at a is identically zero, then $f\equiv 0$ in a neighborhood of a

• (6) ${\mathcal {C}}$ is closed under division by a coordinate

• (7) ${\mathcal {C}}$ is closed under inverse and hence satisfies the implicit function theorem.

An example of such a ${\mathcal {C}}$ is a Denjoy-Carleman quasianalytic class (see [5] for the basic definitions, [16] for the classical theory and [21] for a more general approach).

It follows from [5] that the resolution of singularities holds for finitely generated ideals in ${\mathcal {C}}$; in the same paper (Theorems 6.1 and 6.3), the authors also prove what is known as topological (or geometric) Noetherianity and Lojasiewicz inequalities.

Theorem 4.4

(Topological Noetherianity) A decreasing sequence of germs of ${\mathcal {C}}$-sets stabilizes in a finite number of steps.

Theorem 4.5

(Lojasiewicz inequalities) Let $f,g\in {\mathcal {C}}_p$ and suppose that $\{x\,\ g(x)=0\}\subseteq \{x:f(x)=0\}$ as germs in p, then there exist $c, \lambda >0$ such that

$$\begin{aligned} |g(x)|\ge c|f(x)|^\lambda . \end{aligned}$$

In general, if $Z=\{x:f(x)=0\}$, we can find $c,\nu >0$ such that

$$\begin{aligned} |f(x)|\ge c\textrm{dist}(x,Z)^\nu . \end{aligned}$$

Localizing the argument from [2][Theorem 1.1—part (i)] and employing the two results above, we obtain the following Nullstellensatz-type theorem for the class ${\mathcal {C}}$.

Theorem 4.6

If ${\mathcal {I}}\subseteq {\mathcal {C}}$ is a finitely generated ideal, then

$$\begin{aligned} \root c \of {{\mathcal {I}}}={\mathcal {I}}(Z({\mathcal {I}})). \end{aligned}$$

Moreover, as a consequence of Theorem 4.4, we obtain the following.

Proposition 4.7

Let ${\mathcal {I}}\subset {\mathcal {C}}_p$ be any ideal of germs at the point p. Then, there exists ${\mathcal {J}}\subseteq {\mathcal {I}}$, finitely generated, such that $Z({\mathcal {J}})=Z({\mathcal {I}})$.

Proof

If ${\mathcal {I}}$ is the zero ideal, it is finitely generated. If not, there is $g_1\in {\mathcal {I}}$ such that $(g_1)\ne 0$; if $Z((g_1))=Z({\mathcal {I}})$, we set ${\mathcal {J}}=(g_1)$, otherwise we find $g_2\in {\mathcal {I}}$ such that $Z((g_1,g_2))\ne Z((g_1))$. If $Z((g_1,g_2))=Z({\mathcal {I}})$, we set ${\mathcal {J}}=(g_1,g_2)$, otherwise we continue as before.

If we never obtain that $Z((g_1,\ldots , g_k))=Z({\mathcal {I}})$, we produce a sequence of ideals

$$\begin{aligned} (g_1)\subseteq (g_1,g_2)\subseteq (g_1,g_2,g_3)\subseteq \ldots . \end{aligned}$$

By Theorem 4.4, the sequence of the zero loci stabilizes after a finite number of steps; this means that

$$\begin{aligned} Z((g_1,\ldots , g_k))=Z((g_1,\ldots , g_k, g_{k+1})) \end{aligned}$$

which contradicts the construction of the sequence $\{g_k\}$. Therefore, the previous construction will stop in a finite number of steps, giving

$$\begin{aligned} Z({\mathcal {J}})=Z({\mathcal {I}}) \end{aligned}$$

with ${\mathcal {J}}=(g_1,\ldots , g_k)$. $\square$

Combining these two results and recalling the remark after Theorem 4.1, we obtain that, if $r\in {\mathcal {C}}$, then ${\mathcal {D}}^{(k)}_c={\mathcal {D}}^{(k)}$ for all $k\in {\mathbb {N}}$; therefore,

$$\begin{aligned} I_k(x_0)\cap {\mathcal {C}}_{x_0}=I_{{\mathcal {D}}^{(k)},x_0}\cap {\mathcal {C}}_{x_0}. \end{aligned}$$

As a consequence of Theorem 4.4, the reduction to the core and Kohn’s algorithm terminate in a finite number of steps; moreover, Kohn’s algorithm gives a subelliptic estimate if and only if the core is trivial.

Remark 4.8

We notice that the results we employed here are also presented in [15], where they are used to prove that Kohn’s algorithm terminates successfully if and only if the boundary is of finite D’Angelo type for domains defined by a function in a Denjoy-Carleman quasianalytic class.

5 Other examples of cores

The definition of core can be specialized in a number of interesting situations.

• (1) Given a complex manifold M and a plurisubharmonic function $\phi :M\rightarrow {\mathbb {R}}$, we define the distribution ${\mathcal {N}}_\phi$ given by the kernel of $\partial {\overline{\partial }}\phi$ and its core ${\mathfrak {C}}({\mathcal {N}}_\phi )$ will be referred to as the core of $\phi$.

• (2) Given a weakly complete complex manifold M, let ${\mathcal {E}}$ be the set of (smooth) plurisubharmonic exhaustion functions; we define

  $$\begin{aligned} {\mathcal {N}}_{{\mathcal {E}}}=\bigcap _{\phi \in E}{\mathcal {N}}_\phi \end{aligned}$$

  and the weakly complete core as ${\mathfrak {C}}({\mathcal {N}}_{{\mathcal {E}}})$.

• (3) Given a weakly complete complex manifold M, let ${\mathcal {E}}$ be the set of (smooth) plurisubharmonic exhaustion functions; we define

  $$\begin{aligned} {\mathcal {D}}_{{\mathcal {E}}}=\ker \{d\phi \}_{\phi \in E} \end{aligned}$$

  and the psh core as ${\mathfrak {C}}({\mathcal {D}}_{{\mathcal {E}}})$.

Remark 5.1

Given a domain in ${\mathbb {C}}^{n+1}$ of the form

$$\begin{aligned} \{(z',z_n)\in {\mathbb {C}}^n\times {\mathbb {C}}\,:\ \Re z_n<\phi (z')\} \end{aligned}$$

for $\phi :{\mathbb {C}}^n\rightarrow {\mathbb {R}}$, there is a natural identification between $S_{{\mathfrak {C}}({\mathcal {N}}_\phi )}$ and $S_{{\mathfrak {C}}({\mathcal {N}})}$.

Proposition 5.2

We have that $S_{{\mathcal {N}}_{{\mathcal {E}}}}=S_{{\mathcal {N}}'_{{\mathcal {E}}}}=S_{{\mathcal {D}}_{{\mathcal {E}}}}=S_{{\mathcal {D}}'_{{\mathcal {E}}}}$; therefore, the derived distributions ${\mathcal {N}}'_{{\mathcal {E}}}$ and ${\mathcal {D}}'_{{\mathcal {E}}}$ are perfect, hence they coincide with their core.

Proof

From [4, Proposition 4.2], we have that $S_{{\mathcal {N}}_{{\mathcal {E}}}}=S_{{\mathcal {D}}_{{\mathcal {E}}}}=\Sigma _M$ is the minimal kernel of M, as defined in [19]; therefore, by [19, Lemma 3.1], there exists a minimal function, i.e., a smooth plurisubharmonic exhaustion function $\phi :M\rightarrow {\mathbb {R}}$ such that $\phi$ is strictly plurisubharmonic outside of $\Sigma _M$.

By repeated applications of [12, Lemma 3.4], given $\phi _1,\ldots , \phi _k\in {\mathcal {E}}$, and real numbers $c_0$,$\ldots$, $c_k$, we have that

$$\begin{aligned} \Sigma _M\cap \{\phi =c_0,\ \phi _1=c_1,\ldots ,\ \phi _k=c_k\} \end{aligned}$$

is a local maximum set, if not empty. We need the following property of local maximum sets: if K is a local maximum set, then $T_xK$ contains at least a complex line for each $x\in K$. We prove it separately as Lemma 5.3, after the end of this proof.

Given $p\in S_{{\mathcal {D}}_{{\mathcal {E}}}}$, we have that there exist $\phi _1,\ldots , \phi _k\in {\mathcal {E}}$ such that

$$\begin{aligned} {\mathcal {D}}_{{\mathcal {E}},p}=(\ker d\phi _1\cap \ldots \cap \ker d\phi _k)\vert _p. \end{aligned}$$

Define

$$\begin{aligned} F=S_{{\mathcal {D}}_{{\mathcal {E}}}}\cap \{x\in M\,:\ \phi (x)=\phi (p),\ \phi _1(x)=\phi _1(p),\ \ldots ,\ \phi _k(x)=\phi _k(p)\}; \end{aligned}$$

as we noted before, F is a local maximum set (it contains at least p, so it is not empty) and $T_pF$ contains at least a copy of ${\mathbb {C}}$. Note that

$$\begin{aligned} T_pF\subseteq T_pS_{{\mathcal {D}}_{{\mathcal {E}}}}\cap (\ker d\phi _1\cap \ldots \cap \ker d\phi _k)\vert _p=T_pS_{{\mathcal {D}}_{{\mathcal {E}}}}\cap {\mathcal {D}}_{{\mathcal {E}},p}={\mathcal {D}}'_{{\mathcal {E}},p}. \end{aligned}$$

Therefore, $p\in S_{distr'_{{\mathcal {E}}}}$.

Let now $p\in S_{{\mathcal {N}}_{{\mathcal {E}}}}$. By [18, Theorem 5.2],

$$\begin{aligned} S_{{\mathcal {N}}_{{\mathcal {E}}}}=\Sigma _M=\bigcup _{\alpha \in A}F_\alpha \end{aligned}$$

where each $F_\alpha$ is a compact local maximum set on which every $\phi \in {\mathcal {E}}$ is constant.

By adapting the proof of [8, Theorem 3.13], we can show that, for every $\phi \in {\mathcal {E}}$, $\ker \partial {\overline{\partial }}\phi \cap TF_\alpha \cap JTF_\alpha$ has support equal to $F_\alpha$.

Therefore, the support of $TF_\alpha \cap \ker \partial {\overline{\partial }}\phi \subseteq TS_{{\mathcal {N}}_{{\mathcal {E}}}}\cap \ker \partial {\overline{\partial }}\phi$ contains $F_\alpha$ for all $\alpha \in A$ and all $\phi \in {\mathcal {E}}$, hence the support of ${\mathcal {N}}'_{{\mathcal {E}}}=TS_{{\mathcal {N}}_{{\mathcal {E}}}}\cap {\mathcal {N}}_{{\mathcal {E}}}$ contains $F_\alpha$ for all $\alpha \in A$, i.e., coincides with the support of ${\mathcal {N}}_{{\mathcal {E}}}$. $\square$

Lemma 5.3

If $K\subset M$ is a local maximum set in a complex manifold, then $T_xK$ contains at least a complex line, for all $x\in {\overline{K}}$.

Proof

If $\dim _{\mathbb {R}}T_xK>\dim _{\mathbb {C}}M$, then obviously $T_xK\cap JT_xK\ne \{0\}$ by Grassmann’s formula. So, let us suppose that $k=\dim _{\mathbb {R}}T_xK\le n$.

Now, let (V, x) be a germ of k-dimensional real submanifold such that $(K,x)\subseteq (V,x)$ and $T_xK=T_xV$; if $T_xK$ is totally real, then $T_yV$ is totally real for all y in a neighborhood of x in V. As (V, x) is a germ of totally real submanifold, we can find a psh function u, defined on a small neighborhood $U\subseteq X$ of x, such that $u(x)=0$ and $u(y)<0$ for all $y\in U\cap V$. But, as $(K,x)\subseteq (V,x)$, up to shrinking U, we can suppose that $K\cap U\subseteq V\cap U$, so that u would violate the local maximum property of K.

Therefore, $T_xK$ cannot be totally real, so it contains at least a complex line. $\square$

The support of ${\mathcal {N}}_{{\mathcal {E}}}$ is the so called minimal kernel of a weakly complete space; its geometry has been studied in a number of papers by the second author and others (see, e.g., [10,11,12,13, 18, 19] and the references therein). This “weak integrability property" given by Proposition 5.2 surely hints at the existence of some analytic structure, which, up to now, has been found only in the case of complex surfaces.

6 Rigid domains

In this section, we discuss the relation between Catlin’s property (P) and the triviality of the core (see also [22]).

Let $h:{\mathbb {C}}^n\rightarrow {\mathbb {R}}$ be a ${\mathcal {C}}^2$ function and define

$$\begin{aligned} \Omega _h=\{(z',z_{n+1})\in {\mathbb {C}}^n\times {\mathbb {C}}\,:\ \Re (z_{n+1})+h(z')<0\}. \end{aligned}$$

Let $X=b\Omega _h$, then for $p=(p',p_{n+1})\in X$,

$$\begin{aligned} T_p^{1,0}X=\{Y'_p+\gamma _p\partial _{z_{n+1}}\,:\ Y'_p\in T^{1,0}_{p'}{\mathbb {C}}^n,\ 2\partial h (Y'_p)+\gamma _p=0\}. \end{aligned}$$

In particular, the map $T^{1,0}_pX\ni (Y'_p+\gamma \partial _{z_{n+1}})\mapsto Y'_p\in T^{1,0}_{p'}{\mathbb {C}}^n$ is a ${\mathbb {C}}$-linear isomorphism, which gives an isomorphism of complex vector bundles from $T^{1,0}X$ to $T^{1,0}{\mathbb {C}}^n$.

Moreover, the Levi form on X is

$$\begin{aligned} \lambda _p(Y'_p+\gamma _p\partial _{z_{n+1}}, {\overline{Z}}'_p+{\overline{\delta }}_p\partial _{{\bar{z}}_{n+1}})=i\partial {\overline{\partial }}h_p (Y'_p, {\overline{Z}}'_p) \end{aligned}$$

so it corresponds, under said isomorphism, to the complex Hessian of h on $T^{1,0}{\mathbb {C}}^n$. Hence, $\Omega _h$ is pseudoconvex if and only if h is plurisubharmonic.

Moreover, ${\mathfrak {C}}({\mathcal {N}})$ in $T^{1,0}X$ corresponds, under such an isomorphism to ${\mathfrak {C}}({\mathcal {N}}_h)$ in $T^{1,0}C^n$.

As a special case, when $n=1$, we have that $h:{\mathbb {C}}\rightarrow {\mathbb {R}}$ should be subharmonic; given $K\subseteq {\mathbb {C}}$ closed, we can find a ${\mathcal {C}}^\infty$ function $g:{\mathbb {C}}\rightarrow [0,+\infty )$ such that $K=\{z\in {\mathbb {C}}\,\ g(z)=0\}$.

Solving the equation $\triangle h=g$, we obtain a subharmonic function whose Laplacian vanishes exactly on K, so that ${\mathcal {N}}_h=T^{1,0}{\mathbb {C}}\vert _K$.

At any point $p\in K$, ${\mathcal {N}}_{h,p}\cap {\mathbb {C}}T_p K$ can be either ${\mathcal {N}}_{h,p}$ or $\{0\}$, depending whether $T_pK$ is ${\mathbb {C}}$ or not (respectively).

Example 6.1

Let $C\subset {\mathbb {R}}$ be a Cantor set in [0, 1] and define $K=C+iC\subseteq {\mathbb {C}}$; we consider the function h described above and the corresponding rigid domain $\Omega _h\subseteq {\mathbb {C}}^2$.

As the Cantor set is a perfect set, $T_xC={\mathbb {R}}$ for all $x\in {\mathbb {R}}$, so that $T_zK={\mathbb {C}}$ for all $z\in K$; hence ${\mathcal {N}}_h$ is a perfect distribution in $T^{1,0}{\mathbb {C}}$ and so is the Levi null distribution ${\mathcal {N}}$ on $b\Omega _h$.

Therefore, the Levi core of $b\Omega _h$ is non-trivial and supported on the set

$$\begin{aligned} \{(z,w)\in b\Omega _h\,:\ z\in K\}\cong K\times {\mathbb {R}}. \end{aligned}$$

We notice that the core here is "small," in the sense that, as the Lebesgue measure of C in ${\mathbb {R}}$ is zero, the 3-dimensional Hausdorff measure of $S_{{\mathfrak {C}}({\mathcal {N}})}$ is zero inside $b\Omega _h$.

A bounded version of $\Omega _h$ is easily obtained. Let

$$\begin{aligned} \Omega '_h=\{(z',z_{n+1})\in {\mathbb {C}}^n\times {\mathbb {C}}\,:\ \Re (z_{n+1})+h(z')+ \lambda (\Vert z'\Vert ^2+|z_{n+1}|^2)<0\}, \end{aligned}$$

where $\lambda$ is a nonnegative, increasing, convex function such that $\lambda (t)=0$ if $t\le 2$ and $\lambda (t),\lambda '(t),\lambda ''(t)>0$ if $t>2$. We have that $\Omega _h'$ is pseudoconvex and bounded and strictly pseudoconvex if $\Vert z'\Vert ^2+|z_{n+1}|^2>2$; moreover, $b\Omega _h\cap B_{\sqrt{2}}(0,0)\subseteq b\Omega _h'$; so, the support of the core of $\Omega _h'$ is contained in

$$\begin{aligned} \{(z,w)\in b\Omega _h\,:\ z\in K\}\cap \cap B_{\sqrt{2}}(0,0). \end{aligned}$$

Example 6.2

Let $C\subset [0,1]$ be a Cantor set such that

$$\begin{aligned} C=\left( \frac{1}{5}C + 0 \right) \cup \left( \frac{1}{5}C + \frac{4}{5}\right) \end{aligned}$$

then $\dim _H C=\log (2)/\log (5)<1/2$ ($\dim _H$ is the Hausdorff dimension). This means that $\dim _H(C\times C\times {\mathbb {R}})<2$.

Constructing $\Omega _h'$ on the set $K=C+iC$, we obtain that the support of the core has vanishing 2-dimensional Hausdorff measure; by [17, Remarque p.310], this implies that $U_{\phi ,\epsilon }$ is B-regular, or, in an equivalent formulation, that it satisfies Catlin’s Property (P).

Therefore, we showed that Catlin’s property (P) (or B-regularity) does not imply trivial core.

Unfortunately, these examples all have trivial D’Angelo class: the complex Hessian of the defining function vanishes completely at the points of the core, so every D’Angelo form is zero. Hence, the Diederich-Fornaess exponent of the domains from the two previous examples is 1.

If we assume some more properties of the set K, we can produce other kinds of examples. Suppose that K is a compact made up of points which are regular for the Dirichlet problem, then there exists a smooth function $u:{\mathbb {C}}\setminus K \rightarrow (0,+\infty )$ such that $u(x)\rightarrow 0^+$ when ${\mathbb {C}}{\setminus } K\ni x\rightarrow K$, u has a logarithmic pole at $\infty$, and $\triangle u=0$ on ${\mathbb {C}}\setminus K$, i.e., u is the Green function with pole at $\infty$ for ${{\mathbb {C}}}{{\mathbb {P}}}^1\setminus K$.

Let $\lambda :[0,+\infty )\rightarrow {\mathbb {R}}$ be a nonnegative, convex, increasing function such that $\lambda (t)$ vanishes so quickly for $t\rightarrow 0^+$ such that $\lambda \circ u$ extends to 0 smoothly on K. Let

$$\begin{aligned} \Omega _{K,c}=\{(z,w)\in {\mathbb {C}}^2\,:\ \lambda (u(z))+|w|^2<c\}, \end{aligned}$$

which will be, for almost all c, a smoothly bounded domain. We have that

$$\begin{aligned} \partial {\overline{\partial }}\lambda \circ u(z)=\left| \frac{\partial u}{\partial z}\right| ^2(\lambda ''(u)+\lambda '(u))dz\wedge d{\bar{z}} \end{aligned}$$

so that the complex Hessian of the defining function of $\Omega _{K,c}$ is

$$\begin{aligned} \left| \frac{\partial u}{\partial z}\right| ^2\lambda ''(u)dz\wedge d{\bar{z}}+dw\wedge d{\bar{w}}. \end{aligned}$$

A vector in $T^{1,0}b\Omega _{K,c}$ is of the form

$$\begin{aligned} {\bar{w}}\partial _z - (\partial _z u)\lambda '(u)\partial _w \end{aligned}$$

hence the Levi form of $b\Omega _{K,c}$ is given by

$$\begin{aligned} \left| \frac{\partial u}{\partial z}\right| ^2(|w|^2\lambda ''(u)+\lambda '(u)) \end{aligned}$$

which will vanish if and only if $z\in K$ or $\partial _z u=0$.

Points where $\partial _z u=0$ are of finite type; if K is such that $T_zK={\mathbb {C}}$ for all $z\in K$, then $S_{{\mathfrak {C}}({\mathcal {N}})}\cong K\times {\mathbb {S}}^1$.

Example 6.3

If $C\subseteq [0,1]$ is a suitably constructed Cantor set such that $K=C+iC$ has Hausdorff dimension 1 but ${\mathcal {H}}^1$-measure 0, then the Levi core of $b\Omega _{K,c}$ is supported in a set of vanishing 2-dimensional Hausdorff measure (but of Hausdorff dimension 2).

We note that, for $(z,w)\in S_{{\mathfrak {C}}({\mathcal {N}})}$, a tangent (1, 0)-vector in ${\mathcal {N}}$ has the form

$$\begin{aligned} {\bar{w}}\partial _z, \end{aligned}$$

whereas a (0, 1)-normal vector will be a multiple of

$$\begin{aligned} {\bar{w}}\partial _w; \end{aligned}$$

therefore, as the complex Hessian of the defining function is diagonal, we once again get that the D’Angelo class is trivial (as it contains the null 1-form).