Equivariant asymptotics of Szegö kernels under Hamiltonian ${{\varvec{U}}}(\mathbf{2})$-actions

Abstract

Let M be complex projective manifold and A a positive line bundle on it. Assume that a compact and connected Lie group G acts on M in a Hamiltonian manner and that this action linearizes to A. Then, there is an associated unitary representation of G on the associated algebro-geometric Hardy space. If the moment map is nowhere vanishing, the isotypical components are all finite dimensional; they are generally not spaces of sections of some power of A. One is then led to study the local and global asymptotic properties the isotypical component associated with a weight $k \, \varvec{ \nu }$, when $k\rightarrow +\infty$. In this paper, part of a series dedicated to this general theme, we consider the case $G=U(2)$.

1 Introduction

In many interesting and natural situations, an Hamiltonian action of a Lie group G on a Hodge manifold can be linearized to a polarizing positive line bundle; when this happens, there is an induced unitary representation of G on a certain Hardy space, intrinsically related to the holomorphic structure of the line bundle. One is then led to investigate the decomposition of the latter Hardy space into isotypical components over the irreducible representations of G and how this decomposition reflects the geometry of the underlying action. In particular, if the corresponding moment map is never vanishing, then all the isotypical components are finite dimensional.

For example, in the very special case where $G=S^1$ acts trivially on M and the moment map is taken to be $\Phi _G=1$, the corresponding isotypical components are (naturally isomorphic to) the spaces of global holomorphic sections of powers of A. In general, however, the isotypical components in point do not correspond to subspaces of holomorphic sections of some higher tensor power of the polarizing line bundle; in other words, they generally split non-trivially under the structure $S^1$-action on X.

From the point of view of geometric quantization, the most appropriate heuristic framework for the present discussion is the setting of ‘homogeneous’  quantization treated in [13] (and of course [3]). In fact, a motivation for the present analysis is to revisit the general theme of [13] in the specific context of Toeplitz quantization (in the sense of [3]) by means of the approach to algebro-geometric Szegö kernels developed in [2, 30, 34]; this circle of ideas is ultimately based on the microlocal theory of the Szegö kernel as an FIO developed in [4].

In this work, we shall consider the case $G=U(2)$ and focus on the asymptotics of the isotypical components pertaining to a given ladder representation, in the terminology of [13]. In other words, we shall fix a ray in weight space and study the asymptotic behavior of the isotypes when the representation drifts to infinity along the ray. When G is a torus, this problem was studied in [6, 26, 27]; the case $G=SU(2)$ is the object of [10]. To make this more precise, it is in order to set the geometric stage in detail.

Let M be a connected d-dimensional complex projective manifold, with complex structure J. Let (A, h) be a positive line bundle on M; in other words, A is an holomorphic ample line bundle on M, h is an Hermitian metric on A, and the curvature form of the unique covariant derivative $\nabla$ on A compatible with both the complex and Hermitian structures has the form $\Theta = -2\,\imath \,\omega$, where $\omega$ is a Kähler form on M. We shall denote by $\rho$ the corresponding Riemannian structure on M, given by

$$\begin{aligned} \rho _m (v,w) := \omega _m \big (v, J_m(w)\big ) \quad (m\in M,\, v,w\in T_mM). \end{aligned}$$

(1)

If $A^\vee \supset X{\mathop {\rightarrow }\limits ^{\pi }} M$ is the unit circle bundle in the dual of A, then $\nabla$ naturally corresponds to a connection 1-form $\alpha$ on X, such that $\mathrm {d}\alpha = 2\,\pi ^*(\omega )$. Hence, $(X,\alpha )$ is a contact manifold.

We shall adopt

$$\begin{aligned} \mathrm {d}V_M := \frac{1}{d!} \, \omega ^{\wedge d} \quad {\mathrm {and}} \quad \mathrm {d}V_X := \frac{1}{ 2\pi }\,\alpha \wedge \pi ^* \left( \mathrm {d}V_M \right) \end{aligned}$$

(2)

as volume forms on M and X, respectively; integration will always be meant with respect to the corresponding densities.

Furthermore, $\alpha$ determines an invariant splitting of the tangent bundle of X as

$$\begin{aligned} {\textit{TX}}=\mathcal {V} (X / M )\oplus \mathcal {H}(X/M), \end{aligned}$$

(3)

where $\mathcal {V} (X / M ) := \ker ( \mathrm {d} \pi )$ is the vertical tangent bundle, and $\mathcal {H}(X/M) := \ker ( \alpha )$ is the horizontal tangent bundle. Given $V\in \mathfrak {X}(M)$ (the Lie algebra of smooth vector fields on M), we shall denote by $V^\sharp \in \mathfrak {X}(X)$ its horizontal lift to X. If the vector field $\partial / \partial \theta \in \mathfrak {X}(X)$ is the generator of the structure $S^1$-action, then $\partial _\theta$ spans $\mathcal {V} (X / M )$, and $\langle \alpha , \partial _\theta \rangle =1$.

The holomorphic structure on M, pulled-back to $\mathcal {H}(X/M)$, endows X with a CR structure. Explicitly, the complex structure J on M naturally lifts to a vector bundle endomorphism of TX, also denoted by J, such that $J (\partial _\theta )=0$ and

$$\begin{aligned} J\left( \upsilon ^\sharp \right) = J(\upsilon )^\sharp \quad \big (\upsilon \in \mathfrak {X}(M)\big ). \end{aligned}$$

(4)

The corresponding Hardy space $H(X)\subset L^2(X)$ encapsulates the holomorphic structure of A and its tensor powers. The corresponding orthogonal projector and its distributional kernel are called, respectively, the Szegö projector and the Szegö kernel of X; they will be denoted

$$\begin{aligned} \Pi : L^2(X) \rightarrow H( X ), \quad \Pi (\cdot , \cdot ) \in \mathcal {D}'( X\times X ). \end{aligned}$$

(5)

Consider the unitary group U(2), and its Lie algebra $\mathfrak {u} ( 2 )$, the space of skew-Hermitian $2\times 2$ matrices; in the following, we shall set $G = U ( 2 )$ and $\mathfrak {g} = \mathfrak {u} ( 2 )$ for notational convenience. The standard invariant scalar product $\langle \beta _1, \beta _2 \rangle _{\mathfrak {g}} := {\mathrm {trace}} \left( \beta _1 \, \overline{ \beta }_2 ^t \right)$ yields a unitary isomorphism $\mathfrak {g} \cong \mathfrak {g}^\vee$ intertwining the adjoint and coadjoint representations of G.

Suppose given an holomorphic Hamiltonian action $\mu : G\times M \rightarrow M$ on the Kähler manifold $(M, J , 2\,\omega )$, with moment map $\Phi _G : M \rightarrow \mathfrak {g}^\vee \cong \mathfrak {g}$. For every $\xi \in \mathfrak {g}$, let $\xi _M \in \mathfrak {X} ( M )$ be its associated vector field on M. Then,

$$\begin{aligned} \xi _X := \xi _M ^\sharp - \langle \Phi _G, \xi \rangle \, \partial _\theta \end{aligned}$$

(6)

is a contact vector field on $(X,\alpha )$ [20], and the map $\xi \mapsto \xi _X$ is an infinitesimal action of $\mathfrak {g}$ on $(X, \alpha )$.

We shall assume that the latter infinitesimal action can be integrated to an action ofGonX, i.e., that $\mu$ lifts to an action $\widetilde{\mu } : G\times X\rightarrow X$ preserving the contact and CR structures. Then, pull-back of functions, given by $g\cdot s := \widetilde{\mu }_{g^{-1}}^* ( s )$, is a unitary representation of G on $L^2(X)$ leaving $H(X) \subset L^2( X )$ invariant. This yields a unitary representation

$$\begin{aligned} \widehat{\mu } : G \rightarrow U \big ( H(X) \big ). \end{aligned}$$

(7)

By the Theorem of Peter and Weyl [5, 31], H(X) decomposes as a Hilbert space direct sum of finite-dimensional irreducible representations of G. The latter are in 1:1 correspondence with the pairs $\varvec{ \nu } = (\nu _1, \nu _2 )$ of integers satisfying $\nu _1 > \nu _2$ [33]; namely, $\varvec{\nu }$ corresponds to the irreducible representation

$$\begin{aligned} V_{ \varvec{\nu } } := { \det } ^{ \nu _2 } \otimes {\mathrm {Sym}}^{\nu _1 - \nu _2 -1} \left( \mathbb {C} ^2 \right) ; \end{aligned}$$

(8)

the restriction of its character $\chi _{\varvec{\nu }}$ to the standard torus $T\leqslant G$ is given by

$$\begin{aligned} \chi _{\varvec{\nu }}: \begin{pmatrix} t_1&{}\quad 0\\ 0&{}\quad t_2 \end{pmatrix} \mapsto \frac{t_1^{\nu _1}\,t_2^{\nu _2}-t_1^{\nu _2}\,t_2^{\nu _1}}{t_1-t_2}. \end{aligned}$$

(9)

Therefore, there is an equivariant unitary isomorphism

$$\begin{aligned} H(X) \cong \bigoplus _{ \nu _1>\nu _2 } H(X)_{ \varvec{ \nu } }, \end{aligned}$$

where $H(X)_{ \varvec{ \nu } } \subseteq H(X)$ is the $\varvec{ \nu }$-isotypical component. Correspondingly,

$$\begin{aligned} \Pi = \sum _{ \nu _1 > \nu _2} \Pi _{ \varvec{ \nu } }, \end{aligned}$$

(10)

where $\Pi _{ \varvec{ \nu }}: L^2(X) \rightarrow H(X)_{ \varvec{ \nu } }$ is the orthogonal projector (recall (5)).

In general, $H(X)_{ \varvec{ \nu } }$ may well be infinite dimensional; however, if $\mathbf {0}\not \in \Phi _G(M)$, then $\dim \big ( H(X)_{ \varvec{ \nu } } \big ) < +\infty$ for every $\varvec{ \nu }$ (see §2 of [26]). In this case, each $\Pi _{ \varvec{ \nu } }$ is a smoothing operator, with a distributional kernel

$$\begin{aligned} \Pi _{ \varvec{ \nu } } (\cdot , \cdot ) \in \mathcal {C}^\infty (X \times X ). \end{aligned}$$

(11)

In particular,

$$\begin{aligned} \dim H(X)_{ \varvec{ \nu } } = \int _X \Pi _{ \varvec{ \nu } } ( x, x ) \, \mathrm {d}V_X (x). \end{aligned}$$

(12)

Let us fix a weight $\varvec{ \nu } \in \mathbb {Z}^2 {\setminus } \{\mathbf {0}\}$, and look at the concentration behavior of $\Pi _{ k \varvec{ \nu } } (\cdot , \cdot )$ when $k\rightarrow +\infty$. The Abelian analog of this problem was studied in [26] and [27].

Definition 1.1

If $\varvec{ \nu } \in \mathbb {Z}^2$, let

$$\begin{aligned} D_{ \varvec{ \nu } } := \begin{pmatrix}\nu _1 &{}\quad 0\\ 0&{}\quad \nu _2\end{pmatrix}. \end{aligned}$$

Let us introduce the following loci.

• 1. $\mathcal {O}_{ \varvec{ \nu } }\subset \mathfrak {g}$ is the (co)adjoint orbit of $\imath \, D_{ \varvec{ \nu } }$;

• 2. $\mathcal {C} (\mathcal {O}_{ \varvec{ \nu } } ) := \mathbb {R}_+\cdot \mathcal {O}_{ \varvec{ \nu } }$ is the cone over $\mathcal {O}_{ \varvec{ \nu } }$;

• 3. in M and X, respectively, we have the inverse images

  $$\begin{aligned} M^G_{\mathcal {O}_{ \varvec{ \nu } }} := \Phi _G^{ -1 }\big ( \mathcal {C} (\mathcal {O}_{ \varvec{ \nu } } ) \big ), \quad X^G_{\mathcal {O}_{ \varvec{ \nu } }} := \pi ^{-1} \left( M^G_{\mathcal {O}_{ \varvec{ \nu } }} \right) . \end{aligned}$$

We shall occasionally write $\mathcal {O}$ in place of $\mathcal {O}_{ \varvec{ \nu } }$. Finally, let us define $\mathcal {C}^\infty$ functions

$$\begin{aligned} m\in M^G_{\mathcal {O}_{ \varvec{ \nu }}}\mapsto h_m\,T\in G/T,\quad m\in M^G_{\mathcal {O}_{ \varvec{ \nu }}}\mapsto \lambda _{\varvec{\nu }}(m)\in (0,+\infty ) \end{aligned}$$

by the equality

$$\begin{aligned} \Phi _G(m)=\imath \,\lambda _{\varvec{\nu }}(m)\,h_m D_{\varvec{\nu }}\, h_m^{-1}. \end{aligned}$$

(13)

Our first result is the following.

Theorem 1.1

Assume that $\mathbf {0}\not \in \Phi _G(M)$, and $\Phi _G$ is transverse to $\mathcal {C} (\mathcal {O}_{ \varvec{ \nu } } )$. Let us define the $G\times G$-invariant subset of $X\times X$

$$\begin{aligned} \mathcal {Z}_{\varvec{ \nu }}:= \left\{ ( x, y ) \in X^G_{\mathcal {O}_{ \varvec{ \nu } }}\times X^G_{\mathcal {O}_{ \varvec{ \nu } }}\, : \, y \in G \cdot x \right\} . \end{aligned}$$

Then, uniformly on compact subsets of $(X\times X) {\setminus } \mathcal {Z}_{\varvec{ \nu }}$, we have

$$\begin{aligned} \Pi _{ k\varvec{ \nu } } (x,y )= O\left( k^{ -\infty } \right) . \end{aligned}$$

Corollary 1.1

Uniformly on compact subsets of $X{\setminus } X^G_{\mathcal {O}_{ \varvec{ \nu } }}$, we have

$$\begin{aligned} \Pi _{ k\varvec{ \nu } } (x,x )= O\left( k^{ -\infty } \right) \quad {\mathrm {for}}\quad k\rightarrow +\infty \end{aligned}$$

The hypothesis of Theorem 1.1 implies that $M^G_{\mathcal {O}_{ \varvec{ \nu } }}$ is a compact and smooth real hypersurface of M. Our next step will be to clarify the geometry of $M^G_{\mathcal {O}_{ \varvec{ \nu } }}$. To this end, we need to introduce some further loci related to the action.

Definition 1.2

Let

$$\begin{aligned} M^G_{ \varvec{ \nu } } := \Phi _G^{ -1 }\big ( \imath \,\mathbb {R}_+ \cdot D_{ \varvec{ \nu } } \big ), \quad X^G_{ \varvec{ \nu } } := \pi ^{-1}\left( M^G_{ \varvec{ \nu } } \right) . \end{aligned}$$

(14)

Remark 1.1

Obviously, $M^G_{ \varvec{ \nu } }\subseteq M^G_{\mathcal {O}_{ \varvec{ \nu } }}$. Under the assumptions of Theorem 1.1, $M^G_{ \varvec{ \nu } }$ is a compact submanifold of M, of real codimension 3. Clearly, $M^G_{\mathcal {O}_{ \varvec{ \nu } }}= G \cdot M^G_{ \varvec{ \nu } }$ by the equivariance of $\Phi _G$ (given a G-space Z, and a subset $Z_1\subseteq Z$, we shall denote by $G\cdot Z_1$ the G-saturation of $Z_1$ in Z).

Let $T\leqslant G$ be the standard maximal torus of unitary diagonal matrices, and let $\mathfrak {t}$ be its Lie algebra. Thus, $\mathfrak {t}$ is the space of skew-Hermitian diagonal matrices and is also T-equivariantly identified with the coalgebra $\mathfrak {t}^\vee$. In obvious manner $T\cong S^1\times S^1$ and $\mathfrak {t} \cong \imath \, \mathbb {R}^2$. We shall alternatively think of elements of $\mathfrak {t}$ either as vectors or as matrices, depending on the context.

Given the isomorphisms $\mathfrak {g}^\vee \cong \mathfrak {g}$ and $\mathfrak {t}^\vee \cong \mathfrak {t}$, the restriction epimorphism $\mathfrak {g} ^\vee \rightarrow \mathfrak {t} ^\vee$ corresponds to the diagonal map

$$\begin{aligned} {\mathrm {diag}} : \mathfrak {g} \rightarrow \imath \, \mathbb {R}^2, \quad \imath \, \begin{pmatrix} a &{}\quad z\\ \overline{z} &{}\quad b \end{pmatrix} \mapsto \imath \, \begin{pmatrix} a \\ b \end{pmatrix} \quad (a, b \in \mathbb {R},\, z\in \mathbb {C}). \end{aligned}$$

(15)

The action of T on M induced by restriction of $\mu$ is also Hamiltonian, with moment map

$$\begin{aligned} \Phi _T ={\mathrm {diag}}\circ \Phi _G:M \rightarrow \mathfrak {t}. \end{aligned}$$

(16)

Let us introduce the loci

$$\begin{aligned} M^T_{ \varvec{ \nu }} := \Phi _T^{-1} \left( \mathbb {R}_+\cdot \imath \,\varvec{ \nu }\right) , \quad X^T_{ \varvec{ \nu }} := \pi ^{-1} \left( M^T_{ \varvec{ \nu }}\right) \end{aligned}$$

(17)

Let us assume that $\mathbf {0}\not \in \Phi _T(M)$ and that $\Phi _T$ is transverse to $\mathbb {R}_+\cdot \imath \,\varvec{ \nu }$; then, $M^T_{ \varvec{ \nu }}$ is a compact smooth real hypersurface of M. Since $M^G_{ \varvec{ \nu } } \subseteq M^T_{ \varvec{ \nu } }$, we have $M^G_{\mathcal {O}_{ \varvec{ \nu } }} \subseteq G\cdot M^T_{ \varvec{ \nu }}$.

In Sect. 4.1.2, we shall construct a vector field $\Upsilon =\Upsilon _{\mu ,\varvec{ \nu } }$ tangent to M along $M^G_{\mathcal {O}_{ \varvec{ \nu } }}$, naturally associated with the action and the weight, which is nowhere vanishing and everywhere normal to $M^G_{\mathcal {O}_{ \varvec{ \nu } }}$.

Theorem 1.2

Let us assume that:

• 1. $\Phi _G:M\rightarrow \mathfrak {g}$ and $\Phi _T:M\rightarrow \mathfrak {t}$ are both transverse to $\mathbb {R}_+ \cdot \imath \,D_{\varvec{ \nu }}$;

• 2. $\mathbf {0}\not \in \Phi _T(M)$ (hence also $\mathbf {0}\not \in \Phi _G(M)$);

• 3. $M^G_{ \varvec{ \nu }}\ne \emptyset$ (equivalently, $M^G_{ \mathcal {O}_{ \varvec{ \nu }}}\ne \emptyset$);

• 4. $\nu _1+\nu _2\ne 0$.

Then,

• 1. $M^G_{\mathcal {O}_{ \varvec{ \nu } }}$ is a connected and orientable smooth hypersurface in M and separates M in two connected components: the ‘outside ’  $A := M{\setminus } G\cdot M^T_{ \varvec{ \nu }}$ and the ‘inside’  $B := G\cdot M^T_{ \varvec{ \nu }} {\setminus } M^G_{\mathcal {O}_{ \varvec{ \nu } }}$;

• 2. the normal bundle to $M^G_{\mathcal {O}_{ \varvec{ \nu } }}$ in M is the real line sub-bundle of $\left. TM\right| _{ M^G_{\mathcal {O}_{ \varvec{ \nu } }} }$ spanned by $\Upsilon$;

• 3. $\Upsilon$ is ‘outer’  oriented if $\nu _1+\nu _2>0$ and ‘inner’  oriented if $\nu _1+\nu _2<0$;

• 4. $M^G_{\mathcal {O}_{\varvec{ \nu }}}\cap M^T_{{\varvec{ \nu }}} =M^G_{{\varvec{ \nu }}}$, and the two hypersurfaces meet tangentially along $M^G_{{\varvec{ \nu }}}$.

Remark 1.2

Let us clarify the meaning of the partition $M= A\dot{\cup } M^G_{\mathcal {O}_{\varvec{ \nu }}} \dot{\cup } B$. Clearly, $G\cdot M^T_{ \varvec{ \nu }} = \overline{B}$, $A=\left( G\cdot M^T_{ \varvec{ \nu }} \right) ^c$. For any $m\in M$, let $\mathcal {O}_{\Phi (m)} := \Phi _G ( G \cdot m )$ be the coadjoint orbit of $\Phi _G ( m )$, and let $\lambda _1>\lambda _2$ be the eigenvalues of $-\imath \,\Phi _G(m)$; as follows either by direct verification or by invoking Horn’s Theorem, the projection of $\mathcal {O}_{\Phi (m)}$ in $\mathfrak {t}$ is the segment $J_m$ joining $\imath \, \begin{pmatrix} \lambda _1&\lambda _2 \end{pmatrix}^t$ and $\imath \, \begin{pmatrix} \lambda _2&\lambda _1 \end{pmatrix}^t$. Then, we have:

• 1. $m\in A$ if and only if the orthogonal projection of $\mathcal {O}_{\Phi (m)}$ in $\mathfrak {t}$, ${\mathrm {diag}}( \mathcal {O}_{\Phi (m)} )$, is disjoint from $\imath \,\mathbb {R}_+ \cdot \varvec{ \nu }$;

• 2. $m\in M^G_{\mathcal {O}_{\varvec{ \nu }}}$ if and only if ${\mathrm {diag}}( \mathcal {O}_{\Phi (m)} )\cap (\imath \,\mathbb {R}_+ \cdot \varvec{ \nu })$ is an endpoint of $J_m$;

• 3. $m\in B$ if and only if ${\mathrm {diag}}( \mathcal {O}_{\Phi (m)} )\cap (\imath \,\mathbb {R}_+ \cdot \varvec{ \nu })$ is an interior point of $J_m$.

The next step will be to provide some more precise quantitative information on the rate of decay of $\Pi _{ k\varvec{ \nu } }(\cdot ,\cdot )$ on the complement of $\mathcal {Z}_{\varvec{ \nu }}$. Namely, we shall show that $\Pi _{ k\varvec{ \nu } }(x,y)$ is still rapidly decreasing when either $y\rightarrow G\cdot x$ at a sufficiently slow rate, or when at least one of x and y belongs to the ‘outer’  component A, and converges to $X^G_{\mathcal {O}_{ \varvec{ \nu } }}$ sufficiently slowly.

Let us consider on X the Riemannian structure which is uniquely determined by the following conditions:

• 1. (3) is an orthogonal direct sum;

• 2. $\pi :X\rightarrow M$ is a Riemannian submersion;

• 3. the $S^1$-orbits have unit length.

The corresponding density is $\mathrm {d}V_X$. Let ${\mathrm {dist}}_X :X\times X\rightarrow [0,+\infty )$ denote the associated distance function.

Theorem 1.3

In the situation of Theorem 1.1, assume in addition that G acts freely on $X^G_{\mathcal {O}}$. For any fixed $C,\,\epsilon >0$, we have $\Pi _{ k\varvec{ \nu } }(x,y) = O \left( k^{-\infty } \right)$ uniformly for

$$\begin{aligned} \max \left\{ {\mathrm {dist}}_X ( x, G \cdot y ), \, {\mathrm {dist}}_X \left( x, G \cdot X^T_{ \varvec{ \nu } } \right) \right\} \ge C \, k^{ \epsilon -1/2}. \end{aligned}$$

(18)

Let us clarify the meaning of Theorem 1.3. The closed loci $\mathcal {R}_k\subset X\times X$ defined by (18) form a nested sequence $\mathcal {R}_1 \subseteq \mathcal {R}_2\subseteq \cdots$. For any fixed $C, \epsilon >0$, there exist positive constants $C_j=C_j(C,\epsilon )>0$, $j=1,2,\ldots$, such that the following holds. Given any sequence in $X\times X$ with $( x_k, y_k ) \in \mathcal {R}_k$ for $k=1,2,\ldots$, we have

$$\begin{aligned} \big | \Pi _{ k\varvec{ \nu } }(x_k, y_k ) \big | \le C_j \, k^{ - j} \end{aligned}$$

for every k.

In Theorems 1.4 and 1.5, we shall consider the diagonal and near-diagonal asymptotic behavior of $\Pi _{ k\varvec{ \nu } }$ along $X^G_{\mathcal {O}}$. In the setting of Theorem 1.2, every $x\in X^G_{\mathcal {O}_{ \varvec{ \nu } }}$ has discrete stabilizer subgroup in X. To simplify our exposition, we shall make the stronger assumption that $\widetilde{\mu }$ is actually free along $X^G_{\mathcal {O}_{ \varvec{ \nu } }}$. Before giving the statement, some further notation is needed.

Definition 1.3

If $\xi \in \mathfrak {g}$, we shall denote by $\xi _M\in \mathfrak {X}(M)$ and $\xi _X\in \mathfrak {X}(X)$ the vector fields induced by $\xi$ on M and X, respectively. If $\varvec{\nu }\in \mathbb {Z}^2$, we have the vector fields $(\imath \,D_{\varvec{\nu }})_M$ and $(\imath \,D_{\varvec{\nu }})_X$; similarly, for any $g\,T\in G/T$, we have the vector fields ${{\mathrm {Ad}}}_g(\imath \,D_{\varvec{\nu }})_M$ and ${\mathrm {Ad}}_g(\imath \,D_{\varvec{\nu }})_X$. To simplify notation, we shall set¹

$$\begin{aligned} \varvec{\nu }_M:=(\imath \,D_{\varvec{\nu }})_M, \quad \varvec{\nu }_X:=(\imath \,D_{\varvec{\nu }})_X, \end{aligned}$$

and

$$\begin{aligned} {\mathrm {Ad}}_g(\varvec{\nu })_M:={\mathrm {Ad}}_g(\imath \,D_{\varvec{\nu }})_M, \quad {\mathrm {Ad}}_g(\varvec{\nu })_X:={\mathrm {Ad}}_g(\imath \,D_{\varvec{\nu }})_X. \end{aligned}$$

Occasionally, we shall use the abridged notation $\xi (m)$ for $\xi _M(m)$, $\xi (x)$ for $\xi _X(x)$ with no further mention.

Definition 1.4

Let $\Vert \cdot \Vert _m:T_mM\rightarrow \mathbb {R}$ and $\Vert \cdot \Vert _x:T_xX\rightarrow \mathbb {R}$ be the norm functions. If $\varvec{\nu }=(\nu _1,\nu _2)\in \mathbb {Z}^2$, $\nu _1>\nu _2$, let us set $\varvec{\nu }_\perp :=(-\nu _2,\nu _1)$. With the notation introduced in Definitions 1.1 and 1.3, let us define a $\mathcal {C}^\infty$ function $\mathcal {D}_{\varvec{\nu }}: M^G_{\mathcal {O}_{ \varvec{ \nu } }}\rightarrow (0,+\infty )$ by posing

$$\begin{aligned} \mathcal {D}_{\varvec{\nu }} (m):= \frac{\Vert \varvec{\nu }\Vert }{\left\| {\mathrm {Ad}}_{h_m}(\varvec{\nu }_\perp )_M(m)\right\| _m}. \end{aligned}$$

Remark 1.3

Since by assumption $\widetilde{\mu }$ is locally free on $X^G_{\mathcal {O}_{ \varvec{ \nu } }}$, but not necessarily on $M^G_{\mathcal {O}_{ \varvec{ \nu } }}$, the latter definition warrants an explanation, since it might happen that $\xi _M(m)=0$ for $\xi \in \mathfrak {g}$ not zero and $m\in M^G_{\mathcal {O}_{ \varvec{ \nu } }}$. However, if $x\in X^G_{\mathcal {O}_{ \varvec{ \nu } }}$ and $m=\pi (x)$, then it follows from (6) and the definition of $h_m\,T$ that ${\mathrm {Ad}}_{h_m}(\varvec{\nu }_\perp )_X(x)={\mathrm {Ad}}_{h_m}(\varvec{\nu }_\perp )_M(m)^\sharp$, whence

$$\begin{aligned} \left\| {\mathrm {Ad}}_{h_m}(\varvec{\nu }_\perp )_M(m)\right\| _m=\Vert {\mathrm {Ad}}_{h_m}(\varvec{\nu }_\perp )_X(x)\Vert _x>0. \end{aligned}$$

Let us record one more piece of notation. If $V_3$ is the area of the unit sphere $S^3 \subseteq \mathbb {R}^4$, let us set

$$\begin{aligned} D_{ G/T } := 2\,\pi / V_3. \end{aligned}$$

Theorem 1.4

Under the same hypothesis as in Theorem 1.2, let us assume in addition that G acts freely on $X^G_{\mathcal {O}_{\varvec{\nu }}}$. Then, uniformly in $x\in X^G_{\mathcal {O}_{\varvec{\nu }}}$ we have for $k\rightarrow +\infty$ an asymptotic expansion of the form

$$\begin{aligned} \Pi _{ k\varvec{ \nu } }(x,x)\sim & {} \frac{D_{ G/T }}{\sqrt{ 2 }} \, \frac{1}{ \Vert \Phi _G ( m ) \Vert ^{ d +1/2 } } \, \left( \frac{ k \, \Vert \varvec{ \nu } \Vert }{ \pi } \right) ^{ d -1/2 } \cdot \mathcal {D}_{\varvec{\nu }} (m)\\&\cdot \left[ 1 +\sum _{j \ge 1} k^{- j / 4}\, a_j (\varvec{ \nu }, m ) \right] . \end{aligned}$$

We can refine the previous asymptotic expansion at a fixed diagonal point $(x,x)\in X^G_{\mathcal {O}_{\varvec{\nu }}} \times X^G_{\mathcal {O}_{\varvec{\nu }}}$ to an asymptotic expansion for near-diagonal rescaled displacements; however, for the sake of simplicity we shall restrict the directions of the displacements.

Definition 1.5

If $m\in M$, let $\mathfrak {g}_M(m)\subseteq T_mM$ be the image of the linear evaluation map ${\mathrm {val}}_m : \mathfrak {g} \rightarrow T_mM$, $\xi \mapsto \xi _M(m)$, also, let $\mathfrak {g}_M(m)^{\perp _\omega }\subseteq T_mM$ be its symplectic orthocomplement with respect to $\omega _m$, and let $\mathfrak {g}_M(m)^{\perp _g}\subseteq T_mM$ be its Riemannian orthocomplement with respect to $g_m$. Hence,

$$\begin{aligned} \mathfrak {g}_M(m)^{\perp _h} := \mathfrak {g}_M(m)^{\perp _\omega } \cap \mathfrak {g}_M(m)^{\perp _g} \subseteq T_mM \end{aligned}$$

is the Hermitian othocomplement of the complex subspace generated by $\mathfrak {g}_M(m)$ with respect to $h_m := g_m -\imath \, \omega _m$.

Definition 1.6

If $\mathbf {v}_1, \,\mathbf {v}_2\in T_mM$, following [30] let us set

$$\begin{aligned} \psi _2 (\mathbf {v}_1, \,\mathbf {v}_2 ) := -\imath \,\omega _m (\mathbf {v}_1, \,\mathbf {v}_2 ) -\frac{1}{2} \, \Vert \mathbf {v}_1 - \mathbf {v}_2\Vert _m ^2. \end{aligned}$$

(19)

Here $\Vert \mathbf {v}\Vert _m := g_m ( \mathbf {v}, \mathbf {v} )^{ 1 /2 }$. The same invariant can be introduced in any Hermitian vector space. Given the choice of a system of Heisenberg local coordinates centered at $x\in X$ [30], there is built-in unitary isomorphism $T_m M \cong \mathbb {C} ^d$; with this implicit, (19) will be used with $\mathbf {v}_j\in \mathbb {C}^d$.

The choice of Heisenberg local coordinates centered at $x\in X$ gives a meaning to the expression $x + ( \theta , \mathbf {v})$ for $(\theta , \mathbf {v})\in (-\pi ,\pi ) \times \mathbb {R}^{2 d}$ with $\Vert \mathbf {v} \Vert$ of sufficiently small norm. When $\theta =0$, we shall write $x+\mathbf {v}$.

Theorem 1.5

Let us assume the same hypothesis as in Theorem 1.4. Suppose $C>0$, $\epsilon \in (0, 1/6)$, and if $x\in X$ let us set $m_x:= \pi (x)$. Then, uniformly in $x\in X^G_{\mathcal {O}_{\varvec{\nu }}}$ and $\mathbf {v}_1, \,\mathbf {v}_2 \in \mathfrak {g}_M(m_x)^{\perp _h}$ satisfying $\Vert \mathbf {v}_j \Vert \le C \, k^\epsilon$, we have for $k\rightarrow +\infty$ an asymptotic expansion

$$\begin{aligned}&{ \Pi _{ k\varvec{ \nu } }\left( x + \frac{1}{\sqrt{ k }} \,\mathbf {v}_1, x + \frac{1}{\sqrt{ k }} \,\mathbf {v}_2 \right) }\\&\quad \sim \frac{D_{ G/T }}{\sqrt{ 2 }} \, \frac{ e^{ \psi _2 (\mathbf {v}_1, \mathbf {v}_2) / \lambda _{ \varvec{\nu } } ( m_x ) } }{ \Vert \Phi _G ( m_x ) \Vert ^{ d +1/2 } } \, \left( \frac{ k \, \Vert \varvec{ \nu } \Vert }{ \pi } \right) ^{ d -1/2 } \cdot \mathcal {D}_{\varvec{\nu }} (m_x) \\&\qquad \cdot \left[ 1 +\sum _{j \ge 1} k^{- j / 4}\, a_j (\varvec{ \nu }, m_x; \mathbf {v}_1, \mathbf {v}_2 ) \right] , \end{aligned}$$

where $a_j (\varvec{ \nu }, m_x; \cdot , \cdot )$ is a polynomial function of degree $\le \lceil 3 j/2\rceil$.

Furthermore, we shall provide an integral formula of independent interest for the asymptotics of $\Pi _{ k\varvec{ \nu } }(x',x')$ when $x'\rightarrow X^G_{\mathcal {O}_{\varvec{\nu }}}$ at a ‘fast’  pace from the ‘outside’  (i.e., $x'\in \overline{A}$ in the notation of Theorem 1.2) (Sect. 6.1). While the latter formula is a bit too technical to be described in this introduction, by global integration it leads to a lower bound on $\dim H(X)_{ \varvec{ \nu } }$ which can be stated in a compact form. By (12), with the notation of Theorem 1.2, we have

$$\begin{aligned} \dim H(X)_{ \varvec{ \nu } } = \dim _{ in } H(X)_{ \varvec{ \nu } } + \dim _{ out } H(X)_{ \varvec{ \nu } }, \end{aligned}$$

(20)

where

$$\begin{aligned} \dim _{out} H(X)_{ \varvec{ \nu } } := \int _A \Pi _{ \varvec{ \nu } } ( x, x ) \, \mathrm {d}V_X (x), \end{aligned}$$

and similarly for $\dim _{ in } H(X)_{ \varvec{ \nu } }$, with A replaced by B. Hence, an asymptotic estimate for $\dim _{out} H(X)_{ k\,\varvec{ \nu } }$ when $k\rightarrow +\infty$ implies an asymptotic lower bound for $\dim H(X)_{ k\, \varvec{ \nu } }$. In Theorem 1.6 below, we shall show that $\dim _{out} H(X)_{ k\,\varvec{ \nu } }$ is given by an asymptotic expansion of descending fractional powers of k, the leading power being $k^{ d-1 }$.

Theorem 1.6

Under the assumptions of Theorem 1.4, $\dim _{out} H(X)_{ k\,\varvec{ \nu } }$ is given by an asymptotic expansion in descending powers of $k^{1/4}$ as $k\rightarrow +\infty$, with leading-order term

$$\begin{aligned} \frac{1}{4} \, D_{ G/T } \, \, \left( \frac{ k \, \Vert \varvec{ \nu } \Vert }{ \pi } \right) ^{d -1 } \, \int _{M^G_{\mathcal {O} }}\,\frac{1}{ \Vert \Phi _G (m) \Vert ^{d }}\cdot \mathcal {D}_{\varvec{\nu }} (m) \, \mathrm {d}V_{M^G_{\mathcal {O} } }(m). \end{aligned}$$

Let us make some final remarks.

First, there is a wider scope for the results of this paper, since it builds on microlocal techniques that can be also applied in the almost complex symplectic setting. For the sake of simplicity, we have restricted our discussion to the complex projective setting; nonetheless, assuming the theory in [30] (which in turn builds on [4] and [3]), the present results can be extended to the case where M is a compact symplectic manifold with an integral symplectic form and a polarizing (or quantizing) line bundle A on it. More precisely, given an Hamiltonian compact Lie group action on M linearizing to A, one can find an invariant compatible almost complex structure and then rely on the theory of generalized Szegö kernels developed in [30] to extend the present arguments and constructions.

In closing, it seems in order to clarify further the relation of the present work to the general literature. The asymptotics of Bergman and Szegö kernels have attracted significant interest in recent years, involving algebraic, complex and symplectic geometry, as well as harmonic analysis. Generally, the emphasis has been placed on the perspective of Berezin-Toeplitz quantization, where the parameter of the asymptotics is the index of the Fourier component with respect to the structure $S^1$-action. Natural variants include additional symmetries, stemming from a linearizable Hamiltonian Lie group action. It would be unreasonable for space reasons to give here an account of this body of work, but we refer to [2, 3, 7, 8, 22, 23, 29, 30, 32, 34] and references therein. For some interesting recent extensions in the same spirit to a more abstract geometric setting, see [16] and [17].

In particular, the microlocal approach of [2, 3, 30, 34], of special relevance for the present work, is based on the theory of the Szegö kernel as a Fourier integral operator (see [4]) and has been exploited in [24, 25] to obtain local asymptotics in the G-equivariant Berezin-Toeplitz context.

This said, the perspective of the present work is quite different, and closer in spirit to [13], inasmuch as the structure $S^1$-action remains in the background and does not play any privileged role in the asymptotics (except of course in defining the underlying geometry); rather, as in [26], the additional symmetry is considered per se, on the same footing as the standard circle action in the usual on-diagonal expansion [7, 32, 34], as well as in the near-diagonal rescaled extensions [2, 30]. As in the toric case [26], this changes considerably the geometry of the asymptotics.

The present work covers part of the PhD thesis of the first author at the University of Milano Bicocca.

2 Examples

2.1 Example 1

Let A be the hyperplane line bundle on $M=\mathbb {P}^3$; then, the unit circle bundle $X\subseteq A^\vee {\setminus } (0)$ may be identified with $S^7\subset \mathbb {C}^4{\setminus }\{\mathbf {0}\}$, and the projection $\pi :X\rightarrow \mathbb {P}^3$ with the Hopf map.

Consider the unitary representation of G on $\mathbb {C}^4\cong \mathbb {C}^2\oplus \mathbb {C}^2$ given by

$$\begin{aligned} A\cdot (Z,W)=(AZ,AW); \end{aligned}$$

(21)

here $Z=(z_1,z_2)^t,\,W=(w_1,w_2)^t\in \mathbb {C}^2$. This linear action yields by restriction a contact action $\widetilde{\mu }:G\times S^7\rightarrow S^7$ and descends to an holomorphic action $\mu :G\times \mathbb {P}^3\rightarrow \mathbb {P}^3$. If $\omega _{FS}$ is the Fubini-Study form on $\mathbb {P}^3$, then $\mu$ is Hamiltonian with respect to $2\,\omega _{FS}$. The moment map is

$$\begin{aligned} \Phi _G:[Z:W]\in \mathbb {P}^3\mapsto \frac{\imath }{\Vert Z\Vert ^2+\Vert W\Vert ^2}\,[z_i\,\overline{z}_j+w_i\,\overline{w}_j]\in \mathfrak {g}. \end{aligned}$$

(22)

Furthermore, $\widetilde{\mu }$ is the contact lift of $\mu$.

From this, one can draw the following conclusions:

Lemma 2.1

Under the previous assumptions, we have:

• 1. $-\imath \,\Phi _G([Z:W])$ is a convex linear combination of the orthogonal projections onto the subspaces of $\mathbb {C}^2$ spanned by Z and W, respectively;

• 2. $-\imath \,\Phi _G([Z:W])$ has rank 2 if and only if Z and W are linearly independent, rank 1 otherwise;

• 3. $\Phi _G(M)=\imath \,K$, where K denotes the set of all positive semidefinite Hermitian matrices of trace 1;

• 4. the determinant of $-\imath \,\Phi _G([Z:W])$ is

  $$\begin{aligned} \det \big (-\imath \,\Phi _G([Z:W])\big )=\frac{|Z\wedge W|^2}{(\Vert Z\Vert ^2+\Vert W\Vert ^2)^2}, \end{aligned}$$

  where $Z\wedge W=z_1\,w_2-z_2\,w_1\in \mathbb {C}$;

• 5. the eigenvalues of $-\imath \,\Phi _G([Z:W])$ are both real and given by

  $$\begin{aligned} \lambda _{1,2}([Z:W])=\frac{1}{2}\,\left( 1\pm \sqrt{1-\frac{4\,|Z\wedge W|^2}{(\Vert Z\Vert ^2+\Vert W\Vert ^2)^2}}\right) . \end{aligned}$$

Let us fix $\varvec{\nu }\in \mathbb {Z}^2$ with $\nu _1>\nu _2\ge 0$. Let as above $\mathcal {O}_{\varvec{\nu }}\subseteq \mathfrak {g}$ denote the coadjoint orbit of $\imath \,D_{\varvec{\nu }}$. With $M=\mathbb {P}^3$, the locus $M^G_{\mathcal {O}_{\varvec{\nu }}}=\Phi _G^{-1}(\mathbb {R}_+\cdot \mathcal {O}_{\varvec{\nu }})$ is given by the condition

$$\begin{aligned} \nu _2\,\lambda _1([Z:W])-\nu _1\,\lambda _2([Z:W])=0. \end{aligned}$$

In view of Lemma 2.1, this implies:

Corollary 2.1

Under the previous assumptions,

$$\begin{aligned} M^G_{\mathcal {O}_{\varvec{\nu }}}=\left\{ [Z:W]\in \mathbb {P}^3\,: \,\frac{|Z\wedge W|}{\Vert Z\Vert ^2+\Vert W\Vert ^2}=\frac{\sqrt{\nu _1\,\nu _2}}{\nu _1+\nu _2}\right\} . \end{aligned}$$

Let us now consider transversality. By Lemma 4.1 (see also the discussion in §2 of [26]), $\Phi _G$ is transverse to the ray $\mathbb {R}_+\cdot \imath \,D_{\varvec{\nu }}$ in $\mathfrak {g}$ if and only if $\widetilde{\mu }$ is locally free along $X^G_{\varvec{\nu }}$ in (1.2) (i.e., each $x\in X^G_{\varvec{\nu }}$ has discrete stabilizer).

On the other hand, by (21) $\widetilde{\mu }$ is locally free at $(Z,W)\in S^7$ if and only if $Z\wedge W\ne 0$, and this is equivalent to $\Phi ([Z:W])$ having rank 2; this means that $-\imath \,\Phi _G([Z:W])$ has two positive eigenvalues. Thus, we obtain the following.

Corollary 2.2

The following conditions are equivalent:

• 1. $\Phi _G$ is transverse to $\mathbb {R}_+\cdot \imath \,D_{\varvec{\nu }}$, and $\Phi _G^{-1}(\mathbb {R}_+\cdot \imath \,D_{\varvec{\nu }})\ne \emptyset$;

• 2. $\Phi _G$ is transverse to $\mathcal {O}_{\varvec{\nu }}$, and $\Phi _G^{-1}(\mathbb {R}_+\cdot \mathcal {O}_{\varvec{\nu }})\ne \emptyset$;

• 3. $\nu _1,\,\nu _2> 0$.

Let us now consider the restricted Hamiltonian action of T. Identifying $\mathfrak {t}$ with $\imath \,\mathbb {R}^2$, $\Phi _T:M\rightarrow \mathfrak {t}$ may be written:

$$\begin{aligned} \Phi _T:[Z:W]\in \mathbb {P}^3\mapsto \frac{\imath }{\Vert Z\Vert ^2+\Vert W\Vert ^2}\,\begin{pmatrix} |z_1|^2+|w_1|^2\\ |z_2|^2+|w_2|^2 \end{pmatrix} \in \mathfrak {t}. \end{aligned}$$

(23)

Thus, we obtain

Lemma 2.2

Assume that $\nu _1>\nu _2\ge 0$; then:

• 1. the image of $\Phi _T$ in $\mathfrak {t}\cong \imath \,\mathbb {R}^2$ is

  $$\begin{aligned} \Phi _T(M)=\imath \,\left\{ \begin{pmatrix} x\\ y \end{pmatrix}:\,x+y=1,\,x,y\ge 0\right\} ; \end{aligned}$$

• 2. the locus $M^T_{\varvec{\nu }}=\Phi _T^{-1}(\mathbb {R}_+\cdot \imath \,D_{\varvec{\nu }})$ is given by

  $$\begin{aligned} M^T_{\varvec{\nu }}=\left\{ [Z:W]\in \mathbb {P}^3\,:\,\nu _2\,\left( |z_1|^2+|w_1|^2\right) =\nu _1\,\left( |z_2|^2+|w_2|^2\right) \right\} ; \end{aligned}$$

• 3. $\Phi _T$ is transverse to $\mathbb {R}_+\cdot \imath \,D_{\varvec{\nu }}$ and $M^T_{\varvec{\nu }}\ne \emptyset$ if and only if $\nu _1,\,\nu _2>0$.

Proof

The first two statements follow immediately from (23). As to the third, let us recall again that $\Phi _T$ is transverse to $\mathbb {R}_+\cdot \imath \,D_{\varvec{\nu }}$ if and only if the action of T on $X^T_{\varvec{\nu }}\subset S^7$ is locally free [26].

On the other hand, T acts locally freely at $(Z,W)\in S^7$ if and only if Z and W are neither both scalar multiples of $\mathbf {e}_1$, nor both scalar multiples of $\mathbf {e}_2$, where $(\mathbf {e}_1,\,\mathbf {e}_2)$ is the standard basis of $\mathbb {C}^2$. By 2), there are no points (Z, W) of this form in $X^T_{\varvec{\nu }}$ if and only if $\nu _2>0$. $\square$

Hence, if $\nu _1,\,\nu _2>0$, then both $\Phi _G$ and $\Phi _T$ are transverse to $\mathbb {R}_+\cdot \varvec{\nu }$, and $M^G_{ \varvec{\nu }}\ne \emptyset$, $M^T_{ \varvec{\nu }}\ne \emptyset$. For instance,

$$\begin{aligned} \left[ \sqrt{\frac{\nu _1}{\nu _1+\nu _2}}\,\mathbf {e}_1:\,\sqrt{\frac{\nu _2}{\nu _1+\nu _2}}\,\mathbf {e}_2\right] \in M^G_{ \varvec{\nu }}\cap M^T_{ \varvec{\nu }}. \end{aligned}$$

More generally, we have the following.

Lemma 2.3

For any $\varvec{\nu }$, $M^G_{ \varvec{\nu }}\cap M^T_{ \varvec{\nu }} =\Phi _G^{-1}\left\{ \imath \, (\nu _1+\nu _2)^{-1}\,D_{\varvec{\nu }}\right\}$.

Proof

By Lemma 2.1, $[Z:W]\in M^G_{ \varvec{\nu }}$ if and only if $-\imath \,\Phi _G([Z:W])$ is similar to $D_{\varvec{\nu }/(\nu _1+\nu _2)}$; on the other hand, by Lemma 2.2, $[Z:W]\in M^T_{ \varvec{\nu }}$ if and only if for some $z\in \mathbb {C}$

$$\begin{aligned} -\imath \,\Phi _G([Z:W])= \begin{pmatrix} \nu _1/(\nu _1+\nu _2)&{}z\\ \overline{z}&{}\nu _1/(\nu _1+\nu _2) \end{pmatrix}. \end{aligned}$$

Equaling determinants, we conclude that $z=0$. This concludes the proof. $\square$

Let $\mathfrak {g}_\imath \subseteq \mathfrak {g}$ be the affine hyperplane of the skew-Hermitian matrices of trace $\imath$; we may interpret $\Phi _G$ as a smooth map $\Phi _G':\mathbb {P}^3\rightarrow \mathfrak {g}_\imath$.

Lemma 2.4

If $\nu _1>\nu _2>0$, then $\imath \,(\nu _1+\nu _2)^{-1}\,D_{\varvec{\nu }}\in \mathfrak {g}_\imath$ is a regular value of $\Phi _G'$.

Proof

Clearly, the latter matrix is a regular value of $\Phi '_G$ if and only if $\Phi _G$ is transverse to the ray $\mathbb {R}_+\cdot \imath \,D_{\varvec{\nu }}$; thus, the statement follows from Corollary 2.2. $\square$

By Lemmata 2.3 and 2.4, we obtain

Corollary 2.3

Suppose $\nu _1>\nu _2>0$. Then, with $M=\mathbb {P}^3$:

• 1. $M^G_{\mathcal {O}}$ and $M^T_{\varvec{\nu }}$ are smooth compact (real) hypersurfaces in M;

• 2. $M^G_{\mathcal {O}}\cap M^T_{\varvec{\nu }}$ is a smooth submanifold of M of real codimension 3.

Let us now describe the saturation $G\cdot M^T_{\varvec{\nu }}$.

Lemma 2.5

Under the previous assumptions,

$$\begin{aligned} G\cdot M^T_{\varvec{\nu }}=\left\{ [Z:W]\in \mathbb {P}^3\,: \,\frac{\Vert Z\wedge W\Vert }{\Vert Z\Vert ^2+\Vert W\Vert ^2}\le \frac{\sqrt{\nu _1\,\nu _2}}{\nu _1+\nu _2}\right\} . \end{aligned}$$

Proof

Consider $[Z:W]\in \mathbb {P}^3$ with $(Z,W)\in S^7$. By definition, $[Z:W]\in G\cdot M^T_{\varvec{\nu }}$ if and only if there exists $A\in G$ such that $[AZ:AW]\in M^T_{\varvec{\nu }}$; we may actually require without loss that $A\in SU(2)$. Let us write

$$\begin{aligned} A= \begin{pmatrix} a&{}-\overline{c}\\ c&{}\overline{a} \end{pmatrix}\in SU(2), \,\,\,Z=\begin{pmatrix} z_1\\ z_2 \end{pmatrix},\,\,\, W=\begin{pmatrix} w_1\\ w_2 \end{pmatrix}; \end{aligned}$$

then $[AZ:AW]\in M^T_{\varvec{\nu }}$ if and only if (with some computations)

$$\begin{aligned} 0= & {} \nu _2\,\left( |a\,z_1-\overline{c}\,z_2|^2+|a\,w_1-\overline{c}\,w_2|^2\right) -\nu _1\,\left( |c\,z_1+\overline{a}\,z_2|^2+|c\,w_1+\overline{a}\,w_2|^2\right) \nonumber \\= & {} \nu _2\, \left\| \begin{pmatrix} z_1&{}\quad z_2\\ w_1&{}\quad w_2 \end{pmatrix}\, \begin{pmatrix} a\\ -\overline{c} \end{pmatrix} \right\| ^2 - \nu _1\, \left\| \begin{pmatrix} z_1&{}\quad z_2\\ w_1&{}\quad w_2 \end{pmatrix}\, \begin{pmatrix} c\\ \overline{a} \end{pmatrix} \right\| ^2. \end{aligned}$$

(24)

In other words, $[Z:W]\in G\cdot M^T_{\varvec{\nu }}$ if and only if there exists an orthonormal basis $\mathcal {B}=(V_1,\,V_2)$ of $\mathbb {C}^2$ such that

$$\begin{aligned} \nu _2\, \left\| \begin{pmatrix} z_1&{}\quad z_2\\ w_1&{}\quad w_2 \end{pmatrix}\, V_1 \right\| ^2= \nu _1\, \left\| \begin{pmatrix} z_1&{}\quad z_2\\ w_1&{}\quad w_2 \end{pmatrix}\, V_2 \right\| ^2. \end{aligned}$$

(25)

Now for any $V\in \mathbb {C}^2$ we have

$$\begin{aligned} \left\| \begin{pmatrix} z_1&{}\quad z_2\\ w_1&{}\quad w_2 \end{pmatrix}\, V \right\| ^2= & {} V^t\,\begin{pmatrix} z_1&{}\quad w_1\\ z_2&{}\quad w_2 \end{pmatrix}\, \begin{pmatrix} \overline{z}_1&{}\quad \overline{z}_2\\ \overline{w}_1&{}\quad \overline{w}_2 \end{pmatrix}\, \overline{V}\\= & {} V^t\,\frac{1}{\imath }\,\Phi _G([Z:W])\,\overline{V}. \end{aligned}$$

If $\lambda _1(Z,W)\ge \lambda _2(Z,W)\ge 0$ are the eigenvalues of $-\imath \,\Phi _G([Z:W])$ (Lemma 2.1), we then obtain for any $V\in S^7$

$$\begin{aligned} \lambda _1(Z,W)\ge \left\| \begin{pmatrix} z_1&{}\quad z_2\\ w_1&{}\quad w_2 \end{pmatrix}\, V \right\| ^2\ge \lambda _2(Z,W), \end{aligned}$$

(26)

with left (respectively, right) equality holding if and only if V is an eigenvector of $-\imath \,\Phi _G([Z:W])$ relative to $\lambda _1(Z,W)$ (respectively, $\lambda _2(Z,W)$). We conclude from (25) and (26) that if $(Z,W)\in G\cdot X^T_{\varvec{\nu }}$, then the following inequalities holds:

$$\begin{aligned} \nu _1\, \lambda _1(Z,W) \ge \nu _2\,\lambda _2(Z,W),\quad \nu _2\, \lambda _1(Z,W) \ge \nu _1\,\lambda _2(Z,W). \end{aligned}$$

(27)

While the former is trivial, since $\nu _1>\nu _2>0$ and $\lambda _1(Z,W)\ge \lambda _2(Z,W)\ge 0$, the latter is equivalent to the other

$$\begin{aligned} \frac{\sqrt{\nu _1\,\nu _2}}{\nu _1+\nu _2}\ge \Vert Z\wedge W\Vert . \end{aligned}$$

(28)

Suppose, conversely, that (28) holds. Then, (27) also holds. Let $(W_1,W_2)$ be an orthonormal basis of eigenvectors of $-\imath \,\Phi _G\big ([Z:W]\big )$ with respect to the eigenvalues $\lambda _1(Z,W)$ and $\lambda _2(Z,W)$, respectively. Evaluating the two sides of (25) with $V_1'=W_1$, $V_2'=W_2$ in place of $(V_1,V_2)$, we obtain

$$\begin{aligned} \nu _2\, \left\| \begin{pmatrix} z_1&{}\quad z_2\\ w_1&{}\quad w_2 \end{pmatrix}\, V_1' \right\| ^2=\nu _2\, \lambda _1(Z,W)\ge \nu _1\,\lambda _2(Z,W) =\nu _1\, \left\| \begin{pmatrix} z_1&{}\quad z_2\\ w_1&{}\quad w_2 \end{pmatrix}\, V_2' \right\| ^2. \end{aligned}$$

Using instead $V_1''=W_2$ and $V_2''=W_1$ in place of $(V_1,V_2)$, we obtain

$$\begin{aligned} \nu _2\, \left\| \begin{pmatrix} z_1&{}\quad z_2\\ w_1&{}\quad w_2 \end{pmatrix}\, V_1'' \right\| ^2=\nu _2\, \lambda _2(Z,W)\le \nu _1\,\lambda _1(Z,W) =\nu _1\, \left\| \begin{pmatrix} z_1&{}\quad z_2\\ w_1&{}\quad w_2 \end{pmatrix}\, V_2'' \right\| ^2. \end{aligned}$$

Since $G=U(2)$ is connected and acts transitively on the family of all orthonormal basis of $\mathbb {C}^2$, we conclude by continuity that there exists an orthonormal basis $(V_1,V_2)$ on which (25) is satisfied. $\square$

In view of Corollary 2.1, we deduce

Corollary 2.4

$M^G_{\mathcal {O}_{\varvec{\nu }}}=\partial \left( G\cdot M^T_{\varvec{\nu }}\right)$.

The boundary $\partial \left( G\cdot M^T_{\varvec{\nu }}\right)$ consists of those $[Z:W]\in \mathbb {P}^3$ such that $-\imath \,\Phi _G([Z:W])$ is similar to $(\nu _1+\nu _2)^{-1}\,D_{\varvec{\nu }}$, while the interior $\left( G\cdot M^T_{\varvec{\nu }}\right) ^0$ consists of those $[Z:W]\in \mathbb {P}^3$ such that $-\imath \,\Phi _G([Z:W])$ is similar to a matrix of the form

$$\begin{aligned} \frac{1}{\nu _1+\nu _2}\, \begin{pmatrix} \nu _1&{}\quad z\\ \overline{z}&{}\quad \nu _2 \end{pmatrix}, \end{aligned}$$

for some complex number $z\ne 0$.

Finally, the locus $X'\subseteq X=S^7$ of those (Z, W) at which $\widetilde{\mu }$ is not locally free is defined by the condition $Z\wedge W=0$, and therefore, it is contained in $\left( G\cdot M^T_{\varvec{\nu }}\right) ^0$. It is the unit circle bundle over a non-singular quadric hypersurface in $\mathbb {P}^3$. The stabilizer subgroup of $(Z,W)\in S^7$ is trivial if $Z\wedge W\ne 0$, and it is isomorphic to $S^1$ otherwise.

For any fixed $\varvec{\nu }=(\nu _1,\nu _2)\in \mathbb {Z}^2$ with $\nu _1>\nu _2$, let consider how $V_{k\varvec{\nu }}$ appears in the isotypical decomposition of $H\left( X\right)$ under $\widehat{\mu }$ in (7). The Hopf map $\pi :X=S^7\rightarrow \mathbb {P}^3$ is the quotient map for the standard action $r:S^1\times S^7\rightarrow S^7\subset \mathbb {C}^4$, given by complex scalar multiplication. The corresponding unitary representation of $S^1$ on H(X) yields an isotypical decomposition $H(X)=\bigoplus _{l\in \mathbb {Z}}H_l(X)$, where for $l\in \mathbb {N}$ we set

$$\begin{aligned} H_l(X):=\left\{ f\in H(X)\,:\,f\left( e^{\imath \theta }\,x\right) =e^{\imath \,l\theta }\,f(x)\,\forall \,x=(Z,W)\in X, \,e^{i\theta }\in S^1\right\} . \end{aligned}$$

As is well known, there are natural U(2)-equivariant unitary isomorphisms

$$\begin{aligned} H_l(X)\cong & {} H^0\left( \mathbb {P}^3,\mathcal {O}_{\mathbb {P}^3}(l)\right) \cong {\mathrm {Sym}}^{l}\left( \mathbb {C}^2\oplus \mathbb {C}^2\right) \nonumber \\= & {} \bigoplus _{h=0}^l{\mathrm {Sym}}^{h}\left( \mathbb {C}^2\right) \otimes {\mathrm {Sym}}^{l-h}\left( \mathbb {C}^2\right) . \end{aligned}$$

(29)

On the other hand, a character computation yields the following.

Lemma 2.6

For $p\ge q$,

$$\begin{aligned} {\mathrm {Sym}}^{p}\left( \mathbb {C}^2\right) \otimes {\mathrm {Sym}}^{q}\left( \mathbb {C}^2\right) \cong \bigoplus _{a=0}^q(\det )^{\otimes a}\otimes {\mathrm {Sym}}^{p+q-2a}\left( \mathbb {C}^2\right) . \end{aligned}$$

as U(2)-representations.

Proof of Lemma 2.6

The character of ${\mathrm {Sym}}^{p}\left( \mathbb {C}^2\right)$ is $\chi _{(p+1,0)}$. Since the character of a tensor product of representations is the product of the respective characters, the character of ${\mathrm {Sym}}^{p}\left( \mathbb {C}^2\right) \otimes {\mathrm {Sym}}^{q}\left( \mathbb {C}^2\right)$ is $\chi ':=\chi _{(p+1,0)}\cdot \chi _{(q+1,0)}$. Let us evaluate $\chi$ on a diagonal matrix $D_{\mathbf {z}}$ with diagonal $\mathbf {z}=(z_1,z_2)$. We obtain

$$\begin{aligned} \chi '(D_{\mathbf {z}})= & {} \frac{z_1^{p+1}-z_2^{p+1}}{z_1-z_2}\cdot \left( z_1^q+z_1^{q-1}\,z_2+\cdots +z_1\,z_2^{q-1}+z_2^q\right) \nonumber \\= & {} \frac{1}{z_1-z_2}\cdot \left( \sum _{j=0}^{q}z_1^{p+1+q-j}\,z_2^j-\sum _{j=0}^{q}z_1^j\,z_2^{p+1+q-j}\right) \nonumber \\= & {} \sum _{j=0}^{q}\frac{1}{z_1-z_2}\cdot \left( z_1^{p+1+q-j}\,z_2^j-z_1^j\,z_2^{p+1+q-j}\right) \nonumber \\= & {} \sum _{j=0}^{q}\chi _{(p+1+q-j,j)}(D_{\mathbf {z}}). \end{aligned}$$

(30)

Now, a character is uniquely determined by its restriction to T, and on the other hand, the character of a direct sum is the sum of the characters; therefore, in view of (8), we conclude from (30) that

$$\begin{aligned} {\mathrm {Sym}}^{p}\left( \mathbb {C}^2\right) \otimes {\mathrm {Sym}}^{q}\left( \mathbb {C}^2\right) \cong \bigoplus _{j=0}^{q}V_{(p+1+q-j,j)}=\bigoplus _{j=0}^{q}{\det }^{\otimes j}\otimes {\mathrm {Sym}}^{p+q-2j}\left( \mathbb {C}^2\right) . \end{aligned}$$

$\square$

Therefore,

$$\begin{aligned} H_l(X)\cong \bigoplus _{h=0}^lH_{l,h}(X), \end{aligned}$$

(31)

where we set

$$\begin{aligned} H_{l,h}(X):=\bigoplus _{a=0}^{\min (h,l-h)}(\det )^{\otimes a}\otimes {\mathrm {Sym}}^{l-2a}\left( \mathbb {C}^2\right) . \end{aligned}$$

(32)

In order for the ath summand in (31) to be isomorphic to $V_{k\varvec{\nu }}$, we need to have $a=k\,\nu _2$ and $l-2a=k\,(\nu _1-\nu _2)-1$; hence, in this special case $H(X)_{k\varvec{\nu }} \subseteq H_l(X)$ with $l=k\,(\nu _1+\nu _2)-1$. Let us estimate the multiplicity of $H(X)_{k\varvec{\nu }}$ in $H_l(X)$. In order for the ath summand with $a=k\,\nu _2$ to appear in $H_{lh}(X)$ in (32) for some $h\le k\,(\nu _1+\nu _2)-1$, we need to have

$$\begin{aligned} a=k\,\nu _2\le & {} \min \big (h,k\,(\nu _1+\nu _2)-1-h\big )\nonumber \\\Rightarrow & {} k\,\nu _2\le h,\,\,k\,\nu _2\le k\,(\nu _1+\nu _2)-1-h\nonumber \\\Rightarrow & {} k\,\nu _2\le h\le k\,\nu _1-1. \end{aligned}$$

(33)

Hence, there are $k(\nu _1-\nu _2)-1$ values of h for which $H_{l,h}(X)$ contains one copy of $V_{k\varvec{\nu }}$. The dimension of $H(X)_{k\varvec{\nu }}$ is thus $\big (k(\nu _1-\nu _2)-1\big )\,k(\nu _1-\nu _2)\sim k^2\,(\nu _1-\nu _2)^2+O(k)$.

2.2 Example 2

Next, we shall briefly describe an example on $M=\mathbb {P}^4$, being much sketchier than in the previous case. As before, A will denote the hyperplane line bundle, and $X=S^9$ the dual unit circle bundle.

Let us consider the unitary action of U(2) on $\mathbb {C}^5\cong \mathbb {C}^2\oplus \mathbb {C}^2\oplus \mathbb {C}$ given by

$$\begin{aligned} A\cdot (Z,W,t)=(AZ,AW, \det (A)\, t); \end{aligned}$$

(34)

here $Z=(z_1,z_2)^t,\,W=(w_1,w_2)^t\in \mathbb {C}^2$, $t\in \mathbb {C}$. We shall again denote by $\widetilde{\mu }:G\times S^9\rightarrow S^9$, and $\mu :G\times \mathbb {P}^4\rightarrow \mathbb {P}^4$ the associated contact and Hamiltonian actions. The moment map is now

$$\begin{aligned} \Phi _G:[Z:W:t]\in \mathbb {P}^4\mapsto \frac{\imath }{\Vert Z\Vert ^2+\Vert W\Vert ^2+|t|^2}\,[z_i\,\overline{z}_j+w_i\,\overline{w}_j+\delta _{ij}\,|t|^2]\in \mathfrak {g}. \end{aligned}$$

(35)

Thus $-\imath \,\Phi _G\big ([Z:W:t]\big )\ge 0$ is a rescaling of $\Vert Z\Vert ^2\,p_Z+\Vert W\Vert ^2\,p_W+|t|^2\,I_2$, and its trace varies in [1, 2]. In particular, $\mathbf {0}\not \in \Phi _T(M)$.

Now, $(Z,W,t)\in S^9$ has non-trivial stabilizer under $\widetilde{\mu }$ if and only if either $t=0$ and $Z\wedge W=0$, or else $Z=W=0$. In the former case, $-\imath \,\Phi _G\big ([Z:W:t]\big )$ is similar to $D_{(1,0)}$, and in the latter to $I_2$. Therefore, $\Phi _G$ is transverse to $\mathbb {R}_+\cdot \imath \,D_{\varvec{\nu }}$ for any $\varvec{\nu }$ with $\nu _1>\nu _2>0$.

Furthermore, if $(Z,W,t)\in S^9$ has non-trivial stabilizer K in T under $\widetilde{\mu }$, then Z and W are either both multiples of $e_1$, in which case $K\leqslant \{1\}\times S^1$, or both multiples of $e_2$, in which case $K\leqslant S^1\times \{1\}$. If $t\ne 0$, the condition $\det (A)=1$ for $A\in K$ implies that $A=I_2$, so K is trivial. If $t=0$, then $-\imath \,\Phi _G\big ([Z:W:t]\big )$ is either $D_{(1,0)}$ or $D_{(0,1)}$. On the other hand, if $Z=W=0$, then $-\imath \,\Phi _G\big ([Z:W:t]\big )=I_2$. Thus, $\Phi _T$ is transverse to the ray $\mathbb {R}_+\cdot \imath \,\varvec{\nu }$ if $\nu _1>\nu _2>0$.

Let us fix one such $\varvec{\nu }$, and look for all the copies of $V_{k\varvec{\nu }}$ within $H(X)\cong \bigoplus _{l=0}^{+\infty } H_l(X)$.

For any $l=0,1,2,\ldots$, by Lemma 2.6 we have

$$\begin{aligned} H_l(X)= & {} \bigoplus _{p+q+r=l}{\mathrm {Sym}}^{p}\left( \mathbb {C}^2\right) \otimes {\mathrm {Sym}}^{q}\left( \mathbb {C}^2\right) \otimes {\det }^{\otimes r}\nonumber \\\cong & {} \bigoplus _{p+q+r=l}\bigoplus _{a=0}^{\min (p,q)}{\mathrm {Sym}}^{p+q-2a}\left( \mathbb {C}^2\right) \otimes {\det }^{\otimes (a+r)} \end{aligned}$$

(36)

The general summand in (36) is isomorphic to $V_{k\varvec{\nu }}$ if and only if

$$\begin{aligned} a+r= k\nu _2,\quad p+q-2a=k\,(\nu _1-\nu _2)-1. \end{aligned}$$

(37)

Thus for any $r=0,\ldots ,k\nu _2$ we can set $a=k\nu _2-r$ and then consider all the pairs (p, q) such that

$$\begin{aligned} p+q+2r=k\,(\nu _1+\nu _2)-1. \end{aligned}$$

(38)

We see from (38) that

$$\begin{aligned} k\,(\nu _1+\nu _2)-1\ge l=p+q+r=k\,(\nu _1+\nu _2)-1-r\ge k\,\nu _1-1; \end{aligned}$$

(39)

furthermore, equality holds on the left in (39) when $r=0$ and on the right when $r=k\,\nu _2$; every intermediate value is assumed. Therefore in this case $H(X)_{k\varvec{\nu }}\cap H_l(X)\ne (0)$ for every $l=k\,\nu _1-1,k\,\nu _1,\ldots ,k\,(\nu _1+\nu _2)-1$, so that $H(X)_{k\varvec{\nu }}$ is not a space of sections of any power of A.

Finally, we see from (37) and (38) that the copies of $V_{k\varvec{\nu }}$ within H(X) are in one-to-one correspondence with the triples (p, q, r) of natural numbers such that $0\le r\le k\,\nu _2$ and $p+q=k\,(\nu _1+\nu _2)-2r-1$. It follows that

$$\begin{aligned} \dim \big (H(X)_{k\varvec{\nu }}\big )= k^3\,\nu _1\,\nu _2\,(\nu _1-\nu _2)+O\left( k^2\right) . \end{aligned}$$

3 Proof of Theorem 1.1

3.1 Preliminaries

Before delving into the proof, let us collect some useful pieces of notation and recall some relevant concepts and results.

3.1.1 The Weyl integration formula

For the following, see, e.g., §2.3 of [33]. Let $\mathrm {d}V_G$ and $\mathrm {d}V_T$ denote the Haar measures on G and T, respectively (or the respective smooth densities). They determine a ‘quotient’  measure $\mathrm {d}V_{G/T}$ on G / T.

Definition 3.1

Let us define $\Delta :T\rightarrow \mathbb {C}$ by setting

$$\begin{aligned} \Delta (t):= t_1-t_2 \quad \big (t=(t_1,t_2)\in T\big ); \end{aligned}$$

here we identify T with $S^1\times S^1$ in the natural manner.

Furthermore, for any $f\in \mathcal {C}^\infty (G)$ let us define $A_f:T\rightarrow \mathbb {C}$ by setting

$$\begin{aligned} A_f(t):= \int _{G/T}\,f\left( g\,t\,g^{-1}\right) \,\mathrm {d}V_{G/T}(g\,T). \end{aligned}$$

If f is a class function, $A_f(t)=f(t)$ for any $t\in T$.

Then, the following holds.

Theorem (Weyl) With the assumptions and notation above,

$$\begin{aligned} \int _G\, f(g)\,\mathrm {d}V_G(g) = \frac{1}{2}\,\int _T\,A_f(t)\,\big |\Delta (t)\big |^2\,\mathrm {d}V_T(t). \end{aligned}$$

3.1.2 Ladder representations

For the following concepts, see [13]. We shall use throughout the identification $T^*G\cong G\times \mathfrak {g}^\vee$ induced by right translations. If R and S are manifolds and $\Lambda \subset T^*R\times T^*S$ is a Lagrangian submanifold, the corresponding canonical relation is

$$\begin{aligned} \Lambda ':= \big \{\big ((r,\upsilon ), (s,-\gamma )\big ): \big ((r,\upsilon ), (s,\gamma )\big ) \in \Lambda \big \}. \end{aligned}$$

Definition 3.2

For every weight $\varvec{ \nu }$, let $\chi _{ \varvec{\nu } }: G \rightarrow \mathbb {C}$ be the character of the associated irreducible representation, and let $d_{ \varvec{ \nu }}=\nu _1-\nu _2$ be the dimension of its carrier space. Let us denote by $L=L_{\varvec{ \nu }} := (k\,\varvec{ \nu })_{k=0}^{+\infty }$ the ladder sequence of weights generated by $\varvec{ \nu }$, and set

$$\begin{aligned} \chi _ L := \sum _{ k = 1 } ^{ +\infty } d_{ k \varvec{ \nu } } \, \chi _{ k \varvec{ \nu } } \in \mathcal {D}' (G). \end{aligned}$$

(40)

Definition 3.3

For every $f\in \mathcal {C}(\mathcal {O})$, let $G_f\leqslant G$ be the stabilizer subgroup of f, and let $\mathfrak {g}_f\leqslant \mathfrak {g}$ be its Lie algebra. Let $H_f\leqslant G_f$ be the closed connected codimension-1 subgroup with Lie subalgebra $\mathfrak {h}_f=\mathfrak {g}_f\cap f^\perp$. The locus

$$\begin{aligned} \Lambda _L := \left\{ ( g, r\,f ) \in G \times \mathfrak {g}^\vee : \,f \in \mathcal {O},\,r>0,\, g \in H_f \right\} \end{aligned}$$

(41)

is a Lagrangian submanifold of $T^*G$.

Then, we have the following.

Theorem

(Theorem 6.3 of [13]) $\chi _L$ is a Lagrangian distribution on G, and its associated conic Lagrangian submanifold of $T^*G \cong G \times \mathfrak {g}^\vee$ is $\Lambda _L$ in (41).

Consider the Hilbert space direct sum

$$\begin{aligned} H(X)_L := \bigoplus _{k=1}^{ +\infty } H(X)_{ k\,\varvec{ \nu } }, \end{aligned}$$

and let $\Pi _L: L^2(X) \rightarrow L^2(X)_L$ denote the corresponding orthogonal projector, $\Pi _L(\cdot , \cdot ) \in \mathcal {D}'(X \times X)$ its Schwartz kernel. Then,

$$\begin{aligned} \Pi _{L } ( x, y ) := \int _G \overline{ \chi _{L } (g) } \, \Pi \left( \widetilde{\mu } _{g ^{-1} } ( x ), y \right) \, \mathrm {d}V _G(g). \end{aligned}$$

(42)

We shall express (42) in functorial notation (cfr the discussion on page 374 of loc. cit.), and use basic results on the functorial behavior of wave fronts under pull-backs and push-forwards (see for instance §1.3 of [9] and §VI.3 of [11]) to draw conclusions on the singularities of $\Pi _L$.

To this end, let us consider the map

$$\begin{aligned} f: G\times X \times X \rightarrow X\times X,\quad (g, x, y ) \mapsto \left( \widetilde{\mu }_{g^{-1}} ( x ), y \right) \end{aligned}$$

and the distribution $\widehat{\Pi } := f^* (\Pi ) \in \mathcal {D}' (G \times X \times X )$. Let

$$\begin{aligned} \Sigma :=\{ (x, r \,\alpha _x ) : x\in X, r>0 \} \subset T^*X{\setminus } (0) \end{aligned}$$

(43)

denote the closed symplectic cone sprayed by the connection 1-form; by [4], the wave front of $\Pi$ satisfies

$$\begin{aligned} {\mathrm {WF}}' (\Pi ) = {\mathrm {diag}}(\Sigma ) \subset \Sigma \times \Sigma . \end{aligned}$$

(44)

It follows that ${\mathrm {WF}}' \left( \widehat{\Pi } \right) \subseteq f^*\big ({\mathrm {diag}}(\Sigma )\big )$. This implies the following.

Lemma 3.1

In terms of the identification $T^*G \cong G \times \mathfrak {g}^\vee$ induced by right translations, the canonical relation of $\widehat{\Pi }$ is

$$\begin{aligned} {\mathrm {WF}}' \left( \widehat{\Pi } \right)= & {} \Big \{ \Big ( \big ( g, r\,\Phi _G (m_x ) \big ), ( x, r\, \alpha _x ), (y, r\, \alpha _y ) \Big )\nonumber \\&: g \in G, \, x \in X, \, r>0, \, y = \widetilde{\mu } _{g ^{-1} } ( x ) \Big \}; \end{aligned}$$

(45)

recall that $m_x = \pi (x)$.

Now let us give the functorial reformulation of (42). Consider the diagonal map

$$\begin{aligned} \Delta :G\times X\times X\rightarrow G\times G\times X\times X, \quad (g,x,y)\mapsto (g,g,x,y), \end{aligned}$$

and the projection

$$\begin{aligned} p:G\times X\times X\rightarrow X\times X, \quad (g,x,y) \mapsto (x,y). \end{aligned}$$

Lemma 3.2

The Schwartz kernel $\Pi _L\in \mathcal {D}'(X\times X)$ is given by

$$\begin{aligned} \Pi _L = p_* \left( \Delta ^* \big ( \overline{\chi } _L\boxtimes \widehat{\Pi } \big )\right) . \end{aligned}$$

Let $\sigma :T^*G\rightarrow T^*G$ be given by $(g,f)\mapsto (g,-f)$. Then,

$$\begin{aligned} WF (\overline{\chi } _L\boxtimes \widehat{\Pi })\subseteq & {} \Big (\sigma (\Lambda _L)\times (0)\Big )\cup \Big ( \sigma (\Lambda _L) \times {\mathrm {WF}} \left( \widehat{\Pi } \right) \Big ) \cup \Big ( (0)\times {\mathrm {WF}} \left( \widehat{\Pi } \right) \Big )\\\subset & {} T^*G\times (T^*G \times T^*X\times T^*X). \end{aligned}$$

Therefore, the pull-back $\Delta ^* \big ( \overline{\chi } _L\boxtimes \widehat{\mu }\big )$ is well defined, and

$$\begin{aligned}&{ WF \left( \Delta ^* \big ( \overline{\chi } _L\boxtimes \widehat{\Pi }\big )\right) \subseteq \mathrm {d}\Delta ^* \big (WF (\overline{\chi } _L\boxtimes \widehat{\Pi })\big ) } \nonumber \\&\quad \subseteq \Big (\sigma (\Lambda _L)\times (0)\Big )\cup \mathrm {d}\Delta ^* \Big ( \sigma (\Lambda _L) \times {\mathrm {WF}} \left( \widehat{\Pi } \right) \Big ) \cup {\mathrm {WF}} \left( \widehat{\Pi } \right) \nonumber \\&\quad \subset T^*G\times T^*X\times T^*X. \end{aligned}$$

(46)

Explicitly, we have

$$\begin{aligned}&{ \mathrm {d}\Delta ^* \Big ( \sigma (\Lambda _L) \times {\mathrm {WF}} \left( \widehat{\Pi } \right) \Big )} \nonumber \\&\quad = \Big \{ \Big ( \big ( g, -f+ r \, \Phi _G (m_x ) \big ), ( x, r \, \alpha _x ), ( y, - r \, \alpha _y ) \Big )\nonumber \\&\qquad : f\in \mathcal {C}(\mathcal {O}),\, g \in H_f, \, x \in X, \, r>0, \, y = \widetilde{\mu } _{g ^{-1} } ( x ) \Big \}. \end{aligned}$$

(47)

Using that $\Phi _G$ is nowhere vanishing, we can now apply Proposition 1.3.4 of [9] to conclude the following.

Corollary 3.1

The wave front $WF (\Pi _L)\subseteq \left( T^*X{\setminus } (0)\right) \times \left( T^*X{\setminus } (0)\right)$ of the distributional kernel $\Pi _L$ satisfies

$$\begin{aligned}&{ WF \big (\Pi _L\big ) } \\&\quad \subseteq \Big \{ \Big ( ( x, r \, \alpha _x ), ( y, - r \, \alpha _y ) \Big ): f:=\Phi _G ( x )\in \mathcal {C}(\mathcal {O}),\, y \in H_f\cdot x \Big \}, \end{aligned}$$

where $H_f\cdot x$ is the $H_f$-orbit of x.

Corollary 3.2

Let $SS (\Pi _L)\subseteq X\times X$ be the singular support of the distributional kernel $\Pi _L$. Then, $SS \big (\Pi _L\big ) \subseteq \mathcal {Z}_{\varvec{ \nu }}$.

3.2 The proof

Proof of Theorem 1.1

For every $\varvec{\mu }=(\mu _1,\mu _2)\in \mathbb {Z}^2$ with $\mu _1>\mu _2$, let $P_{ \varvec{\mu } } : L^2( X ) \rightarrow L^2( X )_{ \varvec{\mu } }$ be the orthogonal projector. Clearly

$$\begin{aligned} \Pi _{k \varvec{\nu } } = P_{k \varvec{\nu } } \circ \Pi _L. \end{aligned}$$

(48)

In terms of Schwartz kernels, (48) can be reformulated as follows:

$$\begin{aligned} \Pi _{k \varvec{\nu } } (x,y)= & {} d_{k \varvec{\nu } }\, \int _G \, \mathrm {d}V_G (g)\left[ \overline{\chi _{k \varvec{\nu } }(g)} \, \Pi _L \left( \widetilde{\mu }_{g^{-1}} (x), y \right) \right] . \end{aligned}$$

(49)

Using the Weyl integration, character and dimension formulae, (49) can in turn be rewritten as follows:

$$\begin{aligned}&{ \Pi _{k \varvec{\nu } } (x,y)} \nonumber \\&\quad = \frac{k \, (\nu _1 -\nu _2 )}{(2\pi )^2}\, \int _{(-\pi ,\pi )^2}\,\mathrm {d}\varvec{\vartheta }\, \left[ e^{-\imath \,k \langle \varvec{ \nu },\varvec{ \vartheta } \rangle } \, \left( e^{\imath \,\vartheta _1} -e^{\imath \,\vartheta _2}\right) \,F_{L} \left( x,y;e^{\imath \,\varvec{\vartheta }}\right) \right] , \end{aligned}$$

(50)

where for $t \in T$ we set

$$\begin{aligned} F_{L} (x,y;t) := \int _{G/T}\,\mathrm {d}V_{G/T} (gT)\, \left[ \Pi _L \left( \widetilde{\mu }_{g\,t^{-1} \, g^{-1}} (x), y \right) \right] . \end{aligned}$$

(51)

Now suppose $K\Subset (X\times X) {\setminus } \mathcal {Z}_{\varvec{ \nu }}$. We may assume without loss that K is $G\times G$-invariant. There exist $G\times G$-invariant open subsets $A,B \subset X\times X$ such that

$$\begin{aligned} K\subset A\Subset (X\times X) {\setminus } \mathcal {Z}_{\varvec{ \nu }}, \quad \mathcal {Z}_{\varvec{ \nu }}\subset B\Subset (X\times X){\setminus } K, \quad X\times X= A\cup B. \end{aligned}$$

Hence, A is a $G\times G$-invariant open neighborhood of K in $X\times X$, and the restriction of $\Pi _L$ to A is $\mathcal {C}^\infty$.

Therefore, we get a $\mathcal {C}^\infty$ function

$$\begin{aligned} R:T\times G/T\times A\rightarrow \mathbb {C}, \quad \big (t, gT, (x,y) \big ) \mapsto \Pi _L \left( \widetilde{\mu }_{g\,t^{-1} \, g^{-1}} (x), y \right) . \end{aligned}$$

With $F_L$ as in (51), we obtain a $\mathcal {C}^\infty$ function on $T\times A$ by setting

$$\begin{aligned} \beta : \big (t,(x,y) \big )\mapsto \Delta (t) \,F_{L} (x,y;t). \end{aligned}$$

Let us denote by $\mathcal {F}_T$ the Fourier transform with respect to $t\in T$ of a function on $T\times A$, viewed as a function on $\mathbb {Z}^2\times A$; then (50) may be rewritten

$$\begin{aligned} \Pi _{k \varvec{\nu } } (x,y)= & {} \frac{k }{2}\, (\nu _1 -\nu _2 )\cdot \mathcal {F}_T(\beta ) (k\,\varvec{\nu }; x,y). \end{aligned}$$

(52)

The statement of Theorem 1.1 follows from (52) and the previous considerations. $\square$

4 Proof of Theorem 1.2

We shall assume throughout this section that the assumptions of Theorem 1.2 hold.

4.1 Preliminaries

Before attacking the proof, it is in order to list some useful preliminaries (see also the discussion in §2 of [26]).

For any $m\in M$, let ${\mathrm {val}}_m:\mathfrak {g}\rightarrow T_mM$ be the evaluation map $\xi \mapsto \xi _M(m)$; similarly, for any $x\in X$ let ${\mathrm {val}}_x:\mathfrak {g}\rightarrow T_xX$ be the evaluation map $\xi \mapsto \xi _X(x)$.

4.1.1 Ray transversality and locally free actions

Since $\widetilde{\mu }$ preserves the connection 1-form, the induced cotangent action of G on $T^*X$ leaves the symplectic cone $\Sigma$ in (43) invariant. The restricted action is of course still Hamiltonian, and its moment map $\widetilde{\Phi }_G:\Sigma \rightarrow \mathfrak {g}$ is the restriction to $\Sigma$ of the cotangent Hamiltonian map on $T^*X$.

If $m\in M^G_{\mathcal {O}}$, then by equivariance $\Phi _G$ is transverse to $\mathbb {R}_+\cdot \Phi _G(m)$. Hence,

$$\begin{aligned} \mathrm {d}_m\Phi _G(T_mM)+{\mathrm {span}}\big (\Phi _G(m)\big )= \mathfrak {g}. \end{aligned}$$

(53)

Suppose $x\in \pi ^{-1}(m)\subset X$ and $r>0$, and consider $\sigma =(x,r\alpha _x)\in \Sigma$. Then, it follows from (53) that

$$\begin{aligned} \mathrm {d}_\sigma \widetilde{\Phi }_G(T_\sigma \Sigma )=\mathrm {d}_m\Phi _G(T_mM)+{\mathrm {span}}\big (\Phi _G(m)\big )= \mathfrak {g}. \end{aligned}$$

(54)

Thus $\widetilde{\Phi }_G$ is submersive at any $(x,r\alpha _x)$ with $x\in X^G_{\mathcal {O}}$. If we let $\Sigma ^G_{\mathcal {O}}\cong X^G_{\mathcal {O}}\times \mathbb {R}_+$ denote the inverse image of $X^G_{\mathcal {O}}$ in $\Sigma$, we conclude therefore that G acts locally freely on $\Sigma ^G_{\mathcal {O}}$, and this clearly implies that it acts locally freely on $X^G_{\mathcal {O}}$.

The previous implications may obviously be reversed, and we obtain the following.

Lemma 4.1

The following conditions are equivalent:

• 1. $\Phi _G$ is transverse to $\mathbb {R}_+\cdot \imath \,\varvec{\nu }$;

• 2. $\widetilde{\mu }$ is locally free on $X^G_{\mathcal {O}}$;

• 3. for every $x\in X^G_{\mathcal {O}}$, ${\mathrm {val}}_x$ is injective;

• 4. for every $m\in M^G_{\mathcal {O}}$, ${\mathrm {val}}_m$ is injective on $\Phi _G(m)^{\perp _{\mathfrak {g}}}$.

4.1.2 The vector field $\Upsilon =\Upsilon _{ \mu ,\varvec{ \nu }}$

Let us construct the normal vector field $\Upsilon =\Upsilon _{ \mu ,\varvec{ \nu }}$ to $M^G_{\mathcal {O}}$ appearing in the statement of Theorem 1.2.

By definition, $m\in M^G_{\mathcal {O}_{ \varvec{ \nu } }}$ if and only if $\Phi _G(m)$ is similar to $\imath \,\lambda _{\varvec{ \nu } } ( m )\,D_{\varvec{\nu }}$, for some $\lambda _{\varvec{ \nu } } ( m )>0$ (Definition 1.2). Equating norms and traces, we obtain

$$\begin{aligned} \lambda _{\varvec{ \nu } } ( m ) = \frac{\Vert \Phi _G ( m ) \Vert }{\Vert \varvec{ \nu } \Vert } = -\imath \, \frac{{\mathrm {trace}}\big (\Phi _G(m)\big )}{\nu _1+\nu _2} \quad \left( m\in M^G_{\mathcal {O}_{ \varvec{ \nu } }} \right) . \end{aligned}$$

(55)

Since $\nu _1>\nu _2$, there exists a unique coset $h_m\,T\in G/T$ such that

$$\begin{aligned} \Phi _G (m) = \imath \,\lambda _{\varvec{ \nu } } ( m ) \, h_m \, D_{\varvec{\nu }} \, h_m^{-1}. \end{aligned}$$

(56)

Let us set $\varvec{ \nu }_\perp := \begin{pmatrix} -\nu _2&\nu _1 \end{pmatrix}^t$, and define $\varvec{ \rho }= \varvec{ \rho }_{ \varvec{ \nu }}: M^G_{\mathcal {O}_{ \varvec{ \nu } }} \rightarrow \mathfrak {g}$ by setting

$$\begin{aligned} \varvec{ \rho } (m ) := \imath \, h _m \, D_{\varvec{ \nu }_\perp } \,h_m^{-1} \quad \left( m \in M^G_{\mathcal {O}_{ \varvec{ \nu } }} \right) . \end{aligned}$$

(57)

Then, $\varvec{ \rho } (m )_M\in \mathfrak {X}(M)$ is the vector field on M induced by $\varvec{ \rho } (m )\in \mathfrak {g}$; its evaluation at $m'\in M$ is $\varvec{ \rho } (m )_M(m')$ (and similarly for X).

Definition 4.1

The vector field $\Upsilon =\Upsilon _{\mu , \varvec{ \nu } }$ along $M^G_{\mathcal {O}_{ \varvec{ \nu } }}$ is

$$\begin{aligned} \Upsilon (m) := J_m \big (\varvec{ \rho } (m )_M (m)\big ) \quad \left( m\in M^G_{\mathcal {O}_{ \varvec{ \nu } }}\right) . \end{aligned}$$

With abuse of notation, recalling (4) we shall also denote by $\Upsilon$ the vector field along $X^G_{\mathcal {O}_{ \varvec{ \nu } }}$ given by

$$\begin{aligned} \Upsilon (x) := J_x \big (\varvec{ \rho } (m_x )_X (x)\big ), \quad m_x:=\pi (x). \end{aligned}$$

Notice that

$$\begin{aligned} \langle \Phi _G(m), \varvec{\rho } (m)\rangle =\lambda _{\varvec{ \nu } } ( m ) \, \left\langle \varvec{\nu }, \varvec{\nu }_\perp \right\rangle =0 \quad \left( m\in M^G_{\mathcal {O}_{ \varvec{ \nu } }}\right) . \end{aligned}$$

(58)

Therefore, in view of (6) for any $x\in \pi ^{-1}(m)$ we have

$$\begin{aligned} \varvec{ \rho } (m )_X (x)=\varvec{ \rho } (m )_M^\sharp (x)=\varvec{ \rho } (m )_M(m)^\sharp (x). \end{aligned}$$

(59)

Hence, $\Upsilon (x)=\Upsilon (m)^\sharp$ if $m=\pi (x)$.

4.1.3 A spectral characterization of $G\cdot M^T_{\varvec{\nu }}$

Suppose that $-\imath \,\Phi _G(m)$ has eigenvalues $\lambda _1(m) \ge \lambda _2(m)$. Then, $m\in M^G_{\mathcal {O}}$ if and only if $\lambda _1(m)\nu _2- \lambda _2(m)\,\nu _1=0$. We shall give a similar spectral characterization of $G\cdot M^T_{\varvec{\nu }}$. Notice that if $\lambda _1(m) = \lambda _2(m)$, then $\Phi _G(m)$ is a multiple of the identity, hence certainly $m\not \in G\cdot M^T_{\varvec{\nu }}$. Thus we may as well assume that $\lambda _1(m) > \lambda _2(m)$.

Proposition 4.1

Suppose $m\in M$, and let the eigenvalues of $-\imath \,\Phi _G(m)$ be $\lambda _1(m) > \lambda _2(m)$. Then, $m\in G\cdot M^T_{\varvec{\nu }}$ if and only if

$$\begin{aligned} t(m,\varvec{\nu }):= \frac{\lambda _1(m) \,\nu _2- \lambda _2(m)\,\nu _1}{(\nu _1+\nu _2) \,\big (\lambda _1(m) - \lambda _2(m)\big )} \in [0,1/2). \end{aligned}$$

(60)

Proof of Proposition 4.1

Let us set $\varvec{\lambda }(m):=\big ( \lambda _1(m),\,\lambda _2(m) \big )$, and let $D_{\varvec{\lambda }}$ be the corresponding diagonal matrix. By definition, $m\in G\cdot M^T_{\varvec{\nu }}$ if and only if there exists $g\in SU(2)\leqslant G$ such that ${\mathrm {diag}}\left( g \,D_{\varvec{\lambda }} \,g^{-1}\right) \in \mathbb {R}_+\cdot \varvec{\nu }$. This is equivalent to the condition that there exist $u,w\in \mathbb {C}$ such that

$$\begin{aligned} \begin{pmatrix} u &{} -\overline{w}\\ w &{} \overline{u} \end{pmatrix} \, D_{\varvec{\lambda }}\, \begin{pmatrix} \overline{u}&{} \overline{w}\\ -w &{}u \end{pmatrix}= c \, \begin{pmatrix} \nu _1&{} a\\ \overline{a}&{}\nu _2 \end{pmatrix}, \end{aligned}$$

(61)

for some $c>0$ and $a\in \mathbb {C}$. If we set $t=|w|^2$, we conclude that $m\in M^G_{\mathcal {O}}$ if and only if there exists $t\in [0,1]$ such that

$$\begin{aligned} \varvec{\lambda }_t(m):= \begin{pmatrix} (1-t)\,\lambda _1(m) + t \,\lambda _2(m) \\ t\,\lambda _1(m) + (1-t)\,\lambda _2(m) \end{pmatrix}\in \mathbb {R}_+\, \begin{pmatrix} \nu _1\\ \nu _2 \end{pmatrix}. \end{aligned}$$

(62)

The condition $\varvec{\lambda }_t(m)\wedge \varvec{\nu }=\mathbf {0}$ translates into the equality $t = t(m,\varvec{\nu })$. Hence, we need to have $t(m,\varvec{\nu })\in [0,1]$. Given this, $\varvec{\lambda }_t(m)$ is a positive multiple of $\varvec{\nu }$ if and only if

$$\begin{aligned} \big (1-t(m,\varvec{\nu })\big )\,\lambda _1(m) + t(m,\varvec{\nu }) \,\lambda _2(m) > t(m,\varvec{\nu })\,\lambda _1(m) + \big (1-t(m,\varvec{\nu })\big )\,\lambda _2(m), \end{aligned}$$

and this is equivalent to $t(m,\varvec{\nu })<1/2$.

Conversely, suppose that $t(m,\varvec{\nu })\in [0,1/2)$, and define

$$\begin{aligned} g:= \begin{pmatrix} \sqrt{1-t(m,\varvec{\nu })}&{}-\sqrt{t(m,\varvec{\nu })}\\ \sqrt{t(m,\varvec{\nu })}&{} \sqrt{1-t(m,\varvec{\nu })} \end{pmatrix}. \end{aligned}$$

$\square$

4.2 The proof

Proof of Theorem 1.2

As $\Phi _G$ is equivariant, it is transverse to $\mathbb {R}_+\cdot \imath \,D_{\varvec{\nu }}$ if and only if it is transverse to $\mathbb {R}_+\cdot \mathcal {O}$. Given that $\nu _1>\nu _2$, $\mathcal {O}$ is two dimensional (and diffeomorphic to $S^2$); therefore, $\mathbb {R}_+\cdot \mathcal {O}$ has codimension 1 in $\mathfrak {g}$. Similarly, $\mathbb {R}_+\cdot \imath \,D_{\varvec{\nu }}$ has codimension 1 in $\mathfrak {t}^\vee$. Given that $\mathbf {0}\not \in \Phi _T(M)$, we conclude the following.

Step 4.1

$M^G_{\varvec{\nu }}$, $M^G_{\mathcal {O}}$ and $M^T_{\varvec{\nu }}$ are compact and smooth (real) submanifolds of M. $M^G_{\varvec{\nu }}$ has codimension 3, and $M^G_{\mathcal {O}}$ and $M^T_{\varvec{\nu }}$ are hypersurfaces.

The Weyl chambers in $\mathfrak {t}$ are the half-planes

$$\begin{aligned} \mathfrak {t}_+:=\big \{ \varvec{\mu } \,: \, \mu _1>\mu _2\big \},\quad \mathfrak {t}_-:=\big \{ \varvec{\mu } \,: \, \mu _1<\mu _2\big \}, \end{aligned}$$

and clearly with our identifications $\imath \,D_{\varvec{\nu }}\leftrightarrow \varvec{\nu }\in \mathfrak {t}_+$. Since $\Phi _G(M)\cap \mathfrak {t}_+$ is a convex polytope [14, 15, 18], $\Phi _G(M)\cap \mathbb {R}_+\cdot \imath \,D_{\varvec{\nu }}$ is a closed segment J. Furthermore, for any $a\in J$, the inverse image $\Phi _G^{-1}(a)\subseteq M$ is also connected [19, 21]. Thus we obtain the following conclusion.

Step 4.2

$M^G_{\varvec{\nu }}$, $M^G_{\mathcal {O}}$ and $M^T_{\varvec{\nu }}$ are connected.

Proof of Step 4.2

The previous considerations immediately imply that $M^G_{\varvec{\nu }}$ is connected. Given this, since $M^G_{\mathcal {O}}=G\cdot M^G_{\varvec{\nu }}$, the connectedness of G implies the one of $M^G_{\mathcal {O}}$. Let us consider $M^T_{\varvec{\nu }}$. Since $\Phi _T(M)$ is a convex polytope [1, 14], $\Phi _T(M)\cap \mathbb {R}_+\cdot \imath \,D_{\varvec{\nu }}$ is also a closed segment $J'$. The statement follows since the fibers of $\Phi _T$ are connected again by [19, 21]. $\square$

For any $m\in M^G_{\mathcal {O}}$, let us set

$$\begin{aligned} M^G_{\Phi _G(m)}:=\Phi _G^{-1}\big (\mathbb {R}_+\cdot \Phi _G(m)\big ). \end{aligned}$$

Since $\Phi _G$ is transverse to $\mathbb {R}_+\cdot \varvec{\nu }$, by equivariance it is also transverse to $\mathbb {R}_+\cdot \Phi _G(m)$; hence, $M^G_{\Phi _G(m)}$ is also a connected real submanifold of M, of real codimension 3 and contained in $M^G_{\mathcal {O}}$.

Let us consider the normal bundle $N\big (M^G_{\Phi _G(m)}\big )$ to $M^G_{\Phi _G(m)}\subset M$. For any $\xi \in \mathfrak {g}$, let $\xi ^\perp \subset \mathfrak {g}$ be the orthocomplement to $\xi$. Under the equivariant identification $\mathfrak {g}\cong \mathfrak {g}^\vee$, $\xi ^\perp$ corresponds to $\xi ^0$.

For any subset $L\subseteq \mathfrak {g}$, let $L^{\perp _{\mathfrak {g}}}$ denote the orthocomplement of L (i.e., of the linear span of L) under the pairing $\langle \cdot , \cdot \rangle _{\mathfrak {g}}$.

Lemma 4.2

For any $m\in M^G_{\mathcal {O}}$, we have

$$\begin{aligned} N_m\left( M^G_{\Phi _G(m)}\right) = J_m\circ {\mathrm {val}}_m\left( \Phi _G(m)^{\perp _{\mathfrak {g}}} \right) . \end{aligned}$$

Simlarly, for any $m\in M^T_{\varvec{\nu }}$, we have

$$\begin{aligned} N_m\left( M^T_{\varvec{\nu }}\right) = J_m\circ {\mathrm {val}}_m\left( (\imath \,\varvec{\nu })^{\perp _{\mathfrak {t}}} \right) . \end{aligned}$$

Proof of Lemma 4.2

If $v\in T_mM^G_{\Phi _G(m)}$, then $\mathrm {d}_m\Phi _G(v)= a\,\Phi _G(m)$ for some $a\in \mathbb {R}$. Given $\eta \in \Phi _G(m)^{\perp _{\mathfrak {g}}}$, and with $\rho$ as in (1), we have

$$\begin{aligned} \rho _m \Big (J_m \big (\eta _M(m)\big ), v\Big )= & {} \omega _m \big (\eta _M(m), v \big ) = \mathrm {d}_m\Phi ^\eta (v)\\= & {} \langle \mathrm {d}_m\Phi (v), \eta \rangle _{\mathfrak {g}} = a \langle \Phi _G(m), \eta \rangle _{\mathfrak {g}} =0. \end{aligned}$$

Therefore, $J_m\circ {\mathrm {val}}_m\left( \Phi _G(m)^{\perp _{\mathfrak {g}}} \right) \subseteq N_m(M^G_{\Phi _G(m)})$. Since both $\Phi _G(m)^{\perp _{\mathfrak {g}}}$ and $N_m(M^G_{\Phi _G(m)})$ are three dimensional, it suffices to recall that by Lemma 4.1${\mathrm {val}}_m$ is injective when restricted to $\Phi _G(m)^{\perp _{\mathfrak {g}}}$.

The proof of the second statement is similar. $\square$

For any vector subspace $L\subseteq \mathfrak {g}$, let us set $L_M(m):={\mathrm {val}}_m \big (L)\subseteq T_mM$ ($m\in M$). For any $m\in M^G_{\mathcal {O}}$, given that $M^G_{\mathcal {O}}$ is the G-saturation of $M^G_{\Phi _G(m)}$, we have

$$\begin{aligned} T_mM^G_{\mathcal {O}} = T_mM^G_{\Phi _G(m)} + \mathfrak {g}_M(m). \end{aligned}$$

(63)

Therefore, passing to $\rho _m$-orthocomplements

$$\begin{aligned} N_m\left( M^G_{\mathcal {O}}\right) = N_m\left( M^G_{\Phi _G(m)}\right) \cap \mathfrak {g}_M(m)^{\perp _{\rho _m}}. \end{aligned}$$

(64)

We conclude from Lemma 4.2 and (63) that $N_m\left( M^G_{\mathcal {O}}\right)$ is the set of all vectors $J_m\big (\eta _M(m)\big )\in T_mM$ where $\eta \in \Phi _G(m)^{\perp _{\mathfrak {g}}}$ and $\rho _m\big (J_m\big (\eta _M(m)\big ), \xi _M(m)\big )=0$ for every $\xi \in \mathfrak {g}$. From this remark, we can draw the following conclusion.

Step 4.3

Let $\Upsilon =\Upsilon _{\mu ,\varvec{\nu }}$ be as in Sect. 4.1.2. Then, for any $m\in M^G_{\mathcal {O}}$ we have

$$\begin{aligned} N_m\left( M^G_{\mathcal {O}}\right) = {\mathrm {span}}\big (\Upsilon (m)\big ). \end{aligned}$$

In particular, $M^G_{\mathcal {O}}$ is orientable.

Proof of Step 4.3

By the above,

$$\begin{aligned}&{ N_m\left( M^G_{\mathcal {O}}\right) } \nonumber \\&\quad = \left\{ J_m\big (\eta _M(m)\big ) : \eta \in \Phi _G(m)^{\perp _{\mathfrak {g}}}\,\wedge \, \rho _m\Big (J_m\big (\eta _M(m)\big ), \xi _M(m)\Big ) =0 \, \forall \xi \in \mathfrak {g}\right\} \nonumber \\&\quad = \left\{ J_m\big (\eta _M(m)\big ) : \eta \in \Phi _G(m)^{\perp _{\mathfrak {g}}}\,\wedge \, \omega _m\big (\eta _M(m), \xi _M(m)\big ) =0 \, \forall \xi \in \mathfrak {g}\right\} \nonumber \\&\quad = \left\{ J_m\big (\eta _M(m)\big ) : \eta \in \Phi _G(m)^{\perp _{\mathfrak {g}}}\,\wedge \, \eta _M(m)\in \ker (\mathrm {d}_m\Phi _G)\right\} \nonumber \\&\quad =\left\{ J_m\big (\eta _M(m)\big ) : \eta \in \Phi _G(m)^{\perp _{\mathfrak {g}}}\,\wedge \, \big [\eta , \Phi _G(m)\big ]=0\right\} . \end{aligned}$$

(65)

The latter equality holds because, by the equivariance of $\Phi _G$, we have

$$\begin{aligned} \mathrm {d}_m\Phi _G\big (\eta _M(m)\big )= & {} \left. \frac{\mathrm {d}}{\mathrm {d}t} \Phi _G\left( \mu _{e^{t\eta }}(m)\right) \right| _{t=0} =\left. \frac{\mathrm {d}}{\mathrm {d}t} {\mathrm {Ad}}_{e^{t\eta }}\Phi _G\left( m\right) \right| _{t=0}\nonumber \\= & {} \big [\eta , \Phi _G(m)\big ]. \end{aligned}$$

There exists a unique $h_m\,T\in G/T$ such that $\Phi _G(m)=\imath \,\lambda _{\varvec{\nu }}(m)\,h_m\,D_{\varvec{\nu }}\,h_m^{-1}$. It is then clear that $\langle \Phi _G(m),\eta \rangle _{\mathfrak {g}}=0$ and $\big [\eta , \Phi _G(m)\big ]=0$ if and only if

$$\begin{aligned} \eta \in {\mathrm {span}}\left( \imath \,h_m\,D_{\varvec{\nu }_\perp }\,h_m^{-1}\right) ={\mathrm {span}}\big (\varvec{\rho }(m)\big ), \end{aligned}$$

where $\varvec{\rho }(m)$ is as in (57). This completes the proof of Step 4.3.

$\square$

Step 4.4

$M^G_{\mathcal {O}}\cap M^T_{\varvec{\nu }}=M^G_{\varvec{\nu }}$.

Proof of Step 4.4

Obviously, $M^G_{\mathcal {O}}\cap M^T_{\varvec{\nu }}\supseteq M^G_{\varvec{\nu }}$. Conversely, suppose $m\in M^G_{\mathcal {O}}\cap M^T_{\varvec{\nu }}$. Then, on the one hand $\Phi _G(m)$ is similar to a positive multiple of $\imath \,D_{\varvec{\nu }}$: for a unique $h_m\,T\in G/T$,

$$\begin{aligned} \Phi _G(m)=\imath \,\lambda _{\varvec{\nu }}(m)\,h_m\,D_{\varvec{\nu }}\,h_m^{-1}. \end{aligned}$$

(66)

We can assume without loss that $h_m\in SU(2)$. On the other ${\mathrm {diag}}\big (\Phi _G(m)\big )$ is a positive multiple of $\imath \,\varvec{\nu }$. Hence, the diagonal of $h_m\,D_{\varvec{\nu }}\,h_m^{-1}$ is a positive multiple of $\varvec{\nu }$. Let us write $h_m$ as in (61) and argue as in the proof of Proposition 4.1; using that $\nu _1^2\ne \nu _2^2$, one concludes readily that $h_m$ is diagonal. Hence, $h_m\,D_{\varvec{\nu }}\,h_m^{-1}=D_{\varvec{\nu }}$, and so $\Phi _G(m)\in \mathbb {R}_+\cdot \imath \varvec{\nu }$. Thus $m\in M^G_{\varvec{\nu }}$.

$\square$

Step 4.5

For any $m\in M^G_{\varvec{\nu }}$, $T_mM^G_{\mathcal {O}} = T_mM^T_{\varvec{\nu }}$.

Proof of Step 4.5

If $m\in M^G_{\varvec{\nu }}$, then $h_m=I_2$ in (56) and (57); therefore, $\Upsilon (m)=J_M\left( (\imath \,\varvec{\nu }_\perp ) (m)\right)$. Hence, $N_m\left( M^G_{\mathcal {O}}\right) = {\mathrm {span}}\left( J_m\left( (\imath \,\varvec{\nu }_\perp ) (m)\right) \right)$. The claim follows from this and Lemma 4.2.

$\square$

Step 4.6

$M^G_{\mathcal {O}}= \partial \left( G\cdot M^T_{\varvec{\nu }}\right)$.

Proof of Step 4.6

Suppose $m\in M^G_{\mathcal {O}}$. Thus $\Phi _G(m)=\imath \,\lambda _{\varvec{\nu }}(m)\,h_m\,D_{\varvec{\nu }}\,h_m^{-1}$for a unique $h_m\,T\in G/T$. Let us choose $\delta >0$ arbitrarily small, and let $M(m,\delta )\subseteq M$ be the open ball centered at m and radius $\delta$ in the Riemannian distance on M. Since $\Phi _G$ is transverse to $\mathbb {R}_+\cdot \imath \,\varvec{\nu }$, there exists $\epsilon _1>0$ such that the following holds. For every $\epsilon \in (-\epsilon _1,\epsilon _1)$, there exists $m'\in M(m,\delta )$ with

$$\begin{aligned} \Phi _G(m')= \imath \, \lambda (m')\,h_m\,D_{\varvec{\nu }+\epsilon \,\varvec{\nu }_\perp }\,h_m^{-1} \end{aligned}$$

(67)

for some $\lambda (m')>0$ (see §2 of [28]). This implies that the eigenvalues of $-\imath \,\Phi _G(m')$ are

$$\begin{aligned} \lambda _1(m'):=\lambda (m')\,(\nu _1-\epsilon \,\nu _2), \quad \lambda _2(m'):=\lambda (m')\,(\nu _2+\epsilon \,\nu _1). \end{aligned}$$

Therefore, the invariant defined in (60) takes the following value at $m'$:

$$\begin{aligned} t(m',\varvec{\nu })=-\frac{\epsilon }{\nu _1+\nu _2}\, \frac{\nu _1^2+\nu _2^2}{(\nu _1-\nu _2)-\epsilon \,(\nu _1+\nu _2)}. \end{aligned}$$

(68)

Therefore, if $\epsilon \,(\nu _1+\nu _2)>0$ (and $\epsilon$ is sufficiently small) then $m'\not \in G\cdot M^T_{\varvec{\nu }}$ by Proposition 4.1. This implies $M^G_{\mathcal {O}}\subseteq \partial \left( G\cdot M^T_{\varvec{\nu }}\right)$.

To prove the reverse inclusion, assume that $m\in G\cdot M^T_{\varvec{\nu }}{\setminus } M^G_{\mathcal {O}}$. Then, $t(m,\varvec{\nu })\in [0,1/2)$ by Proposition 4.1. Furthermore, $t(m,\varvec{\nu })\ne 0$, for otherwise $m\in M^G_{\mathcal {O}}$. Hence, $t(m,\varvec{\nu })\in (0,1/2)$; by continuity, then $t(m',\varvec{\nu })\in (0,1/2)$ for every $m'$ in a sufficiently small open neighborhood of m. Hence, Proposition 4.1 implies that $G\cdot M^T_{\varvec{\nu }}{\setminus } M^G_{\mathcal {O}}$ contains an open neighborhood of m in M. Thus $G\cdot M^T_{\varvec{\nu }}{\setminus } M^G_{\mathcal {O}}$ is open, and in particular $m\not \in \partial \left( G\cdot M^T_{\varvec{\nu }}\right)$. $\square$

Step 4.7

$\Upsilon$ is outer oriented if $\nu _1+\nu _2>0$ and inner oriented if $\nu _1+\nu _2<0$.

Proof of Step 4.7

Let denote by $\mathcal {B}_{\varvec{\nu }}$ the collection of all $B\in \mathfrak {g}$ such that ${\mathrm {diag}}\left( g\,B\,g^{-1}\right) \in \mathbb {R}_+\,\imath \,\varvec{\nu }$ for some $g\in G$. Thus $\mathcal {B}_{\varvec{\nu }}$ is a conic and invariant closed subset of $\mathfrak {g}{\setminus } \{0\}$; in addition, $m\in G\cdot M^T_{\varvec{\nu }}$ if and only if $\Phi _G(m)\in \mathcal {B}_{\varvec{\nu }}$.

If $\lambda _1(B)\ge \lambda _2(B)$ are the eigenvalues of $-\imath \,B$, then Proposition 4.1 implies that $B\in \mathcal {B}_{\varvec{\nu }}$ if and only if $\lambda _1(B)> \lambda _2(B)$ and

$$\begin{aligned} t(B,\varvec{\nu }):= \frac{\lambda _1(B) \,\nu _2- \lambda _2(B)\,\nu _1}{(\nu _1+\nu _2) \,\big (\lambda _1(B) - \lambda _2(B)\big )} \in [0,1/2). \end{aligned}$$

In particular, if $t(B,\varvec{\nu })\in (0,1/2)$, then B belongs to the interior of $\mathcal {B}_{\varvec{\nu }}$.

Suppose $m\in M^G_{\varvec{\nu }}$ and consider the path

$$\begin{aligned} \gamma _1:\tau \in (-\epsilon ,\epsilon )\mapsto \Phi _G\big (m+\tau \,\Upsilon (m)\big )\in \mathfrak {g}, \end{aligned}$$

defined for sufficiently small $\epsilon >0$; the expression $m+\tau \,\Upsilon (m)\in M$ is meant in an adapted coordinate system on M centered at m. Then,

$$\begin{aligned} \gamma _1(0)= & {} \Phi _G(m)=\imath \,\lambda _{\varvec{\nu }}(m) \, D_{\varvec{\nu }}, \end{aligned}$$

(69)

$$\begin{aligned} \dot{\gamma }_1(0)= & {} \omega _m \big (\cdot , \Upsilon (m)\big )=\rho _m\big (\cdot , (\imath \,\varvec{\nu }_\perp )_M(m)\big ). \end{aligned}$$

(70)

Let us consider a smooth positive function, $y:(-\epsilon ,\epsilon ) \rightarrow \mathbb {R}_+$, to be determined but subject to the condition $y(0)=\lambda _{\varvec{\nu }}(m)$. Let us define a second smooth path of the form

$$\begin{aligned} \gamma _2(\tau ):= \imath \,y(\tau )\,{\mathrm {Ad}}_{e^{\tau \, \varvec{\xi }}}\left( D_{\varvec{\nu }+a\,\tau \,\varvec{\nu }_\perp }\right) , \end{aligned}$$

(71)

where $a>0$ is a constant also to be determined.

Then,

$$\begin{aligned} \gamma _1(0)= & {} \gamma _2(0) \nonumber \\ \dot{\gamma }_2(0)= & {} \imath \, \left[ \dot{y}(0)\, D_{\varvec{\nu }}+ \lambda _{\varvec{\nu }}(m)\, [\varvec{\xi },\varvec{\nu }] + a\,\lambda _{\varvec{\nu }}(m)\, D_{\varvec{\nu }_\perp }\right] . \end{aligned}$$

(72)

Clearly, we can choose $a>0$ uniquely so that

$$\begin{aligned} a\,\lambda _{\varvec{\nu }}(m)\,\Vert \varvec{\nu }\Vert ^2= \rho _m\big ((\imath \,\varvec{\nu }_\perp )_M(m), (\imath \,\varvec{\nu }_\perp )_M(m)\big ), \end{aligned}$$

(73)

so that $\left\langle \dot{\gamma }_2(0),\varvec{\nu }_\perp \right\rangle = \left\langle \dot{\gamma }_1(0),\varvec{\nu }_\perp \right\rangle$. Having fixed a, we can then choose $\dot{y}(0)$ uniquely so that

$$\begin{aligned} \dot{y}(0)\, \Vert \varvec{\nu }\Vert ^2=\rho _m \big ((\imath \,\varvec{\nu })_M(m), (\imath \,\varvec{\nu }_\perp )_M(m)\big ), \end{aligned}$$

(74)

so that we also have $\left\langle \dot{\gamma }_2(0),\varvec{\nu }\right\rangle = \left\langle \dot{\gamma }_1(0),\varvec{\nu }\right\rangle$. Finally, if we set

$$\begin{aligned} \varvec{\upsilon }_1:= \begin{pmatrix} 0&{}1\\ -1&{}0 \end{pmatrix}, \quad \varvec{\upsilon }_2:= \begin{pmatrix} 0&{}\imath \\ \imath &{}0 \end{pmatrix} \end{aligned}$$

we can choose $\varvec{\xi }\in {\mathrm {span}}_{\mathbb {R}}\big \{\varvec{\upsilon }_1,\,\varvec{\upsilon }_2\big \}$ uniquely so that

$$\begin{aligned} \lambda _{\varvec{\nu }}(m)\,\langle [\varvec{\xi },\varvec{\nu }], \varvec{\upsilon }_j\rangle =\rho _m\big (\varvec{\upsilon }_{jM}(m), (\imath \,\varvec{\nu }_\perp )_M(m)\big ), \end{aligned}$$

(75)

so that in addition $\left\langle \dot{\gamma }_2(0),\varvec{\upsilon }_j\right\rangle = \left\langle \dot{\gamma }_1(0),\varvec{\upsilon }_j\right\rangle$ for $j=1,2$. With these choices, $\gamma _1$ and $\gamma _2$ agree to first order at 0.

Let us remark that when $\tau$ is sufficiently small $\gamma _2(\tau )$ has eigenvalues

$$\begin{aligned} \lambda _1\big (\gamma _2(\tau )\big )= y(\tau )\, (\nu _1-a\,\tau \,\nu _2)> \lambda _2\big (\gamma _2(\tau )\big )= y(\tau )\, (\nu _2+a\,\tau \,\nu _1). \end{aligned}$$

Hence,

$$\begin{aligned} t(B,\varvec{\nu })=-\frac{a\,\tau }{\nu _1+\nu _2}\,\frac{\nu _1^2+\nu _2^2}{\nu _1-\nu _2+a\tau \,(\nu _1+\nu _2)}. \end{aligned}$$

(76)

Thus, if $\nu _1+\nu _2>0$, then $\gamma _2(\tau )\not \in \mathcal {B}_{\varvec{\nu }}$ when $\tau \in (0,\epsilon )$; since $\gamma _1$ and $\gamma _2$ agree to second order at 0, we also have $\Phi _G\big (m+\tau \,\Upsilon (m)\big )\not \in \mathcal {B}_{\varvec{\nu }}$ when $\tau \sim 0^+$. Hence, $\Upsilon$ is outer oriented at m and thus everywhere on $M^G_{\mathcal {O}}$.

The argument when $\nu _1+\nu _2<0$ is similar. $\square$

The proof of Theorem 1.2 is complete. $\square$

5 Proof of Theorem 1.3

5.1 Preliminaries

5.1.1 Recalls on Szegö kernels

Let $\Pi$, $\Pi (\cdot ,\cdot )$ and $\Pi _{\varvec{\nu }}$, $\Pi _{\varvec{\nu }}(\cdot ,\cdot )$ be as in (5) and (11). For any $x,y\in X$, we have

$$\begin{aligned} \Pi _{\varvec{\nu }}(x,y) = d_{\varvec{\nu }}\,\int _G \overline{\chi _{\varvec{\nu }}(g)}\,\Pi \left( \widetilde{\mu }_{g^{-1}}(x),y\right) \, \mathrm {d}V_G(g). \end{aligned}$$

(77)

In view of (9) and the Weyl integration formula (3.1.1), (77) can be rewritten

$$\begin{aligned} \Pi _{\varvec{\nu }}(x,y)= & {} d_{\varvec{\nu }}\,\int _T t^{-\varvec{\nu }}\,\Delta (t)\,F(t;x,y)\,\mathrm {d}V_T(t), \end{aligned}$$

(78)

where $t^{-\varvec{\nu }}=t_1^{-\nu _1}\,t^{-\nu _2}$, and

$$\begin{aligned} F(t;x,y):=\int _{G/T}\Pi \left( \widetilde{\mu }_{gt^{-1}g^{-1}}(x),y\right) \,\mathrm {d}V_{G/T}(g\,T). \end{aligned}$$

(79)

We have already used the structure of the wave front of $\Pi$ in the proof of Theorem 1.1 (see (44)). In the proof of Theorem 1.3, we need to exploit the explicit description of $\Pi$ as an FIO developed in [4] (see also the discussions in [2, 30, 34]).

Namely, up to a smoothing contribution, we have

$$\begin{aligned} \Pi (x,y) \sim \int _0^{+\infty } e^{\imath \,u\,\psi (x,y)}\,s(x,y,u)\,\mathrm {d}u, \end{aligned}$$

(80)

where $\psi$ is essentially determined by the Taylor expansion of the metric along the diagonal and s is a semiclassical symbol admitting an asymptotic expansion $s(x,y,u)\sim \sum _{j\ge 0}u^{d-j}\,s_j(x,y)$. The differential of $\psi$ along the diagonal is

$$\begin{aligned} \mathrm {d}_{\left( x,x\right) }\psi = (\alpha _x,-\alpha _{x})\quad (x\in X). \end{aligned}$$

(81)

5.1.2 An a priori polynomial bound

Let us record the following rough a priori polynomial bound.

Lemma 5.1

There is a constant $C_{\varvec{\nu }}>0$ such that for any $x\in X$ one has

$$\begin{aligned} |\Pi _{k\varvec{\nu }}(x,x)|\le C_{\varvec{\nu }}\,k^{d+1} \end{aligned}$$

for $k\gg 0$.

Proof

Let $r:S^1\times X\rightarrow X$ be the standard structure action on the unit circle bundle X. As in 2.1, let

$$\begin{aligned} H(X)=\bigoplus _{l=0}^{+\infty }H(X)_l \end{aligned}$$

be the decomposition of H(X) as a direct sum of isotypes for the $S^1$-action.

Since $\widetilde{\mu }$ commutes with the structure action of $S^1$ on X, we have

$$\begin{aligned} H(X)_{k\varvec{\nu }}=\bigoplus _{l=0}^{+\infty }H(X)_{k\varvec{\nu }}\cap H(X)_l. \end{aligned}$$

On the other hand, by the theory of [12] we have $H(X)_{k\varvec{\nu }}\cap H(X)_l\ne (0)$ only if the highest weight vector $\mathbf {r}(k\varvec{\nu })$ of the representation indexed by $k\,\varvec{\nu }$ satisfies

$$\begin{aligned} \mathbf {r}(k\varvec{\nu })=(k\,\nu _1-1,k\nu _2)= k\,\varvec{\nu }+(-1,0)\in l\,\Phi _G(M)\subseteq \mathfrak {g}. \end{aligned}$$

(82)

Let us define

$$\begin{aligned} a_G:=\min \Vert \Phi _G\Vert ,\,\,\,\,A_G:=\max \Vert \Phi _G\Vert . \end{aligned}$$

Thus $A_G\ge a_G>0$. Therefore, we need to have

$$\begin{aligned} l\, a_G\le \Vert \mathbf {r}(k\varvec{\nu })\Vert \le k\,\Vert \varvec{\nu }\Vert +1\,\Rightarrow \, l\le L_1(k):=\left\lceil \frac{\Vert \varvec{\nu }\Vert }{a_G}\,k+\frac{1}{a_G}\right\rceil . \end{aligned}$$

(83)

Similarly,

$$\begin{aligned} k\,\Vert \varvec{\nu }\Vert -1\le \Vert \mathbf {r}(k\varvec{\nu })\Vert \le l\,A_G\,\Rightarrow \, L_2(k):=\left\lfloor \frac{\Vert \varvec{\nu }\Vert }{A_G}\,k-\frac{1}{A_G}\right\rfloor \le l. \end{aligned}$$

(84)

On the other hand, in view of the asymptotic expansion of $\Pi _k(x,x)$ from [7, 32, 34] we also have $\Pi _l(x,x)\le 2\,\left( l/\pi \right) ^d$ for $l\gg 0$. We conclude that

$$\begin{aligned} \Pi _{k\varvec{\nu }}(x,x)\le & {} \sum _{l=L_1(k)} ^{L_2(k)}\Pi _l(x,x)\le \frac{2}{\pi ^d}\,\sum _{l=L_1(k)} ^{L_2(k)}l^d\le C_{\varvec{\nu }}\,k^{d+1} \end{aligned}$$

(85)

for some constant $C_{\varvec{\nu }}>0$. $\square$

5.2 The proof

We shall use the following notational short-hand. If $x\in X$, $g\in G$, $t\in T$, let us set

$$\begin{aligned} x(g,t):=\widetilde{\mu }_{g\,t^{-1}\,g^{-1}}(x); \end{aligned}$$

similarly, if $m\in M$

$$\begin{aligned} m(g,t):=\mu _{g\,t^{-1}\,g^{-1}}(m). \end{aligned}$$

If $t=e^{i\varvec{\vartheta }}:=\left( e^{i\vartheta _1},e^{i\vartheta _2}\right)$, we shall write $x(g,t)=x(g,\varvec{\vartheta })$, $m(g,t)=m(g,\varvec{\vartheta })$. Since $\widetilde{\mu }$ is a lifting of $\mu$, if $m=\pi (x)$, then

$$\begin{aligned} m(g,\varvec{\vartheta })= \pi \big (x(g,\varvec{\vartheta })\big ). \end{aligned}$$

Proof of Theorem 1.3

If we replace $\varvec{\nu }$ by $k\,\varvec{\nu }$ in (78) and use angular coordinates on T, we obtain

$$\begin{aligned}&{ \Pi _{k\varvec{\nu }}(x,y)}\nonumber \\&\quad =\frac{k\,(\nu _1-\nu _2)}{(2\pi )^2}\, \int _{-\pi }^\pi \,\int _{-\pi }^\pi \,e^{-ik\langle \varvec{\nu },\varvec{\vartheta }\rangle }\,\Delta \left( e^{i\varvec{\vartheta }}\right) \, F\left( e^{i\varvec{\vartheta }};x,y\right) \,\mathrm {d}\varvec{\vartheta }; \end{aligned}$$

(86)

here $e^{i\varvec{\vartheta }}=\left( e^{\imath \,\vartheta _1},\,e^{\imath \,\vartheta _2}\right)$.

For $\delta >0$, let us define

$$\begin{aligned} V_{\delta }:=\left\{ (x,y)\in X\,:\,{\mathrm {dist}}_X\big (x,G\cdot y)\ge \delta \right\} . \end{aligned}$$

(87)

Proposition 5.1

For any $\delta >0$, we have $\Pi _{k\varvec{\nu }}(x,y)=O\left( k^{-\infty }\right)$ uniformly on $V_\delta$.

Proof of Proposition 5.1

By (44), the singular support of $\Pi$ is the diagonal in $X\times X$. Therefore,

$$\begin{aligned} \beta :\big ((x,y),\,gT,\,t\big )\in V_\delta \times G/T\times T\mapsto \Pi \,\big (x(g,t),y\big )\in \mathbb {C} \end{aligned}$$

(88)

is $\mathcal {C}^\infty$. The same then holds of $\big ((x,y),t\big )\in V_{\delta }\times T\mapsto \Delta (t)\,F(t;x,y)$. Hence, its Fourier transform (86) is rapidly decreasing for $k\rightarrow +\infty$. $\square$

We are thus reduced to assuming that ${\mathrm {dist}}_X\big (x,G\cdot y)< \delta$ for some fixed and arbitrarily small $\delta >0$. Let $\varrho \in \mathcal {C}_0^\infty (\mathbb {R})$ be $\equiv 1$ on $[-1,1]$ and $\equiv 0$ on $\mathbb {R}{\setminus } (-2,2)$. We can write

$$\begin{aligned} \Pi _{\varvec{\nu }}(x,y)=\Pi _{\varvec{\nu }}(x,y)_1+\Pi _{\varvec{\nu }}(x,y)_2, \end{aligned}$$

where the two summands on the right are defined by setting

$$\begin{aligned} \Pi _{\varvec{\nu }}(x,y)_j:= & {} d_{\varvec{\nu }}\,\int _T t^{-\varvec{\nu }}\,\Delta (t)\,F(t;x,y)_j\,\mathrm {d}V_T(t), \end{aligned}$$

(89)

and $F(t;x,y)_1$ is defined as in (79), but with the integrand multiplied by $\varrho \left( \delta ^{-1}\,{\mathrm {dist}}_X\big (x(g,\varvec{\vartheta }),y\big )\right)$; similarly, $F(t;x,y)_2$ is defined as in (79), but with the integrand multiplied by $1-\varrho \left( \delta ^{-1}\,{\mathrm {dist}}_X\big (x(g,\varvec{\vartheta }),y\big )\right)$.

Lemma 5.2

$\Pi _{k\varvec{\nu }}(x,y)_2=O\left( k^{-\infty }\right)$ for $k\rightarrow +\infty$.

Proof of Lemma 5.2

On the support of the integrand in $\Pi _{k\varvec{\nu }}(x,y)_2$, we have ${\mathrm {dist}}_X\big (x(g,t),y\big )\ge \delta$. We can then apply with minor changes the argument in the proof of Proposition 5.1. $\square$

On the support of the integrand in $\Pi _{k\varvec{\nu }}(x,y)_1$, ${\mathrm {dist}}_X\big (x(g,t),y\big )\le 2\,\delta$; therefore, perhaps after discarding a smoothing term contributing negligibly to the asymptotics, we can apply (80). With some passages, we obtain in place of (86):

$$\begin{aligned}&{\Pi _{k\varvec{\nu }}(x,y)\sim \Pi _{k\varvec{\nu }}(x,y)_1}\nonumber \\&\quad \sim \frac{k^2\,(\nu _1-\nu _2)}{(2\pi )^2}\, \int _{-\pi }^\pi \,\int _{-\pi }^\pi \,\int _{G/T}\,\int _0^{+\infty } e^{\imath \,k\,\Psi _{x,y}}\,\mathcal {A}_{x,y}\,\mathrm {d}u\, \mathrm {d}V_{G/T}(gT)\,\mathrm {d}\varvec{\vartheta }; \end{aligned}$$

(90)

we have applied the rescaling $u\mapsto k\,u$ to the parameter in (80), and set

$$\begin{aligned} \Psi _{x,y}= & {} \Psi _{x,y}(u,\varvec{\vartheta },gT):= u\,\psi \left( \widetilde{\mu }_{g\,e^{-\imath \,\varvec{\vartheta }}\,g^{-1}}(x),y\right) -\langle \varvec{\nu },\varvec{\vartheta }\rangle , \end{aligned}$$

(91)

$$\begin{aligned} \mathcal {A}_{x,y}= & {} \mathcal {A}_{x,y}(u,\varvec{\vartheta },gT):= \Delta \left( e^{i\varvec{\vartheta }}\right) \,s'\left( \widetilde{\mu }_{g\,e^{-\imath \,\varvec{\vartheta }}\,g^{-1}}(x),y,k\,u\right) , \end{aligned}$$

(92)

with

$$\begin{aligned} s'\left( \widetilde{\mu }_{g\,e^{-\imath \,\varvec{\vartheta }}\,g^{-1}}(x),y,k\,u\right):= & {} s\left( \widetilde{\mu }_{g\,e^{-\imath \,\varvec{\vartheta }}\,g^{-1}}(x),y,k\,u\right) \nonumber \\&\cdot \varrho \left( \delta ^{-1}\,{\mathrm {dist}}_X\left( \widetilde{\mu }_{g\,t^{-1}\,g^{-1}}(x),y\right) \right) . \end{aligned}$$

(93)

Lemma 5.3

Only a rapidly decreasing contribution to the asymptotics is lost, if in (90) integration in $\mathrm {d}u$ is restricted to an interval of the form (1 / D, D) for some $D\gg 0$.

Proof of Lemma 5.3

Suppose that $x,y\in X$, $\left( g_0\,T,e^{\imath \varvec{\vartheta }_0}\right) \in (G/T)\times T$ and

$$\begin{aligned} {\mathrm {dist}}_X\left( x(g_0,\varvec{\vartheta }_0),y\right) < \delta . \end{aligned}$$

(94)

In view of (81), in any system of local coordinates we have

$$\begin{aligned} \mathrm {d}_{\left( x(g_0,\varvec{\vartheta }_0),y\right) }\psi = (\alpha _{x(g_0,\varvec{\vartheta }_0)},-\alpha _{y})+O(\delta ). \end{aligned}$$

(95)

Let $\mathrm {d}^{(\varvec{\vartheta })}$ denote the differential with respect to the variable $\varvec{\vartheta }$. If $\imath \,\varvec{\eta }\in \mathfrak {t}$, we obtain with $m_x:=\pi (x)$:

$$\begin{aligned}&{\left. \frac{d}{\mathrm {d}\tau }\, x(g_0,\varvec{\vartheta }_0+\tau \,\varvec{\eta })\right| _{\tau =0}}\nonumber \\&\quad =-{\mathrm {Ad}}_{g_0}(\imath \,\varvec{\eta })_X\big (x(g_0,\varvec{\vartheta }_0)\big )\nonumber \\&\quad =-{\mathrm {Ad}}_{g_0}(\imath \,\varvec{\eta })_M\big (m_x(g_0,\varvec{\vartheta }_0)\big )^\sharp + \Big \langle \Phi _G\big (m_x(g_0,\varvec{\vartheta }_0)\big ),{\mathrm {Ad}}_{g_0}(\imath \,\varvec{\eta })\Big \rangle \, \partial _\theta . \end{aligned}$$

(96)

On the other hand, as $\Phi _G$ is G-equivariant we get

$$\begin{aligned}&{\langle \Phi _G\big (m_x(g_0,\varvec{\vartheta }_0)\big ),{\mathrm {Ad}}_{g_0}(\imath \,\varvec{\eta })\rangle = \left\langle {\mathrm {Ad}}_{g_0^{-1}}\Big (\Phi _G\big (m_x(g_0,\varvec{\vartheta }_0)\big )\Big ),\imath \,\varvec{\eta }\right\rangle } \nonumber \\&\quad =\left\langle \Phi _G\Big (\mu _{g_0^{-1}}\big (m_x(g_0,\varvec{\vartheta }_0)\big )\Big ),\imath \,\varvec{\eta }\right\rangle =\left\langle \Phi _T\Big (\mu _{g_0^{-1}}\big (m_x(g_0,\varvec{\vartheta }_0)\big )\Big ),\imath \,\varvec{\eta }\right\rangle . \end{aligned}$$

(97)

Now, (95), (96) and (97) imply

$$\begin{aligned}&{ \left. \frac{d}{\mathrm {d}\tau }\,\psi \Big (x(g_0,\varvec{\vartheta }_0+\tau \,\varvec{\eta }),y\Big ) \right| _{\tau =0}}\nonumber \\&\quad =-\mathrm {d}_{\left( x(g_0,\varvec{\vartheta }_0),y\right) }\psi \Big ({\mathrm {Ad}}_{g_0}(\imath \,\varvec{\eta })_X\big (x(g_0,\varvec{\vartheta }_0)\big ),0\Big )\nonumber \\&\quad =-\alpha _{x(g_0,\varvec{\vartheta }_0)} \Big ({\mathrm {Ad}}_{g_0}(\imath \,\varvec{\eta })_X\big (x(g_0,\varvec{\vartheta }_0)\big )\Big )+ \langle O(\delta ),\varvec{\eta }\rangle \nonumber \\&\quad =\left\langle \frac{1}{\imath }\,\Phi _T\Big (\mu _{g_0^{-1}}\big (m_x(g_0,\varvec{\vartheta }_0)\big )\Big ) + O(\delta ),\varvec{\eta }\right\rangle . \end{aligned}$$

(98)

Let $\mathrm {d}^{(\varvec{\vartheta })}$ denote the differential with respect to $\varvec{\vartheta }$. Recalling (91), we obtain

$$\begin{aligned} \mathrm {d}^{(\varvec{\vartheta })}_{(u,g_0T,\varvec{\vartheta }_0)}\Psi _{x,y}= & {} \frac{u}{\imath }\,\Phi _T\Big (\mu _{g_0^{-1}}(m_x)\Big )-\varvec{\nu }+ O(\delta ). \end{aligned}$$

(99)

By assumption, $\mathbf {0}\not \in \Phi _T(M)$. Let us set

$$\begin{aligned} a_T:=\min \Vert \Phi _T\Vert ,\,\,\,\,A_T:=\max \Vert \Phi _T\Vert . \end{aligned}$$

Then, $A_T\ge a_T>0$, and (99) implies

$$\begin{aligned}&{\left\| \mathrm {d}^{(\varvec{\vartheta })}_{(u,g_0T,\varvec{\vartheta }_0)}\Psi _{x,y}\right\| } \nonumber \\&\quad \ge \max \big \{ u\,a_T-\Vert \varvec{\nu }\Vert +O(\delta ),\,\Vert \varvec{\nu }\Vert -u\,A_T+O(\delta )\big \}. \end{aligned}$$

(100)

Thus, if $D\gg 0$ and $u\ge D$, we have

$$\begin{aligned} \left\| \mathrm {d}^{(\varvec{\vartheta })}_{(u,g_0T,\varvec{\vartheta }_0)}\Psi _{x,y}\right\| \ge \frac{a_T}{2}\,u+1, \end{aligned}$$

(101)

while for $0<u<1/D$

$$\begin{aligned} \left\| \mathrm {d}^{(\varvec{\vartheta })}_{(u,g_0T,\varvec{\vartheta }_0)}\Psi _{x,y}\right\| \ge \frac{\Vert \varvec{\nu }\Vert }{2}. \end{aligned}$$

(102)

The Lemma then follows from (101) and (102) by a standard iterated integration by parts in $\varvec{\vartheta }$ (in view of the compactness of T). $\square$

Suppose that $\rho \in \mathcal {C}^\infty _0\big ((0,+\infty )\big )$ is $\equiv 1$ on (1 / D, D) and is supported on (1 / (2D), 2D). By Lemma 5.3, the asymptotics of (90) are unaltered, if the integrand is multiplied by $\rho (u)$. Thus, we obtain

$$\begin{aligned}&{\Pi _{k\varvec{\nu }}(x,y)}\nonumber \\&\quad \sim \frac{k^2\,(\nu _1-\nu _2)}{(2\pi )^2}\, \int _{-\pi }^\pi \,\int _{-\pi }^\pi \,\int _{G/T}\,\int _{1/(2D)}^{2D} e^{\imath \,k\,\Psi _{x,y}}\,\mathcal {A}'_{x,y}\,\mathrm {d}u\, \mathrm {d}V_{G/T}(gT)\,\mathrm {d}\varvec{\vartheta }; \end{aligned}$$

(103)

with $\mathcal {A}_{x,y}$ as in (92), we have set

$$\begin{aligned} \mathcal {A}'_{x,y}(u,\varvec{\vartheta },gT):=\rho (u)\,\mathcal {A}_{x,y}(u,\varvec{\vartheta },gT). \end{aligned}$$

(104)

Integration in $\mathrm {d}u$ is now over a compact interval

Let $\mathfrak {I}(z)$ denote the imaginary part of $z\in \mathbb {C}$. In view of Corollary 1.3 of [4], there exists a fixed constant D, depending only on X, such that

$$\begin{aligned} \mathfrak {I}\Big (\psi \left( x',x''\right) \Big ) \ge D\,{\mathrm {dist}}_X\left( x',x''\right) ^2\quad (x',x''\in X). \end{aligned}$$

(105)

Proposition 5.2

Uniformly for

$$\begin{aligned} {\mathrm {dist}}_X(x,G\cdot y)\ge C\,k^{\epsilon -1/2}, \end{aligned}$$

(106)

we have $\Pi _{k\varvec{\nu }}(x,y)=O\left( k^{-\infty }\right)$.

Proof of Proposition 5.2

In the range (106), we have

$$\begin{aligned} {\mathrm {dist}}_X\big (x(g,\varvec{\vartheta }),y\big )\ge C\,k^{\epsilon -1/2} \end{aligned}$$

(107)

for every $g\,T\in G/T$ and $e^{\imath \,\varvec{\vartheta }}\in T$. In view of (91) and (105),

$$\begin{aligned} \left| \partial _u\Psi _{x,y}(u,\varvec{\vartheta },gT)\right|= & {} \left| \psi \left( x(g,\varvec{\vartheta }),y\right) \right| \ge \mathfrak {I}\left( \psi \big (x(g,\varvec{\vartheta }),y\big )\right) \nonumber \\\ge & {} D\,{\mathrm {dist}}_X\big (x(g,\varvec{\vartheta }),y\big )^2\ge D\,C^2\,k^{2\epsilon -1}. \end{aligned}$$

(108)

Let us use the identity

$$\begin{aligned} -\frac{\imath }{k}\,\psi \big (x(g,\varvec{\vartheta }),y\big )^{-1}\, \frac{\mathrm {d}}{{\mathrm {d}}u} e^{\imath \,k\,\Psi _{x,y}}=e^{\imath \,k\,\Psi _{x,y}} \end{aligned}$$

(109)

to iteratively integrate by parts in $\mathrm {d}u$ in (103); then by (108) at each step we introduce a factor $O\left( k^{-2\,\epsilon }\right)$. The claim follows. $\square$

To complete the proof of Theorem 1.3, we need to establish the following.

Proposition 5.3

Uniformly for

$$\begin{aligned} {\mathrm {dist}}_X\left( x,G\cdot X^T_{\varvec{\nu }}\right) \ge C\,k^{\epsilon -1/2}, \end{aligned}$$

(110)

we have $\Pi _{k\varvec{\nu }}(x,x)=O\left( k^{-\infty }\right)$ as $k\rightarrow +\infty$.

Remark 5.1

Let ${\mathrm {dist}}_M$ denote the distance function on M; if $m=\pi (x)$, then ${\mathrm {dist}}_X\left( x,G\cdot X^T_{\varvec{\nu }}\right)$$={\mathrm {dist}}_M\left( m,G\cdot M^T_{\varvec{\nu }}\right)$.

Proof of Proposition 5.3

Since G acts on M as a group of Riemannian isometries, (110) means that for any $g\in G$ we have

$$\begin{aligned} C\,k^{\epsilon -1/2}\le {\mathrm {dist}}_M\left( m,\mu _{g}\left( M^T_{\varvec{\nu }}\right) \right) = {\mathrm {dist}}_M\left( \mu _{g^{-1}}\left( m\right) ,M^T_{\varvec{\nu }}\right) . \end{aligned}$$

(111)

On the other hand, as $-\imath \,\Phi ^T$ is transverse to $\mathbb {R}_+\,\varvec{\nu }$, by the discussion in §2.1.3 of [28] there is a constant $b_{\varvec{\nu }}>0$ such that every $u\in [1/(2D),2D]$ we have

$$\begin{aligned} \left\| -\imath \,u\,\Phi ^T\left( \mu _{g^{-1}}\left( m\right) \right) -\varvec{\nu }\right\| \ge b_{\varvec{\nu }}\,C\,k^{\epsilon -1/2}. \end{aligned}$$

(112)

Let us consider (103) with $x=y$:

$$\begin{aligned}&{\Pi _{k\varvec{\nu }}(x,x)}\nonumber \\&\quad \sim \frac{k^2\,(\nu _1-\nu _2)}{(2\pi )^2}\, \int _{-\pi }^\pi \,\int _{-\pi }^\pi \,\int _{G/T}\,\int _{1/(2D)}^{2D} e^{\imath \,k\,\Psi _{x,x}}\,\mathcal {A}'_{x,x}\,\mathrm {d}u\, \mathrm {d}V_{G/T}(gT)\,\mathrm {d}\varvec{\vartheta }. \end{aligned}$$

(113)

Let us choose $\epsilon '\in (0,\epsilon )$ and multiply the integrand in (113) by the identity

$$\begin{aligned} \varrho \left( k^{1/2-\epsilon '}\,{\mathrm {dist}}_X\big (x(g,\varvec{\vartheta }),x\big )\right) + \left[ 1-\varrho \left( k^{1/2-\epsilon '}\,{\mathrm {dist}}_X\big (x(g,\varvec{\vartheta }),x\big )\right) \right] =1. \end{aligned}$$

Here $\varrho$ is as in the discussion preceding Lemma 5.2. We obtain a further splitting

$$\begin{aligned} \Pi _{k\varvec{\nu }}(x,x)\sim \Pi _{k\varvec{\nu }}(x,x)_a+\Pi _{k\varvec{\nu }}(x,x)_b, \end{aligned}$$

(114)

where $\Pi _{k\varvec{\nu }}(x,x)_a$ is given by (113) with the amplitude $\mathcal {A}_{x,x}'$ replaced by

$$\begin{aligned} \mathcal {B}_{x,x}':=\varrho \left( k^{1/2-\epsilon '}\, {\mathrm {dist}}_X\big (x(g,\varvec{\vartheta }),x\big )\right) \,\mathcal {A}_{x,x}'; \end{aligned}$$

(115)

similarly, $\Pi _{k\varvec{\nu }}(x,x)_b$ is given by (113) with the amplitude $\mathcal {A}_{x,x}'$ replaced by

$$\begin{aligned} \mathcal {B}_{x,x}'':=\left[ 1-\varrho \left( k^{1/2-\epsilon '}\,{\mathrm {dist}}_X\big (x(g,\varvec{\vartheta }),x\big )\right) \right] \, \mathcal {A}_{x,x}'. \end{aligned}$$

Lemma 5.4

$\Pi _{k\varvec{\nu }}(x,x)_b=O\left( k^{-\infty }\right)$ as $k\rightarrow +\infty$.

Proof of Lemma 5.4

On the support of $\mathcal {B}_{x,x}''$, we have

$$\begin{aligned} {\mathrm {dist}}_X\big (x(g,\varvec{\vartheta }),x\big )\ge k^{\epsilon '-1/2}. \end{aligned}$$

(116)

Thus, we may again appeal to (109) and iteratively integrate by parts in $\mathrm {d}u$, introducing at each step a factor $O\left( k^{-1}\,k^{1-2\epsilon '}\right) =O\left( k^{-2\epsilon '}\right)$. $\square$

Thus, the proof of the Theorem will be complete once we establish the following.

Lemma 5.5

$\Pi _{k\varvec{\nu }}(x,x)_a=O\left( k^{-\infty }\right)$ as $k\rightarrow +\infty$.

Before attacking the proof of Lemma 5.5, let us prove the following.

Lemma 5.6

If (110) holds, then for any $u\in [1/(2D),2D]$ and $k\gg 0$

$$\begin{aligned} \left\| \mathrm {d}^{(\varvec{\vartheta })}_{(u,gT,\varvec{\vartheta })}\Psi _{x,x}\right\| \ge \frac{b_{\varvec{\nu }}}{2}\,C\,k^{\epsilon -1/2} \end{aligned}$$

(117)

on the support of $\mathcal {B}_{x,x}'$.

Proof of Lemma 5.6

On the support of $\mathcal {B}_{x,x}'$, we have

$$\begin{aligned} {\mathrm {dist}}_X\big (x(g,\varvec{\vartheta }),x\big )\le 2\,k^{\epsilon '-1/2}. \end{aligned}$$

(118)

Thus, instead of (95) we have

$$\begin{aligned} \mathrm {d}_{(x(g,\varvec{\vartheta }),x)}\psi = (\alpha _{x(g,\varvec{\vartheta })},-\alpha _{x})+O\left( k^{\epsilon '-1/2}\right) . \end{aligned}$$

(119)

Therefore, in place of (99) on the support of $\mathcal {B}_{x,x}'$ we have

$$\begin{aligned} \mathrm {d}^{(\varvec{\vartheta })}_{(u,gT,\varvec{\vartheta })}\Psi _{x,x}= & {} \frac{u}{\imath }\,\Phi _T\big (\mu _{g^{-1}}(m_x)\big )-\varvec{\nu }+ O\left( k^{\epsilon '-1/2}\right) . \end{aligned}$$

(120)

Thus, in view of (112) the claim follows since $0<\epsilon '<\epsilon$. $\square$

Given Lemma 5.6, we can prove Lemma 5.5 essentially by iteratively integrating by parts in $\mathrm {d}\varvec{\vartheta }$.

Proof of Lemma 5.5

Since $\widetilde{\mu }$ is free on $X^G_{\mathcal {O}}$, it is also free on a small tubular neighborhood $X'$ of $X^G_{\mathcal {O}}$ in X. Without loss, we may restrict our analysis to $X'$ in view of Theorem 1.1.

On the support of $\mathcal {B}_{x,x}'$, therefore, $e^{\imath \,\varvec{\vartheta }}\in T$ varies in a small neighborhood of $I_2$. Let $f:T\rightarrow [0,+\infty )$ be a bump function compactly supported in a small neighborhood $U\subset T$ of $I_2$ (identified with (1, 1)), and identically $=1$ near $I_2$. Then, we obtain

$$\begin{aligned} \Pi _{k\varvec{\nu }}(x,x)_a\sim & {} \left( \frac{k}{2\pi }\right) ^2\,(\nu _1-\nu _2)\nonumber \\&\cdot \int _{U}\,\int _{G/T}\,\int _{1/(2D)}^{2D} e^{\imath \,k\,\Psi _{x,x}}\,f(t)\,\mathcal {B}_{x,x}'\,\mathrm {d}u\, \mathrm {d}V_{G/T}(gT)\,\mathrm {d}\varvec{\vartheta }. \end{aligned}$$

(121)

Let us introduce the differential operator

$$\begin{aligned} P=\sum _{h=1}^2\frac{\partial _{\vartheta _h}\Psi _{x,x}}{\left( \partial _{\vartheta _1}\Psi _{x,x}\right) ^2 +\left( \partial _{\vartheta _2}\Psi _{x,x}\right) ^2}\,\frac{\partial }{\partial \vartheta _h}, \end{aligned}$$

(122)

so that

$$\begin{aligned} \frac{1}{\imath \,k}\,P\left( e^{ik\Psi _{x,x}}\right) =e^{ik\Psi _{x,x}}. \end{aligned}$$

Thus,

$$\begin{aligned}&{\int _{U}e^{\imath \,k\,\Psi _{x,x}}\,f(t)\,\mathcal {B}_{x,x}'\,\mathrm {d}\varvec{\vartheta }}\end{aligned}$$

(123)

$$\begin{aligned}&\quad =\frac{1}{\imath \,k}\,\sum _{h=1}^2\,\int _{U}\frac{\partial _{\vartheta _h}\Psi _{x,x}}{\left( \partial _{\vartheta _1}\Psi _{x,x}\right) ^2 +\left( \partial _{\vartheta _2}\Psi _{x,x}\right) ^2}\, \frac{\partial }{\partial \vartheta _h}\left[ e^{\imath \,k\,\Psi _{x,x}}\right] \, f\left( e^{\imath \varvec{\vartheta }}\right) \,\mathcal {B}_{x,x}'\,\mathrm {d}\varvec{\vartheta } \nonumber \\&\quad =\frac{\imath }{k}\,\sum _{h=1}^2\,\int _{U}e^{\imath \,k\,\Psi _{x,x}}\,\frac{\partial }{\partial \vartheta _h}\left[ \frac{\partial _{\vartheta _h}\Psi _{x,x}}{\left( \partial _{\vartheta _1}\Psi _{x,x}\right) ^2 +\left( \partial _{\vartheta _2}\Psi _{x,x}\right) ^2}\, f\left( e^{\imath \varvec{\vartheta }}\right) \,\mathcal {B}_{x,x}'\,\right] \,\mathrm {d}\varvec{\vartheta } \nonumber \\&\quad =\frac{\imath }{k}\,\int _{U}e^{\imath \,k\,\Psi _{x,x}}\,P^t\big (f(t)\,\mathcal {B}_{x,x}'\big )\,\mathrm {d}\varvec{\vartheta }, \end{aligned}$$

(124)

where

$$\begin{aligned} P^t(\gamma ):=\sum _{h=1}^2\,\frac{\partial }{\partial \vartheta _h}\left[ \frac{\partial _{\vartheta _h}\Psi _{x,x}}{\left( \partial _{\vartheta _1}\Psi _{x,x}\right) ^2 +\left( \partial _{\vartheta _2}\Psi _{x,x}\right) ^2}\, \gamma \,\right] . \end{aligned}$$

(125)

Iterating, for any $r\in \mathbb {N}$ we have

$$\begin{aligned} \int _{U}e^{\imath \,k\,\Psi _{x,x}}\,f(t)\,\mathcal {B}_{x,x}'\,\mathrm {d}\varvec{\vartheta }= & {} \frac{\imath ^r}{k^r}\,\int _{U}e^{\imath \,k\,\Psi _{x,x}} \,\left( P^t\right) ^r\big (f(t)\,\mathcal {B}_{x,x}'\big )\,\mathrm {d}\varvec{\vartheta }. \end{aligned}$$

(126)

Let us consider the function

$$\begin{aligned} \mathcal {D}:\varvec{\vartheta }\mapsto {\mathrm {dist}}_X\big (x(g,\varvec{\vartheta }),x\big )= {\mathrm {dist}}_X\left( \widetilde{\mu }_{e^{-\imath \,\varvec{\vartheta }}}\circ \widetilde{\mu }_{g^{-1}}(x),\mu _{g^{-1}}(x)\right) . \end{aligned}$$

(127)

We have the following.

Lemma 5.7

For $\varvec{\vartheta }\sim \mathbf {0}$, we have

$$\begin{aligned} {\mathrm {dist}}_X\big (x(g,\varvec{\vartheta }),x\big )= F_1(g\,T; \varvec{\vartheta })+F_2(g\,T; \varvec{\vartheta })+\cdots , \end{aligned}$$

where $F_j(g\,T; \varvec{\vartheta })$ is homogeneous of degree j in $\varvec{\vartheta }$, and $\mathcal {C}^\infty$ for $\varvec{\vartheta }\ne \mathbf {0}$. In addition, $F_1(g\,T; \varvec{\vartheta })=\Vert {\mathrm {Ad}}_g(\varvec{\vartheta })_X(x)\Vert = \left\| \varvec{\vartheta }_X\left( \widetilde{\mu }_{g^{-1}}(x)\right) \right\|$.

For any $c\in \mathbb {N}$ let $\mathcal {D}^{(c)}$ denote a generic iterated derivative of the form

$$\begin{aligned} \frac{\partial ^c\,\mathcal {D}}{\partial \vartheta _{i_1}\,\cdots \partial \vartheta _{i_c}}; \end{aligned}$$

clearly $\mathcal {D}^{(c)}$ is not uniquely determined by c. By Lemma 5.7, as $k\rightarrow +\infty$

$$\begin{aligned} \mathcal {D}^{(c)}=O\left( k^{(c-1)(1/2-\epsilon ')}\right) \end{aligned}$$

where $\varrho \left( k^{1/2-\epsilon '}\, \mathcal {D}\right) \not \equiv 1$. For any multi-index $\mathbf {C}=(c_1,\ldots ,c_s)$, let us denote by $\mathcal {D}^{(\mathbf {C})}$ a generic product of the form $\mathcal {D}^{(c_1)}\cdots \,\mathcal {D}^{(c_s)}$; then,

$$\begin{aligned} \mathcal {D}^{(\mathbf {C})}=O\left( k^{(1/2-\epsilon ')\sum _j(c_j-1)}\right) . \end{aligned}$$

(128)

Lemma 5.8

For any $r\in \mathbb {N}$, $\left( P^t\right) ^r\big (f(t)\,\mathcal {B}_{x,x}'\big )$ is a linear combination of summands of the form

$$\begin{aligned} \varrho ^{ (b)}\left( k^{1/2-\epsilon '}\, D_k(\varvec{\vartheta })\right) \, \frac{P_{a_1}(\Psi _{x,x},\partial \Psi _{x,x})}{\left[ \left( \partial _{\vartheta _1}\Psi _{x,x}\right) ^2 +\left( \partial _{\vartheta _2}\Psi _{x,x}\right) ^2\right] ^{a_2}}\,k^{b(1/2-\epsilon ')}\,\mathcal {D}^{(\mathbf {C})}, \end{aligned}$$

(129)

times omitted factors bounded in k depending on $f_j$ and its derivatives, where:

• 1. $P_{a_1}$ denotes a generic differential polynomial in $\Psi _{x,x}$, homogeneous of degree $a_1$ in the first derivatives $\partial \Psi _{x,x}$;

• 2. if $a:=2a_2-a_1$, then $a,b,\mathbf {C}$ are subject to the bound

  $$\begin{aligned} a+b+\sum _{j=1}^r(c_j-1)\le 2\,r \end{aligned}$$

  (130)

  (the sum is over the $c_j>0$);

• 3. $\mathbf {C}$ is not zero if and only if $b>0$.

Here $\varrho ^{ (l)}$ is the lth derivative of the one-variable real function $\varrho$.

Proof of Lemma 5.8

Let us set $F:=f_j\left( e^{\imath \varvec{\vartheta }}\right) \,\mathcal {B}_{x,x}'$. For $r=1$, we have

$$\begin{aligned}&{\frac{\partial }{\partial \vartheta _h}\left[ \frac{\partial _{\vartheta _h}\Psi _{x,x}}{\left( \partial _{\vartheta _1}\Psi _{x,x}\right) ^2 +\left( \partial _{\vartheta _2}\Psi _{x,x}\right) ^2}\, F\,\right] }\nonumber \\&\qquad =\frac{\partial _{\vartheta _h}\Psi _{x,x}}{\left( \partial _{\vartheta _1}\Psi _{x,x}\right) ^2 +\left( \partial _{\vartheta _2}\Psi _{x,x}\right) ^2}\,\frac{\partial \, F}{\partial \vartheta _h}+ F\,\frac{\partial }{\partial \vartheta _h}\left[ \frac{\partial _{\vartheta _h}\Psi _{x,x}}{\left( \partial _{\vartheta _1}\Psi _{x,x}\right) ^2 +\left( \partial _{\vartheta _2}\Psi _{x,x}\right) ^2}\right] . \end{aligned}$$

(131)

We have

$$\begin{aligned}&{\frac{\partial _{\vartheta _h}\Psi _{x,x}}{\left( \partial _{\vartheta _1}\Psi _{x,x}\right) ^2 +\left( \partial _{\vartheta _2}\Psi _{x,x}\right) ^2}\,\frac{\partial \, F}{\partial \vartheta _h}}\nonumber \\&\qquad =\frac{\partial _{\vartheta _h}\Psi _{x,x}}{\left( \partial _{\vartheta _1}\Psi _{x,x}\right) ^2 +\left( \partial _{\vartheta _2}\Psi _{x,x}\right) ^2}\,\left[ \frac{\partial \, f_j}{\partial \vartheta _h}\,\mathcal {B}_{x,x}'+ \frac{\partial \,\mathcal {B}_{x,x}'}{\partial \vartheta _h}\, f_j\right] . \end{aligned}$$

(132)

Thus, in view of (115), the first summand on the right-hand side of (131) splits as a linear combination of terms as in the statement, with $a_1=a_2=1$, b and $\mathbf {C}$ both zero, or $a_1=a_2=1$, $b=1$, $\mathbf {C}=(1)$. Hence, $a+b+\sum _j(c_j-1)=2$ in either case. On the other hand, the second summand on the right-hand side of (131) satisfies

$$\begin{aligned}&{F\,\frac{\partial }{\partial \vartheta _h}\left[ \frac{\partial _{\vartheta _h}\Psi _{x,x}}{\left( \partial _{\vartheta _1}\Psi _{x,x}\right) ^2 +\left( \partial _{\vartheta _2}\Psi _{x,x}\right) ^2}\right] =\frac{F}{\left[ \left( \partial _{\vartheta _1}\Psi _{x,x}\right) ^2 +\left( \partial _{\vartheta _2}\Psi _{x,x}\right) ^2\right] ^2}}\\&\qquad \cdot \left\{ \partial ^2_{\vartheta _h,\vartheta _h}\Psi _{x,x}\,\left[ \left( \partial _{\vartheta _1}\Psi _{x,x}\right) ^2 +\left( \partial _{\vartheta _2}\Psi _{x,x}\right) ^2 \right] -2\,\partial _{\vartheta _h}\Psi _{x,x}\, \sum _{a=1}^2\partial _{\vartheta _a}\Psi _{x,x}\,\partial ^2_{\vartheta _a\vartheta _h}\Psi _{x,x}\right\} . \end{aligned}$$

This is of the stated type with $a_1=a_2=2$, b and $\mathbf {C}$ both zero. Hence, $a=4-2=2$.

Passing to the inductive step, let us consider (125) with $\gamma$ given by (129), and assume that (130) is satisfied. Let us write $\varrho ^{ (l)}$ for the factor in front in (129). We obtain a linear combination of expressions of the form

$$\begin{aligned} \frac{\partial }{\partial \vartheta _h}\left[ \varrho ^{ (b)}\, \frac{P_{a_1+1}(\Psi _{x,x},\partial \Psi _{x,x})}{\left[ \left( \partial _{\vartheta _1}\Psi _{x,x}\right) ^2 +\left( \partial _{\vartheta _2}\Psi _{x,x}\right) ^2\right] ^{a_2+1}}\,k^{b(1/2-\epsilon ')}\,\mathcal {D}^{(\mathbf {C})}\right] . \end{aligned}$$

(133)

It is clear that (133) splits as a linear combination of summands of the following forms:

$$\begin{aligned} \varrho ^{ (b)}\,\frac{P_{a'}(\Psi _{x,x},\partial \Psi _{x,x})}{\left[ \left( \partial _{\vartheta _1}\Psi _{x,x}\right) ^2 +\left( \partial _{\vartheta _2}\Psi _{x,x}\right) ^2\right] ^{a_2+1}}\,k^{b(1/2-\epsilon ')}\,\mathcal {D}^{(\mathbf {C})}, \end{aligned}$$

(134)

with $a'\in \{a_1,a_1+1,a_1+2\}$;

$$\begin{aligned}&\varrho ^{ (b)}\,\frac{P_{a_1+2}(\Psi _{x,x},\partial \Psi _{x,x})}{\left[ \left( \partial _{\vartheta _1}\Psi _{x,x}\right) ^2 +\left( \partial _{\vartheta _2}\Psi _{x,x}\right) ^2\right] ^{a_2+2}}\,k^{b(1/2-\epsilon ')}\,\mathcal {D}^{(\mathbf {C})}; \end{aligned}$$

(135)

$$\begin{aligned}&\varrho ^{ (b+1)}\frac{P_{a_1+1}(\Psi _{x,x},\partial \Psi _{x,x})}{\left[ \left( \partial _{\vartheta _1}\Psi _{x,x}\right) ^2 +\left( \partial _{\vartheta _2}\Psi _{x,x}\right) ^2\right] ^{a_2+1}}\,k^{(b+1)(1/2-\epsilon ')}\,\mathcal {D}^{(\mathbf {C}')}, \end{aligned}$$

(136)

where $\mathbf {C}'$ is of the form $\mathbf {C}'=(1,\mathbf {C})$;

$$\begin{aligned} \varrho ^{ (b)}\,\frac{P_{a_1+1}(\Psi _{x,x},\partial \Psi _{x,x})}{\left[ \left( \partial _{\vartheta _1}\Psi _{x,x}\right) ^2 +\left( \partial _{\vartheta _2}\Psi _{x,x}\right) ^2\right] ^{a_2+1}}\,k^{b(1/2-\epsilon ')}\,\mathcal {D}^{(\mathbf {C}')}, \end{aligned}$$

(137)

where $\mathbf {C}'$ is obtained from $\mathbf {C}$ (if the latter is not zero) by replacing one of the $c_j$’s by $c_j+1$, and leaving all the others unchanged.

In all these cases, we obtain a term of the form (129), satisfying (130) with r replaced by $r+1$. This completes the proof of Lemma 5.8. $\square$

As $0<\epsilon '<\epsilon$, the general summand (129) is

$$\begin{aligned} O\left( k^{a(1/2-\epsilon )+[b+\sum _j(c_j-1)](1/2-\epsilon ')}\right)= & {} O\left( k^{[a+b+\sum _j(c_j-1)](1/2-\epsilon ')}\right) \\= & {} O\left( k^{2r(1/2-\epsilon ')}\right) = O\left( k^{r(1-2\epsilon ')}\right) . \end{aligned}$$

Making use of the latter estimate in (126), we obtain the following:

Corollary 5.1

For any $r\in \mathbb {N}$,

$$\begin{aligned} \int _{U_j}e^{\imath \,k\,\Psi _{x,x}}\,f(t)\,\mathcal {B}_{x,x}'\,\mathrm {d}\varvec{\vartheta }= & {} O\left( k^{-2r\,\epsilon '}\right) . \end{aligned}$$

(138)

The proof of Lemma 5.5 is thus complete. $\square$

Given (114), Proposition 5.3 follows from Lemmata 5.4 and 5.5. $\square$

Thus, the statement of Theorem 1.3 holds true when $x=y$. The general case follows from this and the Schwartz inequality

$$\begin{aligned} \big |\Pi _{k\varvec{\nu }}(x,y)\big |\le \sqrt{\Pi _{k\varvec{\nu }}(x,x)}\,\sqrt{\Pi _{k\varvec{\nu }}(y,y)}; \end{aligned}$$

in fact both factors on the right-hand side have at most polynomial growth in k by Lemma 5.1, and if say (110) holds, then the first one is rapidly decreasing. The proof of Theorem 1.3 is complete. $\square$

6 Proof of Theorems 1.4, 1.5 and 1.6

6.1 Preliminaries on local rescaled asymptotics

In the proof of Theorems 1.4, 1.5 and 1.6, we are interested in the asymptotics of $\Pi _{k\varvec{\nu }}(x',x'')$ when $(x',x'')$ approaches the diagonal of $X^G_{\mathcal {O}}$ in $X\times X$ along appropriate directions and at a suitable pace.

In Theorems 1.4 and 1.6, we consider $x'=x''$ in a shrinking ‘one-sided’  neighborhood of $X^G_{\mathcal {O}}$. In Theorem 1.5, we shall assume that $(x',x'')$ approaches the diagonal in $X^G_{\mathcal {O}}$ along ‘horizontal’  directions orthogonal to the orbits. We shall treat the former case in detail and then briefly discuss the necessary changes for the latter.

Suppose $x\in X^G_{\mathcal {O}}$ and let $m=\pi (x)$. Let us choose a system of HLC centered at x, and let us consider the collection of points

$$\begin{aligned} x_{\tau ,k}:=x+\frac{\tau }{\sqrt{k}}\,\Upsilon _{\varvec{\nu }}(m), \end{aligned}$$

(139)

where $k=1,2,\ldots$, and $|\tau |\le C\,k^\epsilon$ for some fixed $C>0$ and $\epsilon \in (0,1/6)$. The sign of $\tau$ is chosen so that $\tau \, \Upsilon _{\varvec{\nu }}(m)$ is either zero or outer oriented. Thus, $\tau \, (\nu _1+\nu _2)\ge 0$. We shall provide an integral expression for the asymptotics of $\Pi _{k\varvec{\nu }}(x_{\tau ,k},x_{\tau ,k})$ when $k\rightarrow +\infty$.

Applying as before the Weyl integration and character formulae, inserting the microlocal description of $\Pi$ as an FIO, and making use of the rescaling $u\mapsto k\,u$, $\varvec{\vartheta }\mapsto \varvec{\vartheta }/\sqrt{k}$, we obtain that, as $k\rightarrow +\infty$,

$$\begin{aligned}&{ \Pi _{k\varvec{\nu }}(x_{\tau ,k},x_{\tau ,k}) }\nonumber \\&\quad \sim \dfrac{k\,(\nu _1-\nu _2)}{(2\pi )^2}\, \int _{G/T}\,\mathrm {d}V_{G/T}(gT)\,\int _{-\infty }^{\infty }\mathrm {d}\vartheta _1\,\int _{-\infty }^{\infty }\mathrm {d}\vartheta _2\, \int _0^{+\infty }\,\mathrm {d}u\nonumber \\&\qquad \left[ e^{\imath \,k\, \left[ u\,\psi \left( \widetilde{\mu }_{g\,e^{-\imath \varvec{\vartheta }/\sqrt{k}}\,g^{-1}}(x_{\tau ,k}),x_{\tau ,k}\right) -\langle \varvec{\vartheta },\varvec{\nu }\rangle /\sqrt{k} \right] } \right. \nonumber \\&\quad \qquad \left. \cdot \Delta \left( e^{\imath \varvec{\vartheta }/\sqrt{k}}\right) \, s\left( \widetilde{\mu }_{g\,e^{-\imath \varvec{\vartheta }/\sqrt{k}}\,g^{-1}}(x_{\tau ,k}),x_{\tau ,k},k\,u\right) \right] . \end{aligned}$$

(140)

Integration in $\varvec{\vartheta }=(\vartheta _1, \vartheta _2)$ is over a ball centered at the origin and radius $O\left( k^\epsilon \right)$ in $\mathbb {R}^2$. A cut-off function of the form $\varrho \left( k^{-\epsilon } \, \varvec{\vartheta } \right)$ is implicitly incorporated into the amplitude.

In order to express the previous phase more explicitly, we need the following Definition.

Definition 6.1

Let us define $\varvec{\rho }=\varvec{\rho }_m:G/T\rightarrow \mathfrak {t}\cong \mathbb {R}^2$, $g\,T\mapsto \varvec{\rho }_{g\,T}$, by requiring

$$\begin{aligned} \langle \varvec{\rho }_{g\,T},\varvec{\vartheta }\rangle = \omega _m\Big ({\mathrm {Ad}}_g (\imath \, D_{\varvec{\vartheta }})_M(m), \Upsilon _{\varvec{\nu }}(m)\Big ) \quad ({\vartheta }\in \mathbb {R}^2). \end{aligned}$$

Next, let the symmetric and positive definite matrix $E( g\,T )=E_x( g\,T )$ be defined by the equality

$$\begin{aligned} \varvec{\vartheta }^t \,E( g\,T )\, \varvec{\vartheta } = \big \Vert {\mathrm {Ad}}_g (\imath \, D_{\varvec{\vartheta }}) _X(x) \big \Vert _x ^2 \quad ({\vartheta }\in \mathbb {R}^2). \end{aligned}$$

Furthermore, let us define $\widetilde{\Psi }(u, g\,T, \tau )=\widetilde{\Psi }_m(u, g\,T, \tau )\in \mathfrak {t}$ by setting

$$\begin{aligned} \widetilde{\Psi }(u, g\,T ) := u\, {\mathrm { diag }} \big ( {\mathrm {Ad}}_{g^{-1}} \big ( \Phi _G' (m) \big ) - \varvec{\nu }, \quad \Phi _G' (m) := -\imath \,\Phi _G (m). \end{aligned}$$

Finally, let us pose

$$\begin{aligned} \Psi (u, g\,T, \varvec{\vartheta } ) :=\left\langle \widetilde{\Psi }(u, g\,T ), \varvec{\vartheta } \right\rangle . \end{aligned}$$

The following proposition is proved by a rather lengthy computation, along the lines of those in the proof of Theorem 1.3 and in [26].

Proposition 6.1

$$\begin{aligned}&{ \imath \, k \, \left[ u \, \psi \left( \widetilde{\mu }_{ g \, e^{-\imath \varvec{\vartheta }/\sqrt{k}}\,g^{-1}} (x_{\tau ,k}), x_{\tau ,k} \right) - \frac{1}{\sqrt{k}}\, \langle \varvec{\nu }, \varvec{\vartheta } \rangle \right] }\nonumber \\&\quad = \imath \, \sqrt{k}\, \, \Psi (u, g\,T, \varvec{\vartheta } ) - \frac{ u }{ 2 } \, \varvec{\vartheta } ^t \, E( g\,T )\, \varvec{\vartheta } +2\,\imath \,u\,\tau \,\big \langle \varvec{\rho }_{g\,T}, \varvec{\vartheta }\big \rangle + k \, R_3 \left( \frac{\tau }{\sqrt{k}},\,\frac{ \varvec{\vartheta } }{\sqrt{k}} \right) . \end{aligned}$$

Corollary 6.1

(140) may be rewritten as follows:

$$\begin{aligned}&{ \Pi _{k\varvec{\nu }}(x_{\tau ,k},x_{\tau ,k}) }\nonumber \\&\quad \sim \dfrac{k\,(\nu _1-\nu _2)}{(2\pi )^2}\, \int _{G/T}\,\mathrm {d}V_{G/T}(gT)\,\int _{-\infty }^{\infty }\mathrm {d}\vartheta _1\,\int _{-\infty }^{\infty }\mathrm {d}\vartheta _2\, \int _0^{+\infty }\,\mathrm {d}u\nonumber \\&\qquad \left[ e^{\imath \, \sqrt{k}\, \, \Psi (u, g\,T, \varvec{\vartheta } )}\, \mathcal {A}_{k,\varvec{\nu }}(u, g\,T, \tau , \varvec{\vartheta } ) \right] , \end{aligned}$$

(141)

where (leaving implicit the dependence on x)

$$\begin{aligned} \mathcal {A}_{k,\varvec{\nu }}(u, g\,T, \tau , \varvec{\vartheta } ):= & {} e^{ - \frac{ u }{ 2 } \, \varvec{\vartheta } ^t \, E( g\,T )\, \varvec{\vartheta } +2\,\imath \,u\,\tau \,\big \langle \varvec{\rho }_{g\,T}, \varvec{\vartheta }\big \rangle + k \, R_3 \left( \frac{\tau }{\sqrt{k}}, \frac{ \varvec{\vartheta } }{\sqrt{k}} \right) }\, \Delta \left( e^{\imath \varvec{\vartheta }/\sqrt{k}}\right) \nonumber \\&\cdot s\left( \widetilde{\mu }_{g\,e^{-\imath \varvec{\vartheta }/\sqrt{k}}\,g^{-1}}(x_{\tau ,k}),x_{\tau ,k},k\,u\right) . \end{aligned}$$

(142)

Let $h_m\,T \in G/T$ be the unique coset such that $h_m^{-1}\, \Phi _G(m)\, h_m$ is diagonal. Then, only a rapidly decreasing contribution to the asymptotics is lost in (141), if integration in $\mathrm {d}V_{G/T}$ is localized in a small neighborhood of $h_m\,T$. In the following, a $\mathcal {C}^\infty$ bump function on G / T, supported in a small neighborhood of $h_m \, T$ and identically equal to 1 near $h_m \, T$, will be implicitly incorporated into the amplitude (142).

For some choice of $h_m\in h_m\,T$ and $\delta >0$ sufficiently small, let us consider the real-analytic map

$$\begin{aligned} h: w \in B(0;\delta )\subset \mathbb {C} \mapsto h (w) := h_m \, \exp \left( \imath \begin{pmatrix} 0 &{} w \\ \overline{w} &{} 0 \end{pmatrix} \right) \in G. \end{aligned}$$

By composition with the projection $\pi : G \rightarrow G/T$, we obtain a real-analytic coordinate chart on G / T centered at $h_m\, T \in G/T$, given by $w \in B(0;\delta ) \mapsto h(w) \, T \in G/T$. The Haar volume form on G / T has the form $\mathcal {V}_{G/T}(w) \, \mathrm {d}V_{\mathbb {C}} (w)$, where $\mathrm {d}V_{\mathbb {C}} (w)$ is the Lebesgue measure on $\mathbb {C}$, and $\mathcal {V}_{G/T}$ is a uniquely determined $\mathcal {C}^\infty$ positive function on $B(0;\delta )$. We record the following statements, whose proofs we shall omit for the sake of brevity.

Lemma 6.1

$\mathcal {V}_{G/T}$ is rotationally invariant, that is,

$$\begin{aligned} \mathcal {V}_{G/T}(w) = \mathcal {V}_{G/T} \left( e^{ \imath \,\theta } \, w \right) , \end{aligned}$$

for all $w \in B ( 0; \delta )$ and $e^{ \imath \,\theta } \in S^1$. In particular, $\mathcal {V}_{G/T}$ is given by a convergent power series in $r^2=|w|^2$ on $B ( 0; \delta )$.

Thus, we shall write

$$\begin{aligned} \mathcal {V}_{G/T} ( w ) = \mathcal {V}_{G/T} ( r ) = D_{ G/T } \cdot \mathcal {S} _{G/T} ( r ), \end{aligned}$$

(143)

where $D_{ G/T }>0$ is a constant, and $\mathcal {S} _{G/T} ( r ) = 1 + \sum _j s_j\, r^{ 2 j }$.

Lemma 6.2

Let $V_3$ be the total area of the unit sphere $S^3\subset \mathbb {C}^2$. Then,

$$\begin{aligned} D_{ G/T }=2\,\pi / V_3. \end{aligned}$$

Furthermore, let us introduce the real-analytic function

$$\begin{aligned} \kappa = \kappa _m : w \in B (0, \delta ) \mapsto {\mathrm { diag }} \Big ( {\mathrm {Ad}}_{h(w) ^{-1}} \big ( \Phi _G' (m) \big ) \Big )\in \mathbb {R}^2. \end{aligned}$$

(144)

Then, we also have the following.

Lemma 6.3

$\kappa$ is rotationally invariant and is given by a convergent power series of the following form

$$\begin{aligned} \kappa ( w )= & {} \lambda _{ \varvec{\nu } } ( m )\, \left[ \varvec{\nu } - r ^ 2 \, (\nu _1 - \nu _2 ) \, S_\kappa ( r )\,\mathbf {b} \right] ,\quad \mathbf {b}= \begin{pmatrix} 1\\ -1 \end{pmatrix}, \end{aligned}$$

where $r= | w |$, and $S_\kappa (r)$ is a real-analytic function of r, of the form

$$\begin{aligned} S_\kappa ( r ) = 1 + \sum _{ j \ge 1 } b_j \,r^{2 j }. \end{aligned}$$

If $w=r \, e^{ \imath \theta }$ in polar coordinates, we shall write accordingly $\mathcal {V}_{G/T}=\mathcal {V}_{G/T}( r )$ and $\kappa =\kappa ( r )$.

Recalling Definition 6.1 and (144), let us set

$$\begin{aligned} \widetilde{\Psi }_{ w }(u) : = u \, \kappa ( r ) - \varvec{ \nu }, \quad \Psi _{ w }(u, \varvec{\vartheta } ) :=\left\langle \widetilde{\Psi }_{ w }(u), \varvec{\vartheta } \right\rangle . \end{aligned}$$

(145)

We obtain the following integral formula (dependence on x on the right-hand sides is left implicit).

Proposition 6.2

As $k\rightarrow +\infty$ we have

$$\begin{aligned}&{ \Pi _{k\varvec{\nu }}(x_{\tau ,k},x_{\tau ,k}) }\nonumber \\&\qquad \sim D_{G/T}\,\frac{k\,(\nu _1-\nu _2)}{(2\pi )^2}\, \int _{-\pi }^\pi \,\mathrm {d}\theta \, \int _0 ^{ +\infty } \mathrm {d}\,r \,\left[ I_k(\tau , r, \theta )\right] , \end{aligned}$$

(146)

where

$$\begin{aligned} I_k(\tau , r, \theta )=I_k(\tau , w )&:= \int _{-\infty }^{\infty }\mathrm {d}\vartheta _1\,\int _{-\infty }^{\infty }\mathrm {d}\vartheta _2\, \int _0^{+\infty }\,\mathrm {d}u\nonumber \\&\quad \, \left[ e^{\imath \, \sqrt{k}\, \, \Psi _{w}(u, \varvec{\vartheta } ) }\, \mathcal {A}_{k,\varvec{\nu }}(u, h\left( r\,e^{ \imath \, \theta } \right) \,T, \tau , \varvec{\vartheta } ) \, \mathcal {S}_{G/T}(r)\,r\right] . \end{aligned}$$

(147)

Our next goal is to produce an asymptotic expansion for $I_k(\tau , r, \theta )$.

Definition 6.2

Let us set

$$\begin{aligned} \mathbf {n}_1( r ) := \frac{k ( r )}{\big \Vert k ( r ) \big \Vert }, \end{aligned}$$

and let $\mathbf {n}_2( r )$ be uniquely determined for $|r| < \delta$ so that $\mathcal {B}_{ r } := \left( \mathbf {n}_1( r ), \, \mathbf {n}_2( r ) \right)$ is a positively oriented orthonormal basis of $\mathbb {R}^2$. We shall write the change of basis matrix in the form

$$\begin{aligned} M^{ \mathcal {B}_{ r } }_{ \mathcal {C}_2 } ( id _{ \mathbb {R} ^2 }) = \begin{pmatrix} C ( r ) &{} - S ( r ) \\ S(r) &{} C (r) \end{pmatrix}, \end{aligned}$$

(148)

where $\mathcal {C}_2$ is the canonical basis of $\mathbb {R}^2$, and denote the change of coordinates by $\varvec{\vartheta } = \zeta _1 \, \mathbf {n}_1( w ) + \zeta _2 \, \mathbf {n}_2( w )$.

A straightforward computation then yields the following.

Corollary 6.2

With $w = r \, e^{ \imath \theta }\in B(0; \delta )$ and $I_k(\tau , w )$ as in (147), we have:

$$\begin{aligned} I_k(\tau , w )= & {} \int _{-\infty }^{\infty }\mathrm {d}\zeta _2\, \left[ e^{ -\imath \, \sqrt{k} \, \big \langle \varvec{ \nu }, \mathbf {n}_2( w ) \big \rangle \, \zeta _2}\, J_k( \tau , w; \zeta _2 )\, \mathcal {S}_{G/T}(r)\,r \right] , \end{aligned}$$

(149)

where

$$\begin{aligned}&{ J_k( \tau , w; \zeta _2 ) } \nonumber \\&\quad := \int _{-\infty }^{\infty }\mathrm {d}\zeta _1\, \int _0^{+\infty }\,\mathrm {d}u\, \left[ e^{\imath \, \sqrt{k}\, \Upsilon _{ r }( u, \zeta _1 ) }\, \mathcal {A}_{k,\varvec{\nu }}\big (u, h\left( w \right) \,T, \tau , \varvec{\vartheta }(\varvec{ \zeta } ) \big ) \right] , \end{aligned}$$

(150)

and

$$\begin{aligned} \Upsilon _{ r }( u, \zeta _1 ) := \big [ u\, \Vert \kappa ( r ) \Vert - \langle \varvec{ \nu }, \mathbf {n}_1( r ) \rangle \big ]\, \zeta _1. \end{aligned}$$

Let us view $J_k$ (150) as an oscillatory integral with phase $\Upsilon _{ r }$.

Lemma 6.4

$\Upsilon _{ r }$ has the unique critical point

$$\begin{aligned} P_{ r } = \big ( u ( r ), 0 \big ) := \left( \frac{\langle \varvec{ \nu }, \mathbf {n}_1( r ) \rangle }{ \Vert \kappa (r) \Vert }, 0 \right) . \end{aligned}$$

Furthermore, $\Upsilon _{ r } \big ( P_{ r } \big ) = 0$, and the Hessian matrix is

$$\begin{aligned} H (\Upsilon _{ r } ) _{ P_{ r } } = \begin{pmatrix} 0 &{} \Vert \kappa (r) \Vert \\ \Vert \kappa (r) \Vert &{} 0 \end{pmatrix}. \end{aligned}$$

Hence, its signature is zero and the critical point is non-degenerate.

In view of (142), and recalling that $s_0(x,x)=\pi ^{-d}$, the amplitude in (150) may be rewritten in the following form:

$$\begin{aligned}&{ \mathcal {A}_{k,\varvec{\nu }}\big (u, h(w)\,T, \tau , \varvec{\vartheta }( \varvec{\zeta } ) \big ) }\nonumber \\&\quad \sim e^{ - \frac{ u }{ 2 } \, \varvec{\vartheta } ( \varvec{\zeta } )^t \, E( w )\, \varvec{\vartheta }( \varvec{\zeta } ) +2\,\imath \,u\,\tau \,\big \langle \varvec{\rho }_{h(w)\,T}, \varvec{\vartheta }( \varvec{\zeta } )\big \rangle } \, \left[ e^{\frac{\imath }{\sqrt{k}} \, \vartheta _1( \varvec{\zeta } ) } - e^{\frac{\imath }{\sqrt{k}} \, \vartheta _2( \varvec{\zeta }) } \right] \,\left( \frac{ k \, u }{ \pi } \right) ^d \nonumber \\&\qquad \cdot \left[ 1 + \sum _{j\ge 1} a_j \big ( u, w; \tau , \varvec{\vartheta } ( \varvec{\zeta } ) \big ) \, k^{ - j/2 } \right] ; \end{aligned}$$

(151)

in (151) we have set $E ( w ) := \widetilde{E}\big ( h(w)\,T \big )$, and in view of the exponent $k\,R_3 ( \tau /\sqrt{k}, \varvec{\vartheta } / \sqrt{k} )$ appearing in (142), $a_j( u, w;\cdot , \cdot )$ is an appropriate polynomial in $(\tau ,\varvec{\vartheta })$ of degree $\le 3j$.

Given Lemma 6.4, we may evaluate $J_k$ in (150) by the stationary phase lemma, and obtain an asymptotic expansion in descending powers of $k^{1/2}$. The latter expansion may be inserted in (149), and integrated term by term, thus leading to an asymptotic expansion for $I_k$. The leading-order term of either expansion is determined by the contribution of the leading-order term in the asymptotic expansion for the amplitude in (40), which is given by the following:

$$\begin{aligned} J_k'( \tau , w; \zeta _2 )= & {} \,\left( \frac{ k }{ \pi } \right) ^d \, \int _{-\infty }^{\infty }\mathrm {d}\zeta _1\, \int _0^{+\infty }\,\mathrm {d}u\nonumber \\&\quad \left[ e^{\imath \, \sqrt{k}\, \Upsilon _{ w }( u, \zeta _1 ) }\,u^d\, \left( e^{\frac{\imath }{\sqrt{k}} \, \vartheta _1( \varvec{\zeta } ) } - e^{\frac{\imath }{\sqrt{k}} \, \vartheta _2( \varvec{\zeta }) } \right) \right. \nonumber \\&\quad \left. \cdot e^{ - \frac{ u }{ 2 } \, \varvec{\vartheta } ( \varvec{\zeta } ) ^t \, E( w )\, \varvec{\vartheta } ( \varvec{\zeta } ) +2\,\imath \,u\,\tau \,\big \langle \varvec{\rho }_{h(w)\,T}, \varvec{\vartheta } ( \varvec{\zeta } ) \big \rangle }\right] . \end{aligned}$$

(152)

Definition 6.3

Suppose $w = r \, e^{ \imath \theta }\in B(0; \delta )$ and let C(r) and S(r) be as in (148). Let us set

$$\begin{aligned} \mathfrak {a} ( w ):= & {} u ( r )\, \begin{pmatrix} -S(r)&C(r) \end{pmatrix} \, E \big ( w \big )\, \begin{pmatrix} -S(r) \\ C(r) \end{pmatrix} \\= & {} u ( r )\, \big \Vert {\mathrm {Ad}}_{h(w)} \big ( \mathbf {n}_2 ( r )\big ) _X(x) \big \Vert _x ^2 \end{aligned}$$

and

$$\begin{aligned} \mathfrak {r}(w):= & {} 2 \, u ( r )\, \big \langle \varvec{\rho }_{h(w)\,T}, \mathbf {n}_2 ( r )\big \rangle \\= & {} 2\,u ( r ) \, \omega _m\Big ({\mathrm {Ad}}_{h (w ) } \big (\mathbf {n}_2 ( r ) \big ) _M(m), \Upsilon _{\varvec{\nu }}(m)\Big ). \end{aligned}$$

Given the previous considerations, an application of the Stationary Phase Lemma yields the following.

Definition 6.4

With $|r| < \delta$, let us set $\mathfrak {b} ( r ) := \big \langle \varvec{ \nu }, \mathbf {n}_2( r ) \big \rangle$, and

$$\begin{aligned} D_l ( r ) := \frac{\imath ^l}{l ! \, \Vert \kappa ( r ) \Vert } \,\left[ C( r )^l + ( -1 ) ^{ l - 1 } \, S( r ) ^l \right] . \end{aligned}$$

The definition of $\mathfrak {b} ( r )$ implies:

$$\begin{aligned} \mathfrak {b} ( r ) = -\frac{ ( \nu _1 -\nu _2) \, (\nu _1 + \nu _2 )}{\Vert \varvec{ \nu } \Vert } \, r ^ 2 \, S _1( r ), \end{aligned}$$

(153)

where $S_1$ is a real-analytic function of the form $S_1 ( r ) = 1 + \sum _{j \ge 1 } c_j \, r^{ 2 j }$.

Proposition 6.3

Suppose $x\in X^G _{\mathcal {O}}$, and let $x_{ \tau , k }$ be as in (139). Then, as $k\rightarrow +\infty$ we have

$$\begin{aligned}&{ \Pi _{k\varvec{\nu }}(x_{ \tau , k } , x_{ \tau , k }) }\nonumber \\&\quad \sim D_{G/T}\,\frac{k\,(\nu _1-\nu _2)}{(2\pi )^2}\, \int _{-\pi }^\pi \,\mathrm {d}\theta \, \int _0 ^{ +\infty } \mathrm {d}\,r \,\left[ I_k(\tau , r, \theta )\right] , \end{aligned}$$

(154)

where $I_k(\tau , r, \theta )$ is given by an asymptotic expansion in descending powers of $k^{1/2}$, the leading power being $k ^ { d - 1 }$. As a function of $\tau$, aside from a phase factor, the coefficient of $k^{ d -(1+j) / 2 }$ is a polynomial of degree $\le 3 j$. Up to non-dominant terms, we may replace $I_k(\tau , w )$ by

$$\begin{aligned} I_k(\tau , w )'= & {} - \left( \frac{ k }{ \pi } \right) ^d \, \left( \frac{2\pi }{\sqrt{k}}\right) \, \mathcal {S}_{G/T}(r)\,r \cdot u ( w )^d \, \nonumber \\&\cdot \sum _{l\ge 1} \, \frac{ D_l ( r )}{k^{l/2}}\,\int _{-\infty }^{\infty }\mathrm {d}\zeta _2\, \left[ e^{ -\imath \,\sqrt{k} \, \zeta _2\, \mathfrak {f} _k ( \tau , w ) }\, \zeta _2 ^l \cdot e^{- \frac{1}{2} \, \mathfrak {a} ( w )\, \zeta _2^2 }\right] , \end{aligned}$$

(155)

where for $k=1,2,\ldots$, we have set

$$\begin{aligned} \mathfrak {f} _k ( \tau , w ) := \mathfrak {b} ( r ) - \frac{\tau }{k^{1/2}}\, \mathfrak {r}(w). \end{aligned}$$

(156)

The Gaussian integrals in (155) may be estimated recalling that

$$\begin{aligned} \int _{ -\infty }^{ + \infty } x ^l\, e^{ -\imath \xi \, x -\frac{1}{2} \, \lambda \, x ^2 }\mathrm {d} x = \sqrt{ 2 \pi } \, \frac{(-\imath )^l}{\lambda ^{l+1/2}} \, P_l(\xi )\, e^{ -\frac{1}{ 2 \lambda } \, \xi ^2}, \end{aligned}$$

(157)

where $P_l(\xi )= \xi ^l + \sum _{ j\ge 1 } \, p_{ lj }\, \xi ^{ l - 2 j }$ is a monic polynomial in $\xi$, of degree l and parity $(-1)^l$ (thus the previous sum is finite). Applying (157) with

$$\begin{aligned} \xi = k^{1/2}\, \mathfrak {f}_k(w,\tau ), \quad \lambda = \mathfrak {a} ( w ) \end{aligned}$$

we obtain the following conclusion.

Proposition 6.4

Let us set

$$\begin{aligned} F_l (\tau , w) := \frac{\sqrt{ 2 \pi }}{ l! }\, \left[ \frac{C( r )^l + ( -1 ) ^{ l - 1 } \, S( r ) ^l}{ \Vert \kappa ( r ) \Vert } \right] \, \frac{ P_l\left( \sqrt{k}\,\mathfrak {f} _k( \tau , w ) \right) }{ k^{l/2} \, \mathfrak {a} ( w )^{ l + 1/2} }. \end{aligned}$$

(158)

Up to lower-order terms, we can replace $I_k'$ in (155) by

$$\begin{aligned} I_k(\tau , w )'':= & {} - \left( \frac{ k }{ \pi } \right) ^d \, \left( \frac{2\pi }{\sqrt{k}}\right) \, \mathcal {S}_{G/T}(r)\,r \cdot u ( w )^d \, \nonumber \\&\cdot e^{ -\frac{1}{ 2 } \,k \, \frac{ \mathfrak {f}_k ( \tau , w ) ^2}{\mathfrak {a} ( w )}}\, \sum _{l\ge 1} F_l ( \tau , w ). \end{aligned}$$

(159)

Thus, the leading-order asymptotics of $\Pi _{k\varvec{\nu }}(x_{ \tau , k }, x_{ \tau , k } )$ are obtained by replacing $I_k(\tau , r, \theta )$ in (154) by $I_k(\tau , w )''$ given by (159).

6.2 Proof of Theorem 1.4

We shall set $\tau = 0$ in (154) and obtain an asymptotic estimate for $\Pi _{k\varvec{\nu }}(x, x )$ when $x\in X^G_{\mathcal {O}}$ and $k \rightarrow +\infty$.

Proof of Theorem 1.4

It follows from the definitions that

$$\begin{aligned} \frac{ \mathfrak {f}_k ( 0, w ) ^2}{\mathfrak {a} ( w )} = \frac{ \mathfrak {b} ( r ) ^2}{\mathfrak {a} ( w )} = \lambda _{ \varvec{ \nu } } ( m ) \, D(\varvec{ \nu } ) \,r^4 \, \mathcal {S} ( r, \theta ), \end{aligned}$$

(160)

where $\mathcal {S} (r, \theta ) = 1 +\sum _{j\ge 1} r^j\,d_j( \theta )$, and

$$\begin{aligned} D(\varvec{ \nu } ) := \frac{ ( \nu _1 - \nu _2 ) ^2 \, (\nu _1+\nu _2)^2 }{\Vert {\mathrm {Ad}}_{ h_m } (\varvec{ \nu _\perp })_M (m) \Vert ^2_m}. \end{aligned}$$

(161)

Similarly,

$$\begin{aligned} \frac{ P_l\left( \sqrt{k}\,\mathfrak {f} _k( 0, w ) \right) }{ k^{l/2} \, \mathfrak {a} ( w )^{ l + 1/2} }= & {} \frac{ P_l\left( \sqrt{k}\,\mathfrak {b} ( r )\right) }{ k^{l/2} \, \mathfrak {a} ( w )^{ l + 1/2} } \nonumber \\= & {} \frac{ 1 }{ \mathfrak {a} ( w )^{ l + 1/2} } \, \left[ \mathfrak {b} ( r ) ^l + \sum _{ j\ge 1 } ^{ \lfloor l/2 \rfloor } \, p_{ lj }\, k ^{- j }\,\mathfrak {b} ( r )^{ l - 2j } \right] \nonumber \\= & {} \sum _{j = 0} ^{ \lfloor l/2 \rfloor } \frac{ 1 }{ k^j } \, r^{ 2l - 4j } \, \mathcal {S}_{ l j } ( r, \theta ), \end{aligned}$$

(162)

where $\mathcal {S}_{ l j } (r, \theta )$ is a convergent power series in r. The resulting series may be integrated term by term. The lth summand in (159) then gives rise to a convergent series of summands of the form

$$\begin{aligned} B_{\varvec{ \nu }, l, j } ( m, \theta )\,\frac{1}{ k ^j } \,\int _0^{+\infty } \widetilde{r}^{2l - 4 j + a}\,e^{ -\frac{1}{ 2 } \,k \, \lambda _{ \varvec{\nu } } ( m )\, D ( \varvec{\nu } ) \cdot \widetilde{r} ^4}\, \widetilde{r}\, \mathrm {d}\widetilde{r} = O\left( \frac{1}{ k^{\frac{ l + 1 }{ 2 } + \frac{ a }{ 4 } } }\right) . \end{aligned}$$

(163)

with $j\le \lfloor l/2 \rfloor$ and $a=0,1,2,\ldots$.

The previous discussion shows that $\Pi _{k\varvec{\nu }}(x, x )$ is given by an asymptotic expansion in descending powers of $k^{1/4}$ and that the leading-order term occurs for $l=1$ and $a=0$.

By Lemma 157, $P_1 ( \xi ) = \xi$; by Lemma 6.3, $\Vert \kappa ( r ) \Vert = \lambda _{ \varvec{ \nu } } ( m ) \, \Vert \varvec{ \nu } \Vert \cdot \mathcal {S}'_{\kappa } (r)$, where $\mathcal {S}'_{\kappa } (r)$ is a convergent power series in $r^2$ with $\mathcal {S}'_{\kappa } (0) = 1$.

In view of (153) and (158), we obtain

$$\begin{aligned} F_1 (0, w ) = - \sqrt{2 \pi } \cdot \frac{ (\nu _1 - \nu _2 ) \, (\nu _1 + \nu _2)^2}{\Vert {\mathrm {Ad}}_{ h_m } (\varvec{ \nu _\perp })_M (m) \Vert ^{3}}\, \lambda _{ \varvec{ \nu } } ( m ) ^{ 1/2 }\, r^2 \,\mathcal {S}_{F_1} (r, \theta ), \end{aligned}$$

where $\mathcal {S}_{F_1}$ is real-analytic and $\mathcal {S}'' (0,\theta ) \equiv 1$.

Hence, the leading-order term of the asymptotic expansion of $\Pi _{k\varvec{\nu }}(x, x )$ is given by

$$\begin{aligned} D_{G/T}\,\frac{k\,(\nu _1-\nu _2)}{(2\,\pi ) ^ 2}\,\int _{-\pi } ^{\pi } \,\mathrm {d} \theta \, \int _0 ^{ +\infty } \mathrm {d}\,r \,\left[ L_k(r,\theta )\right] , \end{aligned}$$

(164)

where

$$\begin{aligned} L_k ( r ):= & {} 2^{ 3/2 } \, \frac{ k ^{d -1 /2 } }{ \pi ^{ d - 3/2 } } \, \lambda _{ \varvec{ \nu } } ( m ) ^{ -(d-1/2) } \nonumber \\&\cdot \left[ \frac{ (\nu _1 - \nu _2 ) \, (\nu _1 + \nu _2)^2}{\Vert {\mathrm {Ad}}_{ h_m } (\varvec{ \nu _\perp })_M (m) \Vert ^{3}}\right] \, e^{ -\frac{1}{ 2 } \,k \, \lambda _{ \varvec{ \nu } } ( m ) \, D(\varvec{ \nu } ) \,r^4\, \mathcal {S} ( r, \theta )}\,r^3\,\widetilde{\mathcal {S}} (r, \theta ),\, \end{aligned}$$

(165)

where again $\widetilde{\mathcal {S}}$ is real-analytic and $\widetilde{\mathcal {S}} (0,\theta ) \equiv 1$.

We need to integrate in $\mathrm {d} r$ the product of the last two factors in (165). Let us perform the coordinate change $s = \sqrt{k} \, r ^2\, \mathcal {S} ( r, \theta )^{1/2}$, and argue as above. To leading order, we are reduced to computing

$$\begin{aligned} \frac{1}{ 2\, k }\, \int _0 ^{ + \infty } \mathrm {d} s \left[ e^{ -\frac{1}{ 2 } \,\lambda _{ \varvec{ \nu } } ( m ) \, D(\varvec{ \nu }) \,s^2 }\,s \right] = \frac{1}{2 \,k } \cdot \frac{1}{\lambda _{ \varvec{ \nu } } ( m ) \, D(\varvec{ \nu })}. \end{aligned}$$

Inserting this in (164), we conclude that the leading-order term in the asymptotic expansion of $\Pi _k(x,x)$ is

$$\begin{aligned} \frac{D_{ G/T }}{\sqrt{ 2 }} \, \frac{1}{ \Vert \Phi _G ( m ) \Vert ^{ d +1/2 } } \, \left( \frac{ k \, \Vert \varvec{ \nu } \Vert }{ \pi } \right) ^{ d -1/2 } \cdot \frac{ \Vert \varvec{ \nu } \Vert }{ \Vert {\mathrm {Ad}}_{ h_m } (\varvec{ \nu _\perp })_M (m) \Vert }. \end{aligned}$$

The proof of Theorem 1.4 is complete. $\square$

7 Proof of Theorem 1.5

The proof is a modification of the one of Theorem 1.4, so the discussion will be sketchy. We shall set

$$\begin{aligned} x_{j,k} := x +\frac{1}{ \sqrt{k} } \, \mathbf {v}_j, \quad j=1,2. \end{aligned}$$

Definition 7.1

With the previous notation, let us set

$$\begin{aligned}&{ \Gamma (\varvec{\vartheta }, g\,T, \mathbf {v}_j) } \\&\quad := -\frac{ 1 }{ 2 } \, \left[ \Big \langle {\mathrm {diag}}\big ({\mathrm {Ad}}_{g^{-1}} (\Phi _G' (m))\big ), \varvec{\vartheta } \Big \rangle ^2 +\Big \Vert \mathbf {v}_1 - \mathbf {v}_2 + {\mathrm {Ad}}_g (\imath \, D_{\varvec{\vartheta }}) _M(m) \Big \Vert _m ^2\right] \\&\qquad +\, \imath \, \Big [- \omega _m (\mathbf {v}_1, \mathbf {v}_2) +\omega _m\big ( {\mathrm {Ad}}_g (\imath \, D_{\varvec{\vartheta }}) _M(m),\mathbf {v}_1 + \mathbf {v}_2\big ) \Big ]. \end{aligned}$$

Then, the same computations leading to Proposition 6.1 yield the following.

Proposition 7.1

$$\begin{aligned}&{ \imath \, k \, \left[ u \, \psi \left( \widetilde{\mu }_{ g \, e^{-\imath \varvec{\vartheta }/\sqrt{k}}\,g^{-1}} (x_{1,k}), x_{2,k} \right) - \frac{1}{\sqrt{k}}\, \langle \varvec{\nu }, \varvec{\vartheta } \rangle \right] }\nonumber \\&\quad = \imath \, \sqrt{k}\, \, \Psi (u, g\,T, \varvec{\vartheta } ) + u \, \Gamma (\varvec{\vartheta }, g\,T, \mathbf {v}_j) + k \, R_3 \left( \frac{\mathbf {v}_j }{\sqrt{k}},\,\frac{ \varvec{\vartheta } }{\sqrt{k}} \right) . \end{aligned}$$

Remark 7.1

Assuming $\mathbf {v}_1, \,\mathbf {v}_2 \in \mathfrak {g}_M(m_x)^{\perp _h}$, recalling Definition 6.1 we have

$$\begin{aligned} \Gamma (\varvec{\vartheta }, g\,T, \mathbf {v}_j) = \psi _2 ( \mathbf {v}_1, \mathbf {v}_2 )-\frac{ 1 }{ 2 } \, \varvec{\vartheta }^t \,E( g\,T )\, \varvec{\vartheta }. \end{aligned}$$

In place of Corollary 6.1, we then obtain the following:

$$\begin{aligned}&{ \Pi _{k\varvec{\nu }}(x_{1, k},x_{2 , k}) }\nonumber \\&\qquad \sim \dfrac{k\,(\nu _1-\nu _2)}{(2\pi )^2}\, \int _{G/T}\,\mathrm {d}V_{G/T}(gT)\,\int _{-\infty }^{\infty }\mathrm {d}\vartheta _1\,\int _{-\infty }^{\infty }\mathrm {d}\vartheta _2\, \int _0^{+\infty }\,\mathrm {d}u\nonumber \\&\qquad \quad \left[ e^{\imath \, \sqrt{k}\, \, \Psi (u, g\,T, \varvec{\vartheta } )}\, \mathcal {A}'_{k,\varvec{\nu }}(u, g\,T, \varvec{\vartheta }, \mathbf {v}_j ) \right] , \end{aligned}$$

(166)

with the new amplitude

$$\begin{aligned} \mathcal {A}'_{k,\varvec{\nu }}(u, g\,T, \varvec{\vartheta }, \mathbf {v}_j ):= & {} e^{u \, \psi _2 ( \mathbf {v}_1, \mathbf {v}_2 ) - \frac{ u }{ 2 } \, \varvec{\vartheta } ^t \, E( g\,T )\, \varvec{\vartheta } + k \, R_3 \left( \frac{\tau }{\sqrt{k}}, \frac{ \varvec{\vartheta } }{\sqrt{k}} \right) }\, \Delta \left( e^{\imath \varvec{\vartheta }/\sqrt{k}}\right) \nonumber \\&\cdot s\left( \widetilde{\mu }_{g\,e^{-\imath \varvec{\vartheta }/\sqrt{k}}\,g^{-1}}(x_{1,k}),x_{2,k},k\,u\right) . \end{aligned}$$

(167)

Similarly, in place of (151) we now have the following expansion:

$$\begin{aligned}&{ \mathcal {A}'_{k,\varvec{\nu }}(u, g\,T, \varvec{\vartheta }, \mathbf {v}_j ) }\nonumber \\&\qquad \sim e^{u \, \psi _2 ( \mathbf {v}_1, \mathbf {v}_2 ) - \frac{ u }{ 2 } \, \varvec{\vartheta } ^t \, E( g\,T )\, \varvec{\vartheta } + k \, R_3 \left( \frac{\tau }{\sqrt{k}}, \frac{ \varvec{\vartheta } }{\sqrt{k}} \right) } \, \left[ e^{\frac{\imath }{\sqrt{k}} \, \vartheta _1( \varvec{\zeta } ) } - e^{\frac{\imath }{\sqrt{k}} \, \vartheta _2( \varvec{\zeta }) } \right] \,\left( \frac{ k \, u }{ \pi } \right) ^d \nonumber \\&\quad \qquad \cdot \left[ 1 + \sum _{j\ge 1} a_j \big ( u, w; \mathbf {v}_1, \mathbf {v}_2, \varvec{\vartheta } ( \varvec{\zeta } ) \big ) \, k^{ - j/2 } \right] , \end{aligned}$$

(168)

where $a_j$ is, as a function of $\mathbf {v}_1$ and $\mathbf {v}_2$, a polynomial of degree $\le 3j$.

With these changes, Theorem 1.5 can be proved by applying the arguments in the proof of Theorem 1.4 with minor modifications.

8 Proof of Theorem 1.6

Proof

Let $A'\subset X$ be a one-sided ‘outer’  tubular neighborhood of $X^G_{ \mathcal {O}}$, that is, the intersection of A with a tubular neighborhood of $X^G_{ \mathcal {O}}$ in X.

By Theorem 1.1, we have

$$\begin{aligned}&{ \dim _{out} H(X)_{ k\,\varvec{ \nu } } } \nonumber \\&\quad = \int _A \Pi _{ k\, \varvec{ \nu } } ( x, x ) \, \mathrm {d}V_X (x) \sim \int _{A'}\Pi _{ k\, \varvec{ \nu } } ( x, x ) \, \mathrm {d}V_X (x). \end{aligned}$$

(169)

Let us denote by $\sigma ( \varvec{ \nu } )$ the sign of $\nu _1 + \nu _2$. Then, locally along $X^G_{ \mathcal {O}}$, for some sufficiently small $\delta >0$ we can parametrize $A'$ by a diffeomorphism

$$\begin{aligned} \Gamma :X^G_{ \mathcal {O} } \times [0, \delta ) \rightarrow A', \quad (x,\tau ) \mapsto x +\tau \, \sigma ( \varvec{ \nu } ) \, \Upsilon _{ \varvec{ \nu } } ( m_x ), \end{aligned}$$

where $m_x = \pi (x)$. The latter expression is meant in terms of a collection of smoothly varying systems of Heisenberg local coordinates centered at $x\in X^G_{ \mathcal {O} }$, locally defined along $X^G_{ \mathcal {O}}$ (to be precise, one ought to work locally on $X^G_{ \mathcal {O} }$, introduce an appropriate open cover of $X^G_{ \mathcal {O} }$, and a subordinate partition of unity; however for the sake of exposition we shall omit details on this).

We shall set $x_{ \tau } := \Gamma ( x, \tau )$, and write

$$\begin{aligned} \Gamma ^* (\mathrm {d}V _X ) = \mathcal {V} _X ( x, \tau ) \, \mathrm {d}V _{ X ^G_{ \mathcal {O} }} ( x ) \,\mathrm {d}\tau , \end{aligned}$$

where $\mathcal {V} _X : X^G_{ \mathcal {O} } \times [0, \delta ) \rightarrow (0,+\infty )$ is $\mathcal {C}^\infty$ and $\mathcal {V} _X ( x, 0 ) = \big \Vert \Upsilon _{ \varvec{ \nu } } ( m_x )\big \Vert$.

Hence, we obtain

$$\begin{aligned}&{ \dim _{out} H(X)_{ k\,\varvec{ \nu } } } \nonumber \\&\quad \sim \int _{X ^G_{ \mathcal {O} }} \, \mathrm {d}V _{ X ^G_{ \mathcal {O} }} ( x )\, \int _0^{\delta } \, \mathrm {d} \tau \, \left[ \mathcal {V} _X ( x, \tau ) \, \Pi _{ k\, \varvec{ \nu } } ( x_{ \tau }, x_{ \tau } ) \right] . \end{aligned}$$

(170)

By Theorem 1.3, only a rapidly decreasing contribution to (170) is lost, if integration in (170) is restricted to the locus where $\tau \le C \, k^{\epsilon - 1/2 }$. Thus, the asymptotics of $\dim _{out} H(X)_{ k\,\varvec{ \nu } }$ are unchanged, if the integrand is multiplied by a rescaled cut-off function $\varrho \left( k ^{ 1/2 - \epsilon } \, \tau \right)$, where $\varrho$ is identically one sufficiently near the origin in $\mathbb {R}$, and vanishes outside a slightly larger neighborhood.

With the rescaling $\tau \mapsto \tau / \sqrt{ k }$, we obtain

$$\begin{aligned} \dim _{out} H(X)_{ k\,\varvec{ \nu } } \sim \frac{1}{ \sqrt{ k } } \, \int _{X ^G_{ \mathcal {O} }} \, \mathrm {d}V _{ X ^G_{ \mathcal {O} }} ( x )\,\Big [ \mathcal {H} _{ k } ( x ) \Big ], \end{aligned}$$

where with $x_{ \tau , k } := \Gamma \left( x, k^{ -1/2 } \, \tau \right)$ we have set

$$\begin{aligned} \mathcal {H} _{ k } ( x ) := \int _0^{ +\infty } \, \mathrm {d} \tau \, \left[ \varrho \left( k ^{ - \epsilon } \, \tau \right) \, \mathcal {V} _X \left( x, \frac{\tau }{ \sqrt{ k } } \right) \, \Pi _{ k\, \varvec{ \nu } } ( x_{ \tau , k }, x_{ \tau , k} ) \right] . \end{aligned}$$

(171)

Integration in $\mathrm {d} \tau$ is now over an expanding interval of the form $\left[ 0, C' \, k^{ \epsilon } \right)$.

Let us consider the asymptotics of (171). Having in mind (159), and inserting the Taylor expansion of $\mathcal {V} _X$, we are led to considering double integrals of the form

$$\begin{aligned}&{ \frac{1}{ k^{(l+j)/2} }\, \int _0^{ +\infty } \, \mathrm {d} \tau \,\int _0^{ +\infty } \, \mathrm {d} r } \nonumber \\&\qquad \left[ r \, C( r ) ^l \, \tau ^j \, \mathcal {S}' (r )\frac{ P_l\left( \sqrt{k}\,\mathfrak {f} _k( \tau , w ) \right) }{ \mathfrak {a} ( w )^{ l + 1/2} } \, \cdot e^{ -\frac{1}{ 2 } \,k \, \frac{ \mathfrak {f}_k ( \tau , w ) ^2}{\mathfrak {a} ( w )}} \right] , \end{aligned}$$

(172)

with $l \ge 1$ and $j \ge 0$, and their analogs with S(r) in place of C(r) ; $\mathcal {S}'$ is some real-analytic function (dependence on $\theta$ and x is implicit).

In view of (156), we have

$$\begin{aligned} \frac{\mathfrak {f} _k ( \sigma ( \varvec{ \nu } ) \, \tau , w )}{\sqrt{ \mathfrak {a} ( w )}} = - \sigma ( \varvec{ \nu } ) \, \left[ \frac{ ( \nu _1 -\nu _2 ) \, | \nu _1 + \nu _2 |}{\Vert \varvec{ \nu } \Vert \, \sqrt{ \mathfrak {a} ( 0 )} } \, r ^ 2 \, S _1( r ) + \frac{\tau }{k^{1/2}}\, \, \frac{\mathfrak {r} ( 0 ) }{ \sqrt{ \mathfrak {a} ( 0 ) } }\, S _2 ( r,\theta )\right] , \end{aligned}$$

where again $S _2 ( 0,\theta )=1$. Therefore, with the change of variables

$$\begin{aligned} s := k^{ 1 / 4 } \, r\, \sqrt{ S _1( r ) }, \quad \widetilde{\tau } := \tau \, S _2 ( r,\theta ) \end{aligned}$$

we obtain

$$\begin{aligned} \frac{\mathfrak {f} _k ( \sigma ( \varvec{ \nu } ) \, \tau , w )}{\sqrt{ \mathfrak {a} ( w )}} = -\frac{ \sigma ( \varvec{ \nu } ) \, }{ \sqrt{ k } } \, \left[ \frac{ ( \nu _1 -\nu _2) \, |\nu _1 + \nu _2 |}{\Vert \varvec{ \nu } \Vert \, \sqrt{ \mathfrak {a} ( 0 )} } \, s^2 + \frac{\mathfrak {r} ( 0 ) }{ \sqrt{ \mathfrak {a} ( 0 ) } } \, \widetilde{\tau } \right] . \end{aligned}$$

Therefore, we also have

$$\begin{aligned} \mathfrak {f} _k ( \sigma ( \varvec{ \nu } ) \, \tau , w ) = -\frac{ \sigma ( \varvec{ \nu } ) }{ \sqrt{ k } } \, \left[ \frac{ ( \nu _1 -\nu _2) \, | \nu _1 + \nu _2 | }{\Vert \varvec{ \nu } \Vert } \, s^2 + \mathfrak {r} ( 0 ) \, \widetilde{\tau } \right] \cdot \left[ 1 + R_1\ \left( \frac{s}{\root 4 \of {k}} \right) \right] . \end{aligned}$$

With the substitution $a = s^2$, (172) may be rewritten as a linear combination of summands of the form

$$\begin{aligned}&{ \frac{1}{ k^{(l+j+1)/2} }\, \int _0^{ +\infty } \, \mathrm {d} \tau \,\int _0^{ +\infty } \, \mathrm {d} a } \nonumber \\&\qquad \left[ C\left( \frac{\sqrt{a}}{\root 4 \of {k}} \right) ^l \, \left( A_1 \, a + B_1 \, \tau \right) ^{b} \, \tau ^j\cdot \left[ 1 + R_1 \left( \frac{ \sqrt{a} }{\root 4 \of {k}} \right) \right] \cdot e^{ -\frac{1}{ 2 } \, \left( A_1 \, a + B_1 \, \tau \right) ^2} \right] \nonumber \\&\quad = O \left( \frac{1}{ k^{(l+j+1)/2} } \right) . \end{aligned}$$

(173)

Hence, the leading contribution occurs for $l=1$, $j = 0$, and dropping the term $R_1 \left( k^{ - 1/4 } \, \sqrt{a} \right)$. The conclusion of Theorem 1.6 then follows by a fairly simple computation. $\square$