The twisted Selberg trace formula and the twisted Selberg zeta function for compact orbifolds

Abstract

We propose a version of the Selberg trace formula for compact hyperbolic orbifolds $\Gamma \backslash {\mathbb {H}}^{2n+1}$ for non-unitary representations of $\Gamma$ and establish that the associated Selberg zeta function admits a meromorphic continuation to ${\mathbb {C}}$.

1 Introduction

The Selberg zeta function is an important tool in the study of the spectral theory of locally symmetric Riemannian spaces. This zeta function is defined by an infinite product over the closed geodesics that only converges in a complex half-space. For its investigation it is useful to understand if it admits a meromorphic continuation. The purpose of this paper is to prove the existence of a meromorphic continuation of the Selberg zeta functions on compact odd-dimensional hyperbolic orbifolds by establishing a suitable Selberg trace formula.

Theorem 1

Suppose ${\mathcal {O}}= \Gamma \backslash {\mathbb {H}}^{2n+1}$ is a compact odd-dimensional hyperbolic orbifold, $\chi$ is a (possibly non-unitary) finite-dimensional representation of $\Gamma$, and $\tau$ is a unitary finite-dimensional representation of SO(2n). Then the Selberg zeta function $Z(s, \tau , \chi )$ (see Definition 33 below) admits a meromorphic continuation to ${\mathbb {C}}$.

The key point of the proof of Theorem 1 is to show that the residues of $Z'(s,\tau , \chi )/Z(s,\tau , \chi )$ are integers. If ${\mathcal {O}}$ is a compact hyperbolic manifold and $\chi$ is unitary, this was proven by [1]. Later on their result was extended to non-compact finite volume hyperbolic manifolds with cusps when $\chi$ is unitary [7] and when $\chi$ is a restriction of a representation of $SO_0(1,2n+1)$ [17]. Using a slightly different approach, the theorem was proved in [24] for compact orbifolds when $\chi$ and $\tau$ are trivial representations.

The approach of [1, 7, 17] is due to Selberg and consists in applying the Selberg trace formula to a certain test function which makes $Z'(s,\tau , \chi )/Z(s,\tau , \chi )$ appear as one of the terms in the geometric side of the formula. In order to adapt their approach, we need to prove a more general version of the Selberg trace formula.

The Selberg trace formula has a rich history starting from the classical work [19], but has mostly been constrained to unitary representations $\chi$ of $\Gamma$. The non-unitary case was first studied in [16] under the assumption that $\Gamma$ contains no non-trivial elements of finite order, also called elliptic elements, and no non-trivial unipotents elements, also called parabolic elements, which means ${\mathcal {O}}$ is a compact manifold. We drop the former restriction on $\Gamma$ and prove:

Theorem 2

Let G be a connected real semisimple Lie group of non-compact type with finite center, K a maximal compact subgroup of G, and $\Gamma \subset G$ a discrete subgroup such that ${\mathcal {O}}:= \Gamma \backslash G / K$ is compact. Let $\chi$ be a (possibly non-unitary) finite-dimensional representation of $\Gamma$, and $\tau$ be a unitary finite-dimensional representation of K. For the non-selfadjoint Laplacian $\Delta _{\chi , \tau }^\#$ defined in Sect. 7.2 and $\varphi$ belonging to the space of Paley-Wiener functions ${\mathcal {P}}({\mathbb {C}})$ defined in Sect. 4,

$$\begin{aligned} \sum _{\lambda \in \text {spec}(\Delta _{\chi , \tau }^\#)} m(\lambda ) \varphi (\lambda ^{1/2}) = \sum _{\{\gamma \} \subset \Gamma } {\text {vol}}(\Gamma _\gamma \backslash G_\gamma ) E_\gamma (h_\varphi ). \end{aligned}$$

In the formula above, $m(\lambda )$ is the multiplicity of $\lambda$; $\{\gamma \}$ denotes the conjugacy class of $\gamma$; $G_\gamma$ and $\Gamma _\gamma$ are the centralizers of $\gamma$ in G and $\Gamma$, respectively. Finally, $E_\gamma (h_\varphi )$ are the orbital integrals, defined by

$$\begin{aligned} E_\gamma (h_\varphi ):= \int _{G_\gamma \backslash G} {\text {tr}}\, h_\varphi (g \gamma g^{-1}) d \dot{g}, \end{aligned}$$

where $h_\varphi$ is the integral kernel of the self-adjoint Laplacian $\varphi (\widetilde{\Delta }_\tau ^{1/2})$ defined in Sect. 5.

To complete the proof of Theorem 1 we apply Theorem 2 to the case $G/K = {\mathbb {H}}^{2n+1}$. The major remaining problem is to study the orbital integrals $I_\varphi (\gamma )$ for the elliptic elements $\gamma \in \Gamma$. We show that the Fourier transform of orbital integrals is given by polynomials:

Lemma 3

In the above setup $G/K = {\mathbb {H}}^{2n+1}$ the orbital integral $I_\varphi (\gamma )$ for elliptic $\gamma \in \Gamma$ equals

$$\begin{aligned} I_\varphi (\gamma ) = \sum _{\sigma ' \in \widehat{SO(2n)}} \int _{\mathbb {R}}\Theta _{\sigma ', \lambda }(\varphi ) P_\gamma (i \lambda ) d \lambda , \end{aligned}$$

where $\widehat{SO(2n)}$ is the unitary dual of SO(2n), $\Theta _{\sigma ', \lambda }(\varphi )$ is the character of the unitarily induced representation $\pi _{\sigma ', \lambda }$ of G, and $P_\gamma (\lambda )$ is a certain even polynomial.

Orbital integrals have so far been computed for $G/K = {\mathbb {H}}^2$, ${\mathbb {H}}^3$ and ${\mathbb {H}}^{2n}$ in [4, 9] and [23] respectively. The computation of orbital integrals is not only useful for the proof of Theorem 1, but also for other applications of the Selberg trace formula. For example, we will use Lemma 3 in the upcoming papers to study the behavior of the analytic torsion of odd-dimensional compact and non-compact finite-volume orbifolds.

2 Introduction

In this section we will establish some notation and recall some basic facts about hyperbolic orbifolds and representations of the involved Lie groups.

2.1 Orbifolds

Definition 1

Let $\widetilde{U}$ be a Riemannian manifold and $G_U$ be a discrete group of isometries acting effectively on $\widetilde{U}$, that is for every $g_1, g_2 \in G_U$, $g_1 \ne g_2$ there exists a point $u \subset \widetilde{U}$ with $g_1 u \ne g_2 u$. We assume that $G_U$ acts properly discontinuously, that is for any $x,y \in \widetilde{U}$ there exist $U_x \supset x$ and $U_y \supset y$, both $U_x$ and $U_y$ open, such that

$$\begin{aligned} \{g \in G_U: (g\cdot U_x) \cap U_y \ne \emptyset \} \end{aligned}$$

is finite. Then $G_U \backslash \widetilde{U}$ is a good Riemannian orbifold.

Throughout the article we assume that the orbifolds we are dealing with are good.

Definition 2

Let Q be a topological space. An orbifold chart on Q is a triple $(\widetilde{U},G_U, \phi _U)$, where $\widetilde{U}$ is a connected open subset in ${\mathbb {R}}^n$, $G_U$ is a finite group acting on $\widetilde{U}$, $\phi _U: \widetilde{U} \rightarrow Q$ is a map with an open image $\phi _U(\widetilde{U})$ which induces a homeomorphism from $G_U\backslash \widetilde{U}$ to $\phi _U(\widetilde{U})$. Further we put $U:= \phi _U(\widetilde{U})$. In this case, $(\widetilde{U},G_U, \phi _U)$ is said to uniformize U.

Definition 3

Define $\lambda$ to be a smooth embedding between two orbifold charts $(\widetilde{U}_1, G_1, \varphi _1)$ and $(\widetilde{U}_2, G_2, \varphi _2)$, if $\lambda$ is a smooth embedding between $\widetilde{U}_1$ and $\widetilde{U}_2$ such that $\varphi _2 \circ \lambda = \varphi _1$. Two orbifolds charts $(\widetilde{U}_i, G_i, \varphi _i)$ uniformizing $U_i$, $i=1,2$ are called compatible if for any point $x \in U_1 \cap U_2$ there exists an open neighborhood V of x and an orbifold chart $(\widetilde{V}, H, \phi )$ of V such that there are two smooth embeddings $\lambda _i: (\widetilde{V}, H, \phi ) \rightarrow (\widetilde{U}_i, G_i, \varphi _i)$, $i=1,2$.

Definition 4

An orbifold atlas on an orbifold ${\mathcal {O}}$ is a collection of pairwise compatible orbifold charts $(\widetilde{U}_i, G_i, \varphi _i)$ uniformizing $U_i$ with $i \in I$ such that ${\mathcal {O}}= \cup _{i\in I} U_i$.

Definition 5

Two orbifolds atlases are equivalent if their union is an orbifold atlas. An orbifold structure on ${\mathcal {O}}$ is an equivalence class of orbifold atlaces on ${\mathcal {O}}$.

Definition 6

Let ${\mathcal {O}}= G_W \backslash \widetilde{W}$ be an orbifold. The vector orbibundle $E \rightarrow {\mathcal {O}}$, associated to a representation $\rho : G_W \rightarrow \text {End}(V_\rho )$ is defined as

$$\begin{aligned} E:= G_U \backslash (\widetilde{W} \times V_\rho ) \rightarrow G_U \backslash \widetilde{W}, \end{aligned}$$

where $G_W$ acts on $(w, v) \in \widetilde{W} \times V_\rho$ as follows:

$$\begin{aligned} g: (w,v) \mapsto (g w, \rho (g) v), \quad g \in G_W. \end{aligned}$$

Definition 7

We define smooth sections of the orbibundle E as

$$\begin{aligned} C^\infty ({\mathcal {O}}, E) = \{ f \in C^\infty (\widetilde{W}, V_\rho ), f(\gamma x) = \rho (\gamma ) f(x), \quad x \in \widetilde{W}, \gamma \in G_W \}. \end{aligned}$$

Definition 8

Let F be a fundamental domain for the action of $G_U$ on $\widetilde{W}$. Then

$$\begin{aligned} C_c^\infty ({\mathcal {O}}, E) = \{f \in C^\infty ({\mathcal {O}}, E), \quad {\text {supp}}(f|_F) \text { is compact} \}. \end{aligned}$$

Remark 1

Note that the definition above does not depend on the choice of a fundamental domain F.

2.2 Locally homogeneous vector bundles

Let $G = \text {SO}_0(1,2n+1)$, $K=\text {SO}(2n+1)$. Let $G = NAK$ be an Iwasawa decomposition of G with respect to K. For each $g \in G$ there are uniquely determined elements $n(g) \in N$, $a(g) \in A$, $\kappa (g) \in K$ such that $g = n(g) a(g) \kappa (g).$ Let M be the centralizer of A in K, thus $M = \text {SO}(2n).$ Denote the Lie algebras of G, K, A, M and N by $\mathfrak {g}$, $\mathfrak {k}$, $\mathfrak {a}$, $\mathfrak {m}$ and $\mathfrak {n}$, respectively. Define the Cartan involution $\theta :\mathfrak {g} \rightarrow \mathfrak {g}$ by $\theta (Y) = - Y^t$, $Y \in \mathfrak {g},$ and let $\mathfrak {g}=\mathfrak {k} \oplus \mathfrak {p}$ be the Cartan decomposition of $\mathfrak {g}$ with respect to $\theta$.

Let $\nu : K \rightarrow GL(V_\nu )$ be a finite-dimensional unitary representation of K on $(V_\nu , \langle \cdot , \cdot \rangle _\nu )$.

Definition 9

[13, p. 4] Denote by

$$\begin{aligned} \widetilde{E}_\nu := (G \times V_\nu )/K \rightarrow G/K \end{aligned}$$

the associated homogeneous vector bundle, where K acts on $G \times V_{\nu }$ by

$$\begin{aligned} (g,v) k = (gk, \nu (k^{-1}) v), \quad g\in G, \, k\in K, \, v\in V_\nu . \end{aligned}$$

Denote by $C^\infty (G/K, \widetilde{E}_\nu )$ the space of smooth sections of $\widetilde{E}_\nu$. Let $C_0^\infty (G/K, \widetilde{E}_\nu )$ be the sections of $\widetilde{E}_\nu$ with compact support.

Note that $\langle \cdot , \cdot \rangle _\nu$ induces a G-invariant metric on $\widetilde{E}_\nu$. Denote by $L^2(G/K, \widetilde{E}_\nu )$ the space of $L^2$-sections of $\widetilde{E}_\nu$. Let

$$\begin{aligned} C^\infty (G;\nu ){} & {} := \{ f:G\rightarrow V_\nu \, | \, f\in C^\infty , f(gk)=\nu (k^{-1})f(g), \quad g\in G, k\in K \};\nonumber \\ C^\infty (\Gamma \backslash G; \nu ){} & {} := \{ f \in C^\infty (G; \nu ), f(\gamma g)=f(g) \quad g\in G, \gamma \in \Gamma \}. \end{aligned}$$

(1)

Similarly, we denote by $C_c^\infty (G; \nu )$ the subspace of compactly supported functions in $C^\infty (G;\nu )$ and by $L^2(G;\nu )$ the completion of $C_c^\infty (G;\nu )$ with respect to the inner product

$$\begin{aligned} \langle f_1,f_2\rangle = \int _{G / K} \langle f_1(g), f_2(g)\rangle d\dot{g}. \end{aligned}$$

Proposition 4

[13, p. 4] There is a canonical isomorphism

$$\begin{aligned} C^\infty (G/K, \tilde{E}_\nu ) \cong C^\infty (G; \nu ). \end{aligned}$$

(2)

Similarly, there are isomorphisms of topological vector spaces $C_c^\infty (G/K, \tilde{E}_\nu ) \cong C_c^\infty (G; \nu )$ and $L^2(G/K, \tilde{E}_\nu ) \cong L^2(G;\nu )$.

Definition 10

Let $\nabla ^\nu$ be the canonical G-invariant connection on $\widetilde{E}_\nu$ defined by

$$\begin{aligned} \nabla _{g_* Y}^\nu f(g K):= \frac{d}{dt}\Big | _{t = 0} (g \exp (tY))^{-1} f(g \exp (tY) K), \end{aligned}$$

where $f \in C^\infty (G;\nu )$ and $Y \in \mathfrak {p}$.

Definition 11

Denote by $\widetilde{\Delta }_\nu = (\widetilde{\nabla }^\nu )^* \widetilde{\nabla }^\nu$ the associated Bochner-Laplace operator.

Let $\Omega \in Z(\mathfrak {g}_{\mathbb {C}})$ and $\Omega _K \in Z(\mathfrak {k}_{\mathbb {C}})$ be the Casimir elements of G and K, respectively. Assume that $\nu$ is irreducible. Let R denote the right regular representation of G on $C^\infty (G; \nu )$.

Proposition 5

[13, Proposition 1.1] With respect to (2), we have

$$\begin{aligned} \tilde{\Delta }_\nu = -R(\Omega ) + \nu (\Omega _K) {\text {Id}}, \end{aligned}$$

(3)

where $\nu (\Omega _K) \ge 0$ is the Casimir eigenvalue of $\nu$.

Definition 12

Let $E_\nu := \Gamma \backslash \tilde{E}_\nu$ be the locally homogeneous vector orbibundle over $\Gamma \backslash G/K$ induced by $\tilde{E}_\nu$.

2.3 Lie groups

Let $H: G \rightarrow \mathfrak {a}$ be defined by

$$\begin{aligned} H(g):= \log a(g). \end{aligned}$$

(4)

There is a G-invariant metric on G/K which is unique up to scaling. Suitably normalized, it is the hyperbolic metric, and G/K is isometric to ${\mathbb {H}}^{2n+1}$.

Denote by $E_{i,j}$ the matrix in $\mathfrak {g}$ whose (i, j)’th entry is 1 and the other entries are 0. Let

$$\begin{aligned} \begin{aligned} H_1&:= E_{1,2} + E_{2,1},\\ H_j&:= \sqrt{-1} \cdot (E_{2j-1,2j} - E_{2j,2j-1}), \quad j = 2, \ldots , n+1. \end{aligned} \end{aligned}$$

Then $\mathfrak {a} = {\mathbb {R}}H_1$, where $\mathfrak {a}$ is from Sect. 2.3.

Definition 13

Define $A^+ = \{\exp (tH_1), \, t>0 \}$.

Let $\mathfrak {b} = \sqrt{-1}\cdot {\mathbb {R}}H_2 + \cdots + \sqrt{-1}\cdot {\mathbb {R}}H_{n+1}$ be the standard Cartan subalgebra of $\mathfrak {m}$. Moreover, $\mathfrak {h} = \mathfrak {a} \oplus \mathfrak {b}$ is a Cartan subalgebra of $\mathfrak {g}$. Denote by $\mathfrak {h}_{{\mathbb {C}}}, \mathfrak {g}_{{\mathbb {C}}}, \mathfrak {m}_{{\mathbb {C}}}, \mathfrak {b}_{{\mathbb {C}}}, \mathfrak {k}_{{\mathbb {C}}}$ the complexification of $\mathfrak {h}, \mathfrak {g}, \mathfrak {m}, \mathfrak {b}, \mathfrak {k}$, respectively. Define $e_i \in \mathfrak {h}_{{\mathbb {C}}}^*$ with $i = 1, \ldots , n+1$, by

$$\begin{aligned} e_i(H_j) = \delta _{i,j}, \, 1\le i,j\le n+1. \end{aligned}$$

(5)

The sets of roots of $(\mathfrak {g}_{\mathbb {C}}, \mathfrak {h}_{{\mathbb {C}}})$ and $(\mathfrak {m}_{{\mathbb {C}}}, \mathfrak {b}_{{\mathbb {C}}})$ are given by

$$\begin{aligned} \begin{aligned} \Delta (\mathfrak {g}_{\mathbb {C}}, \mathfrak {h}_{{\mathbb {C}}}) = \{ \pm e_i \pm e_j, 1 \le i< j \le n+1\},\\ \Delta (\mathfrak {m}_{{\mathbb {C}}}, \mathfrak {b}_{{\mathbb {C}}}) = \{ \pm e_i \pm e_j, 2\le i < j \le n+1 \}. \end{aligned} \end{aligned}$$

(6)

We fix a positive systems of roots by

$$\begin{aligned} \begin{aligned} \Delta ^+(\mathfrak {g}_{\mathbb {C}}, \mathfrak {h}_{{\mathbb {C}}}) = \{ e_i \pm e_j, 1 \le i< j \le n+1\},\\ \Delta ^+(\mathfrak {m}_{{\mathbb {C}}}, \mathfrak {b}_{{\mathbb {C}}}) = \{ e_i \pm e_j, 2\le i < j \le n+1 \}. \end{aligned} \end{aligned}$$

(7)

The half-sum of the positive roots $\Delta ^+(\mathfrak {m}_{{\mathbb {C}}}, \mathfrak {b}_{{\mathbb {C}}})$ equals

$$\begin{aligned} \rho _M = \sum _{j=2}^{n+1} \rho _j e_j,\quad \rho _j = n+1-j. \end{aligned}$$

(8)

Let $M'$ be the normalizer of A in K, and let

$$\begin{aligned} W(A)=M'/M \end{aligned}$$

(9)

be the restricted Weyl group. It has order 2 and acts on finite-dimensional representations of M [17, p. 18]. Denote by $w_0$ the non-identity element of W(A).

2.4 Representations

Let $\sigma : M \mapsto \text {End}(V_\sigma )$ be a finite-dimensional irreducible representation of M.

Definition 14

We define ${\mathcal {H}}^\sigma$ to be the space of measurable functions $f:K\mapsto V_\sigma$ such that

1.:

$f(mk)=\sigma (m) f(k)$ for all $k \in K$ and $m \in M$;

2.:

$\int _K || f(k)||^2 dk < \infty$.

Recall $H: G \rightarrow \mathfrak {a}$, $\kappa : G \rightarrow K$ as in Sect. 2.3, and $e_1 \in \mathfrak {h}_{\mathbb {C}}^*$ is as in Sect. 2.3. For $\lambda \in {\mathbb {R}}$, define the representation $\pi _{\sigma , \lambda }$ of G on ${\mathcal {H}}^\sigma$ by the following formula:

$$\begin{aligned} \pi _{\sigma ,\lambda } (g) f(k):= e^{(i \lambda + n) (H(kg))} f(\kappa (kg)). \end{aligned}$$

(10)

2.5 Compact hyperbolic orbifolds

The main subject of study of the article are compact hyperbolic orbifolds $\Gamma \backslash {\mathbb {H}}^{2n+1}$. Compactness implies that all non-identity elements of $\Gamma$ are either hyperbolic or elliptic.

Definition 15

An element $\gamma \in \Gamma$ is called hyperbolic if

$$\begin{aligned} l(\gamma ):= \inf _{x \in {\mathbb {H}}^{2n+1}} d(x, \gamma x) > 0, \end{aligned}$$

where d(x, y) denotes the hyperbolic distance between x and y.

Remark 2

Some authors use the term “loxodromic” instead of “hyperbolic”.

Lemma 6

[25, Lemma 6.6] For hyperbolic $\gamma$ there exists $g \in G$, $m_\gamma \in SO(2n)$, $a_\gamma \in A^+$, where $A^+$ is from Definition 13, such that $g \gamma g^{-1} = m_\gamma a_\gamma .$ Here $a_\gamma$ is unique, and $m_\gamma$ is determined up to conjugacy in SO(2n).

Definition 16

A non-identity element $\gamma \in \Gamma$ is called elliptic if it is of finite order.

An alternative definition is the following: an element $\gamma$ is elliptic if and only if it is conjugate to a non-identity element in K, so without loss of generality we may assume $\gamma$ is of the form:

$$\begin{aligned} \gamma = {\text {diag}} \left( \overbrace{\left( {\begin{matrix} 1 &{} 0 \\ 0 &{} 1 \end{matrix}} \right) , \ldots , \left( {\begin{matrix} 1 &{} 0 \\ 0 &{} 1 \end{matrix}} \right) }^k, \overbrace{R_{\phi _{k+1}}, \ldots , R_{\phi _{n+1}}}^{n-k+1}\right) , \end{aligned}$$

(11)

where $n-k+1\ne 0$ and $R_\phi = \left( {\begin{matrix} \cos \phi &{} \sin \phi \\ -\sin \phi &{} \cos \phi \end{matrix}} \right)$, $\phi \in (0,2\pi )$. There is an even number of eigenvalue 1 in (11), because an element $\gamma$ should belong to $\text {SO}_0(1,2n+1)$.

Definition 17

An elliptic element $\gamma$ is regular if the centralizer $G_\gamma$ of $\gamma$ in G is isomorphic to $\text {SO}_0(1,1) \times \text {SO}(2)^{n-1}$.

3 Orbifolds, orbibundles and pseudodifferential operators

3.1 Sobolev spaces

To define Sobolev norms on an orbifold ${\mathcal {O}}$, we first define Sobolev norms locally. Let $\widetilde{U}$ and $G_U$ be as in Definition 2. Note that if $G_U$ is finite, then

$$\begin{aligned} C_0^\infty (G_U \backslash \widetilde{U}, (G_U \backslash (\widetilde{U}\times {\mathbb {R}}^k)) \cong C_0^\infty (\widetilde{U}, \widetilde{U} \times {\mathbb {R}}^k)^{G_U}, \end{aligned}$$

(12)

where $C_0^\infty (\widetilde{U}, \widetilde{U} \times {\mathbb {R}}^k)^{G_U}$ denotes the space of $G_U$-equivariant sections of $C_0^\infty (\widetilde{U}, \widetilde{U} \times {\mathbb {R}}^k)$ and $C_0^\infty (G_U \backslash \widetilde{U}, (G_U \backslash (\widetilde{U}\times {\mathbb {R}}^k))$ is from Definition 8. The space $C_0^\infty (\widetilde{U}, \widetilde{U} \times {\mathbb {R}}^k)$ is equipped with usual Sobolev norm $||\cdot ||_s$, and this norm restricts to ${G_U}$-invariant sections. We equip $C_0^\infty (\widetilde{U}, \widetilde{U} \times {\mathbb {R}}^k)^{G_U}$, and hence $C_0^\infty (G_U \backslash \widetilde{U}, (G_U \backslash (\widetilde{U}\times {\mathbb {R}}^k))$ with the following norm:

$$\begin{aligned} || f' ||_{s; U}:= \frac{1}{|{G_U}|} || f||_s \end{aligned}$$

(13)

for $f' \in C_0^\infty (G_U \backslash \widetilde{U}, (G_U \backslash (\widetilde{U}\times {\mathbb {R}}^k))$ and the corresponding element $f \in C_0^\infty (\widetilde{U}, \widetilde{U} \times {\mathbb {R}}^k)$. Next we use an orbifold atlas and a partition of unity to define the Sobolev norm on the space of smooth sections of an orbibundle $E \rightarrow {\mathcal {O}}$. Sobolev norms defined using equivalent atlases will be themselves equivalent. The space $H^s({\mathcal {O}},E)$ denotes the completion of $C^\infty ({\mathcal {O}};E)$ with respect to any of these norms; put $L^2({\mathcal {O}}; E):= H^0({\mathcal {O}};E)$.

Remark 3

The isomorphism (12) does not necessarily hold if ${G_U}$ is infinite. For example, let $\gamma$ act on ${\mathbb {R}}$ by $x \cdot \gamma = x+1$ and put ${G_U} = \{\gamma ^n: n \in {\mathbb {Z}}\}$. Then $C_0^\infty ({\mathbb {R}})^{G_U} = \{0\}$, but $C_0^\infty ({G_U}\backslash {\mathbb {R}})\ne \{0\}$.

3.2 Pseudodifferential operators

We recall some basic facts about pseudodifferential operators on orbibundles. For more details see [2, p. 28], [10, Sect. 2.2].

Definition 18

Let $E \rightarrow {\mathcal {O}}$ be an orbibundle. For any orbifold chart $(\widetilde{U}, G_U, \phi _U)$ of ${\mathcal {O}}$, let $(\widetilde{U}\times V_\rho , G_U, \widetilde{\phi }_U)$ be a local trivialization of E over $(\widetilde{U}, G_U, \phi _U)$ as in [10, Sect. 2.2]. A linear mapping $A: C^\infty ({\mathcal {O}}, E) \rightarrow C^\infty ({\mathcal {O}}, E)$ is a pseudodifferential operator on $E \rightarrow {\mathcal {O}}$ of order m if:

• 1. the Schwartz kernel of A is smooth outside any neighborhood of the diagonal in ${\mathcal {O}}\times {\mathcal {O}}$,

• 2. for any $x \in {\mathcal {O}}$ and for any local trivialization $(\widetilde{U}\times V_\rho , G_U, \widetilde{\phi }_U)$ of E over an orbifold chart $(\widetilde{U}, G_U, \phi _U)$ with $x \in U$, the operator

  $$\begin{aligned} C_c^\infty (U, E) \ni f \mapsto A(f) |_U \in C^\infty (U, E) \end{aligned}$$

  is given by the restriction to $G_U$-invariant functions of a pseudodifferential operator $\widetilde{A}$ of order m on $C^\infty (\widetilde{U}, V_\rho )$ that commutes with the induced $G_U$-action on $C^\infty (\widetilde{U}, V_\rho )$.

Definition 19

A pseudodifferential operator A on ${\mathcal {O}}$ is elliptic if a pseudodifferential operator $\widetilde{A}$ is elliptic for any choice of orbifold charts.

The Sobolev embedding and the Kondrachov–Rellich theorem are valid as in the case of manifolds:

Proposition 7

(Sobolev embedding) For $s > s'$, the embedding $H^s({\mathcal {O}}) \subset H^{s'}({\mathcal {O}})$ is continuous.

Proposition 8

(Kondrachov–Rellich theorem) Let ${\mathcal {O}}$ be compact and $s > s'$, then the embedding $H^s({\mathcal {O}}) \subset H^{s'}({\mathcal {O}})$ is compact.

Proof of Propositions 7 and 8

Instead of the original proofs [21, p. 60], one chooses a partition of unity and reduces the theorems to their local versions in a single chart. As sections over orbifold charts are $G_U$-invariant sections over the corresponding smooth charts, the desired proofs are obtained by repeating the local arguments from [21] verbatim for the subspaces of $G_U$-invariant sections. $\square$

Remark 4

For another proof of the Sobolev embedding and the Kondrachov–Rellich theorem on orbifolds, see [3].

Remark 5

Let ${\mathcal {O}}$ be compact. Note that any pseudodifferential operator of order 0 extends to a bounded operator in $L^2({\mathcal {O}}, E)$; compare [21, Theorem 6.5]. Moreover, the Proposition 8 implies that any pseudodifferential operator of negative order is compact; compare [21, Corollary 6.2].

Theorem 9

Let H be a second order elliptic pseudodifferential operator acting on sections of an orbibundle E over a compact good orbifold ${\mathcal {O}}$ with the leading symbol

$$\begin{aligned} \sigma (H) (x, \xi ) = || \xi ||^2_x \cdot {\text {Id}}_{E_x}, \quad x \in {\mathcal {O}}, \xi \in T^*_x{\mathcal {O}}. \end{aligned}$$

(14)

For a subset $I \subset [-\pi , \pi ]$, let

$$\begin{aligned} \Lambda _I:= \{ r e^{i \phi }: 0 \leqslant r < \infty , \, \phi \in I\} \end{aligned}$$

and

$$\begin{aligned} B_R(0):= \{ x\in {\mathbb {C}}, \,|x|\le R\}. \end{aligned}$$

Then for every $0< \varepsilon < \pi /2$, there exists $R > 0$ such that the spectrum of H is contained in the set $B_R(0) \cup \Lambda _{[- \varepsilon , \varepsilon ]}$. Moreover, the spectrum of H is discrete, and there exists $R \in {\mathbb {R}}$ such that for $|\lambda | > R$ and $\lambda \not \in \Lambda _{[-\varepsilon ,\varepsilon ]}$,

$$\begin{aligned} ||(H - \lambda )^{-1}|| \leqslant C / |\lambda |. \end{aligned}$$

Proof

The proof of theorem is similar to the smooth case for which we refer to [21, Theorems 9.3 and 8.4], except for the following: in the case of manifolds a partition of unity reduces the proof to ${\mathbb {R}}^n$, whereas in our case it is $G_U \backslash {\mathbb {R}}^n$, where $G_U$ is a finite group. $\square$

4 Functional analysis

In this section we refine the necessary facts from functional analysis from [16, Sect. 2] for the case of compact orbifolds. The main difference from the case of compact manifolds is that we replace all theorems involving Sobolev spaces to their orbifold analogues from the previous section. Note that though we assume our orbifold is good this assumption is not used until the end of this section. The requirement on ${\mathcal {O}}$ to be compact is crucial, because we will need Remark 5.

Let $E \rightarrow {\mathcal {O}}$ be a Hermitian orbibundle, pick a Hermitian metric in E and let $\nabla$ be a covariant derivative in E which is compatible with the Hermitian metric.

Definition 20

The operator

$$\begin{aligned} \Delta _E = \nabla ^* \nabla \end{aligned}$$

(15)

is the Bochner–Laplacian associated to the connection $\nabla$ and the Hermitian fiber metric.

By [2, Theorem 3.5], the Bochner–Laplace operator $\Delta _E$ is essentially selfadjoint. We denote its selfadjoint extension by the same symbol. Consider the class of elliptic operators

$$\begin{aligned} H: C^\infty ({\mathcal {O}}, E) \rightarrow C^\infty ({\mathcal {O}}, E), \end{aligned}$$

(16)

which are perturbations of the Laplace operator $\Delta _E$ by a first order differential operator, i.e.

$$\begin{aligned} H = \Delta _E + D_1, \end{aligned}$$

(17)

where $D_1: C^\infty ({\mathcal {O}}, E) \rightarrow C^\infty ({\mathcal {O}}, E)$ is a first order differential operator.

For every $0< \epsilon < \pi /2$ there exists $R >0$ such that the spectrum of H is contained in $B_R(0) \cup \Lambda _{[-\epsilon , +\epsilon ]}$ by Theorem 9. Though H is not self-adjoint in general, it has nice spectral properties. The reason is the following: $D_1 (\Delta _E-\lambda )^{-1}$ is a pseudodifferential operator of order $-1$, and hence by Remark 5 is compact. This implies [12] that $L^2({\mathcal {O}}, E)$ is the closure of the algebraic direct sum of finite-dimensional H-invariant subspaces $V_k$

$$\begin{aligned} L^2({\mathcal {O}}, E) = \overline{ \bigoplus _{k \geqslant 1} V_k}, \end{aligned}$$

(18)

such that the restriction of H to $V_k$ has a unique eigenvalue $\lambda _k$, and for each k there exists $N_k \in {\mathbb {N}}$ such that $(H - \lambda _k \cdot {\text {Id}})^{N_k} V_k = 0$, and $|\lambda _k| \rightarrow \infty$.

Denote by $\text {spec}(H)$ the spectrum of H. Suppose that $0 \not \in \text {spec}(H)$. It follows from Theorem 9 that there exists an Agmon angle $\theta$ for H, and we can define the square root $H^{1/2}_\theta$. If $\theta$ is fixed, we simply denote $H_\theta ^{1/2}$ by $H^{1/2}$. Note that $H^{1/2}$ is a classical pseudodifferential operator with the leading symbol

$$\begin{aligned} \sigma (H^{1/2}) (x, \xi ) = || \xi ||_x \cdot {\text {Id}}_{E_x}. \end{aligned}$$

(19)

By the spectral theorem we can define $\Delta _E^{1/2}$. The principal symbols of $H^{1/2}$ and $\Delta _E^{1/2}$ coincide, hence

$$\begin{aligned} H^{1/2} = \Delta ^{1/2}_E+B_0, \end{aligned}$$

(20)

where $B_0$ is a pseudodifferential operator of order zero.

Lemma 10

The resolvent of $H^{1/2}$ is compact, and the spectrum of $H^{1/2}$ is discrete. There exists $b > 0$ and $d \in {\mathbb {R}}$ such that the spectrum of $H^{1/2}$ is contained in the domain

$$\begin{aligned} \Omega _{b,d}:= \{ \lambda \in {\mathbb {C}}: \text {Re}(\lambda ) > d,\, |\text {Im}(\lambda )|<b \}. \end{aligned}$$

Proof

The proof is similar to [16, Lemma 2.3]. First note that $H^{1/2}$ is an elliptic pseudodifferential operator of order 1, hence by Remark 5 its resolvent is compact, that implies the spectrum of $H^{1/2}$ is discrete. Second, the operator $B_0$ extends to a bounded operator in $L^2({\mathcal {O}}, E)$ by Remark 5; denote

$$\begin{aligned} b:=2 \cdot ||B_0||. \end{aligned}$$

(21)

Recall that [8, Chapter V, (3.16)] for $\lambda \not \in \text {spec}(\Delta _E^{1/2})$,

$$\begin{aligned} || (\Delta _E^{1/2}-\lambda \cdot {\text {Id}})^{-1}|| \le |\text {Im}(\lambda )|^{-1}. \end{aligned}$$

(22)

The equations (21) and (22) imply

$$\begin{aligned} || B_0 \cdot (\Delta _E^{1/2} - \lambda \cdot {\text {Id}})^{-1}|| \le 1/2, \quad |\text {Im}(\lambda )| \ge b, \end{aligned}$$

and hence $I + B_0 \cdot (\Delta _E^{1/2} - \lambda \cdot {\text {Id}})^{-1}$ is invertible for such $\lambda$, and

$$\begin{aligned} ||(I + B_0 \cdot (\Delta _E^{1/2} - \lambda \cdot {\text {Id}})^{-1})^{-1}|| \le 2, \quad |\text {Im}(\lambda )| \ge b. \end{aligned}$$

Moreover,

$$\begin{aligned} (H^{1/2}-\lambda \cdot {\text {Id}})^{-1} = (\Delta _E - \lambda \cdot {\text {Id}})^{-1} \cdot \left( I + B_0 \cdot (\Delta _E^{1/2} - \lambda \cdot {\text {Id}})^{-1}\right) ^{-1}, \end{aligned}$$

that together with (22) implies

$$\begin{aligned} ||(H^{1/2}-\lambda \cdot {\text {Id}})^{-1}|| \le 2 \cdot |\text {Im}(\lambda )|^{-1}, \quad |\text {Im}(\lambda )|\ge b, \end{aligned}$$

hence the $\text {spec}(H^{1/2}) \subset \Omega _{b,d}$. $\square$

It follows from the spectral decomposition (18) that $H^{1/2}$ has the same spectral decomposition as H with eigenvalues $\lambda ^{1/2}$, $\lambda \in \text {spec}(H)$ and multiplicities $m(\lambda ^{1/2}) = m(\lambda )$.

We need to introduce some class of function for further use.

Definition 21

Denote by $PW({\mathbb {C}})$ be the space of Paley-Wiener functions on ${\mathbb {C}}$, that is

$$\begin{aligned} PW({\mathbb {C}}) = \cup _{R>0} PW^R({\mathbb {C}}) \end{aligned}$$

with the inductive limit topology. Above $PW^R({\mathbb {C}})$ is the space of entire functions $\phi$ on ${\mathbb {C}}$ such that for every $N \in {\mathbb {N}}$ there exists $C_N > 0$ such that

$$\begin{aligned} |\phi (\lambda )| \leqslant C_N (1+|\lambda |)^{-N} e^{R|\text {Im}(\lambda )|}, \quad \lambda \in {\mathbb {C}}. \end{aligned}$$

Proposition 11

Given $h\in C_c^\infty ((-R,R))$, let

$$\begin{aligned} \varphi (\lambda ) = \frac{1}{\sqrt{2 \pi }} \int _{\mathbb {R}}h(r) e^{-ir\lambda } dr, \quad \lambda \in {\mathbb {C}}, \end{aligned}$$

(23)

be the Fourier-Laplace transform of h. Then $\varphi$ satisfies (23) for every $N \in {\mathbb {N}}$, that is $\varphi \in PW^R({\mathbb {C}})$. Conversely, by the Paley-Wiener theorem, every $\phi \in PW^R({\mathbb {C}})$ is the Fourier-Laplace transform of a function in $C_c^\infty ((-R,R))$.

Recall that we are assuming $0 \not \in \text {spec}(H)$.

Definition 22

For $b > 0$ and $d \in {\mathbb {R}}$ let $\Gamma _{b,d}$ be the contour which is the union of the two half-lines $L_{\pm b,d}:= \{ z \in {\mathbb {C}}: \text {Im}(z)=\pm b, \, \text {Re}(z)\geqslant d \}$ and the semicircle $S = \{d +be^{i \theta }: \pi /2 \leqslant \theta \leqslant 3\pi /2 \}$, oriented clockwise.

By Lemma 10 there exists $b>0$, $d \in {\mathbb {R}}$ such that $\text {spec}(H^{1/2})$ is contained in the interior of $\Gamma _{b,d}$. For an even Paley-Wiener function $\varphi$ (or $\varphi (\cdot ) = e^{- t (\cdot )^2}$) put

$$\begin{aligned} \varphi (H^{1/2}):= \frac{i}{2\pi }\int _{\Gamma _{b,d}} \varphi (\lambda ) (H^{1/2}-\lambda )^{-1} d\lambda . \end{aligned}$$

(24)

Lemma 12

$\varphi (H^{1/2})$ is an integral operator with a smoothing kernel.

Proof

The proof follows in the same way as in [16, Lemma 2.4]. For $k, l \in {\mathbb {N}}$ we have

$$\begin{aligned} H^k \varphi (H^{1/2}) H^{l} = \frac{i}{2\pi }\int _{\Gamma _{b,d}} \lambda ^{2(k+l)} \varphi (\lambda ) (H^{1/2}-\lambda )^{-1} d\lambda . \end{aligned}$$

The operator $H^k \varphi (H^{1/2}) H^{l}$ is a bounded operator in $L^2({\mathcal {O}},E)$, since $\lambda \mapsto \lambda ^{2(k+l)} \varphi (\lambda )$ is rapidly decreasing on $L_{\pm b,d}$. One can easily observe that as in the case of manifolds, $H^{s}({\mathcal {O}},E)$ is the completion of $C^\infty ({\mathcal {O}},E)$ with respect to the norm $|| ({\text {Id}}+ H)^{s/2} f||$, where $|| \cdot ||$ is the $L^2$-norm. It follows that for all $s,r \in {\mathbb {R}}$, $\varphi (H^{1/2})$ extends to a bounded operator from $H^s({\mathcal {O}},E)$ to $H^r({\mathcal {O}},E)$ and hence is a smoothing operator. $\square$

5 The heat kernel

Let $E \rightarrow {\mathcal {O}}$ be an orbibundle from Definition 6, equip it with a Hermitian fibre metric and let $\Delta$ be the Bochner–Laplace operator acting on sections of E; assume that E is a complex orbibundle of rank k. Our goal is to construct and study the heat kernel for $\Delta$.

5.1 Existence and uniqueness of the heat kernel

Definition 23

We say that $K \in \Gamma ((0,\infty ) \times {\mathcal {O}}\times {\mathcal {O}},E\boxtimes E^*)$ is a heat kernel, if it satisfies:

• 1. K is $C^0$ in all three variables, $C^1$ in the first, and $C^2$ in the second,

• 2. $(\frac{\partial }{\partial t}+ \Delta _x) K(t,x,y) = 0$, where $\Delta _x$ denotes $\Delta$ that acts on the second variable,

• 3. $\lim _{t \rightarrow +0} K(t,x, \cdot ) = \delta _x$ for all $x \in {\mathcal {O}}$, where $\delta _x$ is the Dirac distribution.

If ${\mathcal {O}}= \Gamma \backslash {\mathbb {H}}^{2n+1}$ is a compact hyperbolic orbifold (hence, $\Gamma$ is finitely generated), the existence and the uniqueness of the heat kernel follow from the following two lemmas:

Lemma 13

[20, Lemma 8] A finitely generated group $\Gamma$ of matrices over a field of characteristic zero has a normal torsion-free subgroup $\Gamma _0$ of finite index. $\square$

Lemma 14

Let $E \rightarrow {\mathcal {O}}$ be an orbibundle over a good Riemannian orbifold; moreover, assume that ${\mathcal {O}}= \Gamma _f \backslash M$, where M is a compact manifold and $\Gamma _f$ is a finite group of orientation-preserving isometries of M. Denote by $\widetilde{E}$ the lift of E to M, and let $K(t,\widetilde{x},\widetilde{y})$ be the heat kernels on $\widetilde{E}$. Let $\pi :M \mapsto {\mathcal {O}}$ be a natural projection, and let

$$\begin{aligned} R_\gamma : \widetilde{E} \mapsto \widetilde{E} \end{aligned}$$

(25)

be the induced bundle map for $\gamma \in \Gamma _f$. Then the heat kernel $K^{\mathcal {O}}(t,x,y)$ on E equals

$$\begin{aligned} K^{\mathcal {O}}(t,x,y) = \sum _{\gamma \in \Gamma _f} K(t, \tilde{x},\gamma \tilde{y}) \circ R_\gamma , \end{aligned}$$

(26)

where $\tilde{x}$ and $\tilde{y}$ are elements of $\pi ^{-1}(x)$ and $\pi ^{-1}(y)$, respectively. $\square$

5.2 Computation of the heat asymptotics

Lemma 15

For $K^{\mathcal {O}}(t,x,y)$ as in Lemma 14, we have

$$\begin{aligned} \int _{\mathcal {O}}{\text {tr}}\, K^{\mathcal {O}}(t,x,x)\, d{\text {vol}}_{\mathcal {O}}(x) \sim I_e(t) + \sum _{\gamma \in \Gamma _f, \gamma \ne e} I_\gamma (t), \quad t \rightarrow 0, \end{aligned}$$

(27)

where

$$\begin{aligned} \begin{aligned} I_e(t) \sim t^{-\dim ({\mathcal {O}})/2} \sum _{k=0}^\infty a_k t^k, \quad t\rightarrow 0,\\ I_\gamma (t) \sim t^{-\dim (N_\gamma )/2} \sum _{k=0}^\infty a^\gamma _k t^k, \quad t \rightarrow 0. \end{aligned} \end{aligned}$$

Above $a_k$, $a_k^\gamma$ are some coefficients in ${\mathbb {C}}$, and $N_\gamma$ is the fixed point set of $\gamma$ in M.

Proof

It follows from (26) that

$$\begin{aligned} \begin{aligned}&\int _{\mathcal {O}}{\text {tr}}\, K^{\mathcal {O}}(t,x,x) \, d {\text {vol}}_{\mathcal {O}}(x) = \frac{1}{|\Gamma _f|} \int _M {\text {tr}}\, K(t, \tilde{x}, \tilde{x}) \, d {\text {vol}}_M (\widetilde{x}) \\&\quad +\frac{1}{|\Gamma _f|} \sum _{e \ne \gamma \in \Gamma _f} \int _M {\text {tr}}\, \left( K(t, \tilde{x}, \gamma (\tilde{x}))\circ R_\gamma \right) \, d {\text {vol}}_M (\widetilde{x}), \end{aligned} \end{aligned}$$

(28)

where $d {\text {vol}}_M$ and $d {\text {vol}}_{\mathcal {O}}$ denotes the Riemannian measure on M and ${\mathcal {O}}$, respectively. We study the asymptotic behavior of (28) following [5]. The asymptotic expansion of the first summand in the right hand side of (28) follows from the following theorem:

Theorem 16

[5, Theorem 1.7.6] There exist $a_k(x):M \rightarrow {\mathbb {C}}$, $k=0,\ldots , \infty$ such that

$$\begin{aligned} \int _M {\text {tr}}\, K(t, x, x) dx \sim \sum _{k=0}^\infty t^{(k-\dim (M))/2} \int _M a_k(x) \, d{\text {vol}}_M (x), \quad t\rightarrow 0. \end{aligned}$$

Remark 6

The leading coefficient is given by $a_0(x) = (4 \pi )^{-\dim (M)/2}$.

The asymptotic expansion of the second summand of the right hand side of (28) follows with minor modification from [5, Lemma 1.8.2]:

Theorem 17

There exist $a_n^\gamma (x):N_\gamma \rightarrow {\mathbb {C}}$, $k=0,\ldots , \infty$ such that

$$\begin{aligned} \int _M {\text {tr}}\,\left( K(t, \widetilde{x}, \gamma \widetilde{x} ) \circ R_\gamma \right) \, d {\text {vol}}_M(\widetilde{x}) \sim \sum _{n=0}^\infty t^{(n-\dim (N_\gamma ))/2} \int _{N_\gamma } a^{\gamma }_n(x) d{\text {vol}}_{N_\gamma } (x), \end{aligned}$$

where $d{\text {vol}}_{N_\gamma }(x)$ denotes the Riemannian measure on $N_\gamma$.

Proof

By [5, Lemma 1.8.2],

$$\begin{aligned} \int _M {\text {tr}}\left( R_\gamma ^{-1} \circ K(t, \gamma \widetilde{x}, \widetilde{x} ) \right) \,d {\text {vol}}_M(\widetilde{x}) \sim \sum _{n=0}^\infty t^{(n-\dim (N_\gamma ))/2} \int _{N_\gamma } a^{\gamma }_n(x) d{\text {vol}}_{N_\gamma } (x). \end{aligned}$$

(29)

Note that

$$\begin{aligned} \begin{aligned} {\text {tr}}\left( R_\gamma ^{-1} \circ K(t, \gamma \widetilde{x}, \widetilde{x} ) \right)&= {\text {tr}}\left( R_\gamma \circ R_\gamma ^{-1} \circ K(t, \gamma \widetilde{x}, \widetilde{x} ) \circ R_\gamma ^{-1} \right) \\&={\text {tr}}\left( K(t, \gamma \widetilde{x}, \widetilde{x} ) \circ R_\gamma ^{-1} \right) . \end{aligned} \end{aligned}$$

(30)

Let $\widetilde{y} = \gamma \widetilde{x}$, then

$$\begin{aligned} \int _M {\text {tr}}\left( K(t, \gamma \widetilde{x}, \widetilde{x} ) \circ R_\gamma ^{-1} \right) d {\text {vol}}_M(\widetilde{x}) = \int _M {\text {tr}}\left( K(t, \widetilde{y}, \gamma ^{-1} \widetilde{y}) \circ R_{\gamma ^{-1}} \right) d {\text {vol}}_M(\widetilde{y}). \end{aligned}$$

(31)

Putting together (29), (30) and (31) implies Theorem 17. $\square$

Applying Theorems 16 and 17 finishes the proof of Lemma 15. $\square$

As usual, we establish the Weyl law.

Theorem 18

Let

$$\begin{aligned} N(r, \Delta ):= \sum _{\lambda \in \text {spec}(\Delta ), |\lambda |\le r} m(\lambda ) \end{aligned}$$

be the counting function of the spectrum of $\Delta$, where eigenvalues are counted with algebraic multiplicity. Then

$$\begin{aligned} N(r,\Delta ) = \frac{\textrm{rk}(E) {\text {vol}}({\mathcal {O}})}{(4 \pi )^{(\dim {\mathcal {O}})/2} \Gamma ((\dim {\mathcal {O}})/2+1)} r^{(\dim {\mathcal {O}})/2} + o(r^{(\dim {\mathcal {O}})/2}), \quad r \rightarrow \infty . \end{aligned}$$

Proof

Follows from the Tauberian theorem, Lemma 15 and Remark 6. $\square$

6 Functional analysis, part II

Analogously to [16, Lemma 2.2], we obtain:

Theorem 19

(The Weyl law) For the counting function N(r, H) defined as in Theorem 18,

$$\begin{aligned} N(r,H) = \frac{\textrm{rk}(E)\, {\text {vol}}({\mathcal {O}})}{(4 \pi )^{(\dim {\mathcal {O}}) /2} \Gamma ((\dim {\mathcal {O}}) /2+1)} r^{(\dim {\mathcal {O}})/2} + o(r^{(\dim {\mathcal {O}}) /2}), \quad r \rightarrow \infty . \end{aligned}$$

(32)

Proof

The Weyl law for $\Delta _E$ from Theorem 18 and the compactness of $D_1 \cdot (\Delta _E - \lambda \cdot {\text {Id}})^{-1}$ implies (32) [12, I, Corollary 8.5]. $\square$

We need to establish an auxiliary result about smoothing operators. The proof of the following lemma literally repeats [16, Proposition 2.5]:

Lemma 20

Let

$$\begin{aligned} A: L^2({\mathcal {O}}, E) \rightarrow L^2({\mathcal {O}}, E) \end{aligned}$$

be an integral operator with a smooth kernel K; denote by $d \mu (x)$ the Riemanian measure on ${\mathcal {O}}$. Then A is a trace class operator and

$$\begin{aligned} {\text {Tr}}(A) = \int _{\mathcal {O}}{\text {tr}}\, K(x,x) d\mu (x). \end{aligned}$$

Proof

The proof generalizes [11, Chapter VII, $\S$1]. Let $\{\varphi _j\}_{j \in {\mathbb {N}}}$ be an orthonormal basis of $L^2({\mathcal {O}}, E)$ consisting of eigensections of $\Delta _E$ with eigenvalues $0 \le \lambda _1 \le \lambda _2 \le \ldots \rightarrow \infty$. We can expand K in the orthonormal basis as

$$\begin{aligned} K(x,y) = \sum _{i,j=1}^\infty a_{i,j} \phi _i(x) \otimes \phi ^*_j(y), \end{aligned}$$

(33)

where $a_{i,j} = \langle A \phi _i, \phi _j\rangle$. Note that

$$\begin{aligned} (1+\lambda _i+\lambda _j)^N a_{i,j} = \langle (I + \Delta _E \otimes I + I \otimes \Delta _E)^N A, \phi _i \otimes \phi _j^*\rangle , \end{aligned}$$

hence for every $N \in {\mathbb {N}}$, there exists $C_N$ such that

$$\begin{aligned} |a_{i,j}| \le C_N (1+\lambda _i+\lambda _j)^{-N} \end{aligned}$$

for any $i,j \in {\mathbb {N}}$. Together with Theorem 18 it implies that the right hand side of (33) converges in the $C^\infty$-topology.

Definition 24

Define $P_{i,j}$, $i,j = 1, \ldots , \infty$ to be the integral operator with kernel $\phi _i \otimes \phi _j^*$. Put

$$\begin{aligned} A_1 = \sum _{i,j=1}^\infty a_{i,j} (1+\lambda _j)^n P_{i,j}. \end{aligned}$$

Definition 25

Define $P_j$, $j = 1, \ldots , \infty$ to be the orthogonal projection of $L^2({\mathcal {O}}, E)$ onto ${\mathbb {C}}\phi _j$. Put

$$\begin{aligned} A_2 = \sum _{j=1}^\infty (1+\lambda _j)^{-n} P_{j}. \end{aligned}$$

By Theorem 18, both $A_1$ and $A_2$ are Hilbert-Schmidt operators, hence $A = A_1 A_2$ is of trace class; moreover, by (33):

$$\begin{aligned} {\text {Tr}}\, A = \sum _{i=1}^\infty a_{i,i} = \sum _{i,j=1}^\infty a_{i,j} \int _{{\mathcal {O}}} \langle \phi _i(x), \phi _i(x) \rangle d\mu (x) = \int _{{\mathcal {O}}} {\text {tr}}\, K(x,x) d\mu (x). \end{aligned}$$

$\square$

Now we apply this result to $\varphi (H^{1/2})$, where $\varphi \in PW({\mathbb {C}})$ and $\varphi$ is even (or $\varphi (\cdot ) = e^{- t (\cdot )^2}$). Let $K_\varphi (x,y)$ be the kernel of $\varphi (H^{1/2})$. Then by Lemma 20, $\varphi (H^{1/2})$ is a trace class operator, and we have

$$\begin{aligned} {\text {Tr}}\, \varphi (H^{1/2}) = \int _{\mathcal {O}}{\text {tr}}\, K_\varphi (x,x) d \mu (x). \end{aligned}$$

(34)

Moreover, the following lemma holds:

Lemma 21

Let $\varphi \in PW({\mathbb {C}})$ be even (or $\varphi (\cdot ) = e^{- t (\cdot )^2}$). Then we have

$$\begin{aligned} \sum _{\lambda \in \text {spec}(H)} m(\lambda )\varphi (\lambda ^{1/2}) = \int _{\mathcal {O}}{\text {tr}}\, K_\varphi (x,x) d\mu (x), \end{aligned}$$

(35)

where $m(\lambda )$ is the multiplicity of $\lambda$.

Proof

By Lidskii’s theorem [6, Theorem 8.4], the trace is equal to the sum of the eigenvalues of $\varphi (H^{1/2})$, counted with their algebraic multiplicities. One can show that $\varphi (H^{1/2})$ leaves the decomposition (18) invariant and that $\varphi (H^{1/2})|_{V_k}$, has the unique eigenvalue $\varphi (\lambda _k^{1/2})$. Applying Lidskii’s theorem and (34), we get Lemma 21. $\square$

7 The Selberg trace formula

7.1 The wave equation

In this subsection we give a description of the kernel $K_\varphi$ of the smoothing operator $\varphi (H^{1/2})$ in terms of the solution of the wave equation. Let ${\mathcal {O}}= \Gamma \backslash {\mathbb {H}}^n$ be a hyperbolic orbifold; further denote $G = \text {SO}_0(1,n)$ and $K=\text {SO}(n)$. Let $\Gamma _0$ be as in Lemma 13, then $\Gamma _0 \backslash G/K$ is a manifold. Let $\rho : \Gamma \rightarrow \text {GL}({\mathbb {C}}^n)$ be a finite-dimensional representation of $\Gamma$, and let

$$\begin{aligned} E = \Gamma \backslash (G/K \times {\mathbb {C}}^n) \rightarrow {\mathcal {O}}\end{aligned}$$

be the associated vector orbibundle. Let $\rho _0$ be the restriction of $\rho$ to $\Gamma _0$, and denote by

$$\begin{aligned} E_0 = \Gamma _0 \backslash (G/K \times {\mathbb {C}}^n) \rightarrow \Gamma _0 \backslash G/K \end{aligned}$$

the associated vector bundle. Note that every $f \in C^\infty ({\mathcal {O}}, E)$ can be pulled back to $f_0 \in C^\infty (\Gamma _0 \backslash G/K, E_0)$ as well as every $(\Gamma _0 \backslash \Gamma )$-invariant $f_0 \in C^\infty (\Gamma _0 \backslash G/K, E_0)$ can be pushed down to $f \in C^\infty ({\mathcal {O}}, E)$.

Proposition 22

Denote by $||\cdot ||_{s; \Gamma _0 \backslash G/K}$ and $||\cdot ||_{s; {\mathcal {O}}}$ the s-Sobolev norms on $H^s(\Gamma _0 \backslash G/K, E_0)$ and $H^s({\mathcal {O}}, E)$, respectively. Then there exist $C, c>0$ such that for any f and $f_0$ as above, the following inequality holds:

$$\begin{aligned} c\cdot ||f_0 ||_{s; \Gamma _0 \backslash G/K} \le ||f ||_{s; {\mathcal {O}}} \le C \cdot ||f_0 ||_{s; \Gamma _0 \backslash G/K} \end{aligned}$$

Proof

Follows from the fact that $\Gamma _0 \backslash G/K$ is a finite covering of ${\mathcal {O}}$. $\square$

Consider the wave equation:

$$\begin{aligned} \left( \partial ^2 /\partial t^2 + H \right) u = 0, \quad u(0,x; f) = f(x), \quad u_t(0, x; f) = 0 \end{aligned}$$

(36)

for $u(t,x;f) \in C^\infty ({\mathbb {R}}\times {\mathcal {O}}, E)$.

Lemma 23

For each $f \in C^\infty ({\mathcal {O}}, E)$, there is a unique solution $u(t,x;f) \in C^\infty ({\mathbb {R}}\times {\mathcal {O}}, E)$ of the wave equation (36). Moreover for every $T>0$ and $s \in {\mathbb {R}}$, there exists $C > 0$ such that for every $f \in C^\infty ({\mathcal {O}}, E)$,

$$\begin{aligned} || u(t,\cdot ;f ) ||_{s;{\mathcal {O}}} \leqslant C || f||_{s;{\mathcal {O}}}, \quad |t|\leqslant T \end{aligned}$$

(37)

Proof

The proof follows from [16, Proposition 3.1]: first pull-back the wave equation to $C^\infty ({\mathbb {R}}\times \Gamma _0 \backslash G/ K, E_0)$ and solve it there. The solution of the pulled-back wave equation satisfies [16, (3.2)], that is the estimates of type (37), but for Sobolev norms $||~\cdot ~||_{s; \Gamma _0 \backslash G/K}$ instead of $|| \cdot ||_{s;{\mathcal {O}}}$. Moreover, it is invariant under $\Gamma _0 \backslash \Gamma$, because the pull-back of the initial conditions is invariant under $\Gamma _0 \backslash \Gamma$. Second, push the solution down to $C^\infty ({\mathcal {O}}, E)$ and use Proposition 22. $\square$

Lemma 24

Let $\varphi \in PW({\mathbb {C}})$ (or $\varphi (\cdot ) = e^{- t (\cdot )^2}$) and $\widehat{\varphi }$ be the Fourier transform of $\varphi |_{\mathbb {R}}$. Then for every $f \in C^\infty ({\mathcal {O}}, E)$, we have

$$\begin{aligned}{}[\varphi (H^{1/2}) f](\cdot ) = \frac{1}{\sqrt{2\pi }} \int _{\mathbb {R}}\widehat{\varphi }(t) u(t,\cdot ;f) dt. \end{aligned}$$

(38)

Proof

We follow [16, Proposition 3.2]. Let $\Gamma _{b,d}$ be as in Definition 22, choose $c>0$ such that

$$\begin{aligned} \text {spec}(H+c) \subset \{z \in {\mathbb {C}}: \text {Re}\,z > 0\}. \end{aligned}$$

For $\sigma >0$, define the operator $\cos (tH^{1/2}) e^{-\sigma (H+c)}$ by:

$$\begin{aligned} \cos (tH^{1/2}) e^{-\sigma (H+c)}:= \frac{i}{2\pi } \int _{\Gamma _{b,d}} \cos (t \lambda ) e^{-\sigma (\lambda ^2+c)} (H^{1/2}-\lambda )^{-1} d \lambda . \end{aligned}$$

Note that for $f \in C^\infty ({\mathcal {O}}, E)$,

$$\begin{aligned} \left( \cos (tH^{1/2}) e^{-\sigma (H+c)} f\right) (x) - u(t,x;f) \end{aligned}$$

is the unique solution of the wave equation (36) with initial condition $e^{-\sigma (H+c)} f-f$; the proof is by substitution and does not change if ${\mathcal {O}}$ is an orbifold. The rest of the proof is to show that

$$\begin{aligned} \frac{1}{\sqrt{2 \pi }} \int _{{\mathbb {R}}} \widehat{\varphi }(t) \cos (tH^{1/2}) e^{-\sigma (H+c)} f \; dt = \frac{i}{2 \pi } \int _{\Gamma _{b,d}} \varphi (\lambda ) e^{-\sigma (\lambda ^2+c)} (H^{1/2}-\lambda )^{-1} f \; d \lambda ,\nonumber \\ \end{aligned}$$

(39)

and that the right hand side of (39) converges to $\varphi (H^{1/2}) f$ as $\sigma \rightarrow 0$, whereas the left hand side converges to $(2\pi )^{-1} \int _{{\mathbb {R}}} \hat{\varphi }(t) u(t,\cdot ;f) \, dt$. Proof of convergence is analogous to the manifold case. $\square$

Now we would like to lift the wave equation once again, but now to G/K. Let

$$\begin{aligned} \widetilde{E}:= (G/K) \times {\mathbb {C}}^n \rightarrow G/K \end{aligned}$$

be a lift of E to G/K and let

$$\begin{aligned} \widetilde{H}: C^\infty (G/K, \widetilde{E}) \rightarrow C^\infty (G/K, \widetilde{E}) \end{aligned}$$

be the lift of H to G/K. Let $\widetilde{u}(u,\widetilde{x};f)$ and $\widetilde{f}$ be the pull-back to G/K of u(t, x; f) and f, respectively. Then the following holds:

$$\begin{aligned} \left( \partial ^2 /\partial t^2 + \widetilde{H} \right) \widetilde{u} = 0, \quad \widetilde{u}(0,x; f) = f(x), \quad \widetilde{u}_t(0, x; f) = 0 \end{aligned}$$

(40)

As in [16, (3.15)], with the help of the finite propagation speed argument one can show that it does not matter if:

• 1. either we solve the wave equation (36) on ${\mathcal {O}}$ and then pull the solution back to G/K,

• 2. or we first pull back the initial condition to G/K and then solve the wave equation (40).

Let d(x, y) denote the geodesic distance of $x,y \in G/K$. For $\delta > 0$ define $U_\delta := \{ (x,y)\in G/K \times G/K: d(x,y) < \delta \}$.

Lemma 25

There exists $\delta > 0$ and $H_\varphi \in C^\infty (G/K \times G/K, \text {Hom} (\tilde{E},\tilde{E}))$ with ${\text {supp}}\, H_\varphi \subset U_\delta$, such that for all $\psi \in C^\infty (G/K,\tilde{E})$ we have

$$\begin{aligned} \frac{1}{\sqrt{2\pi }} \int _{\mathbb {R}}\widehat{\varphi }(t) u(t,\tilde{x},\psi ) dt = \int _{G/K} H_\varphi (\tilde{x},\tilde{y}) (\psi (\tilde{y})) d \tilde{y}. \end{aligned}$$

Proof

The proof follows [16, Proposition 3.3] and is based on the finite propagation speed argument, that is valid for orbifolds as well. $\square$

Using Lemmas 24 and 25 we obtain

$$\begin{aligned}{}[\varphi (H^{1/2})f ](\tilde{x}) = \int _{G/K} H_\varphi (\tilde{x},\tilde{y})(\tilde{f}(\tilde{y}))\, d\tilde{y} \end{aligned}$$

(41)

for all $f\in C^\infty (X,E)$. Let $F \subset M$ be a fundamental domain for the action of $\Gamma$ on G/K, and let

$$\begin{aligned} R_\gamma : \widetilde{E} \mapsto \widetilde{E} \end{aligned}$$

(42)

be the induced bundle map for $\gamma \in \Gamma$. Note that

$$\begin{aligned} \widetilde{f}(\gamma \widetilde{y}) = R_\gamma (\widetilde{f}(\widetilde{y})), \quad \gamma \in \Gamma . \end{aligned}$$

(43)

Arguing as in [16] by rewriting $\int _{G/K}$ as $\sum _{\gamma \in \Gamma } \int _{\gamma F}$ in (41) and using (43), one can show that the kernel $K_\varphi$ of $\varphi (H^{1/2})$ is given by

$$\begin{aligned} K_\varphi (x,y) = \sum _{\gamma \in \Gamma } H_\varphi (\tilde{x}, \gamma \tilde{y})\circ R_\gamma , \end{aligned}$$

(44)

where $\widetilde{x}$, $\widetilde{y}$ are any lifts of x and y to F. Together with Lemma 21 we obtain an analogue of [16, Proposition 3.4]:

Lemma 26

Let $\varphi \in PW$ be even (or $\varphi (\cdot ) = e^{- t (\cdot )^2}$). Then we have

$$\begin{aligned} \sum _{\lambda \in \text {spec}(H)} m(\lambda ) \varphi (\lambda ^{1/2}) = \sum _{\gamma \in \Gamma } \int _F {\text {tr}}(H_\varphi (\tilde{x}, \gamma \tilde{x}) \circ R_\gamma ) d\tilde{x}. \end{aligned}$$

7.2 The twisted Bochner–Laplace operator

In this section we follow [16, Sect. 4] to introduce the twisted non-selfadjoint Laplacian $\Delta _{E, \chi }^\#$. Let ${\mathcal {O}}$ be a hyperbolic odd-dimensional orbifold ${\mathcal {O}}= \Gamma \backslash G/K$, where $G = \text {SO}_0(1,2n+1)$, $K = \text {SO}(2n+1)$.

Let $\chi : \Gamma \rightarrow GL(V_\chi )$ be a finite-dimensional representation of $\Gamma$, and let $F \rightarrow {\mathcal {O}}$ be the associated orbibundle over ${\mathcal {O}}$; let $\nabla ^F$ be a canonical flat connection on F. Let E be a Hermitian vector orbibundle over ${\mathcal {O}}$ with a Hermitian connection $\nabla ^E$.

Definition 26

We equip $E \otimes F$ with a product connection $\nabla ^{E \otimes F}$, defined by

$$\begin{aligned} \nabla _Y^{E \otimes F}:= \nabla _Y^E \otimes 1 + 1 \otimes \nabla _Y^F \end{aligned}$$

for $Y \in C^\infty (G/K, T(G/K))$.

Definition 27

The twisted connection Laplacian $\Delta _{E, \chi }^\#$ associated to $\nabla ^{E \otimes F}$ is given by

$$\begin{aligned} \Delta _{E, \chi }^\#:= - {\text {Tr}}(\nabla ^{E \otimes F})^2, \end{aligned}$$

where $(\nabla ^{E \otimes F})^2$ is the invariant second covariant derivative.

Definition 28

Denote by $\Delta _E$ the Bochner–Laplace operator $(\nabla ^E)^* \nabla ^E$.

Remark 7

The principal symbol of $\Delta _{E, \chi }^\#$ is given by

$$\begin{aligned} \sigma (\Delta _{E, \chi }^\#)(x,\xi ) = || \xi ||_x^2 \cdot {\text {Id}}_{(E\otimes F)_x}. \end{aligned}$$

Let $\widetilde{E}$ and $\widetilde{F}$ be the pullback to G/K of E and F, respectively. Note that

$$\begin{aligned} C^\infty (G/K, \widetilde{E} \otimes \widetilde{F}) \cong C^\infty (G/K, \widetilde{E}) \otimes V_\chi . \end{aligned}$$

Let $\widetilde{\Delta }^\#_{E, \chi }$ and $\widetilde{\Delta }_E$ be the lift of $\Delta ^\#_{E,\chi }$ and $\Delta _E$ to G/K, respectively.

Note that the operator $\widetilde{\Delta }_{E, \chi }^\#$ splits as follows:

$$\begin{aligned} \widetilde{\Delta }_{E, \chi }^\# = \widetilde{\Delta }_E \otimes {\text {Id}}, \end{aligned}$$

(45)

where $\widetilde{\Delta }_E \otimes {\text {Id}}$ acts on $C^\infty (G/K, \widetilde{E}) \otimes V_\chi$. Then for any $\psi \in C_c^\infty (G/K, \widetilde{E})$, the unique solution of the equation

$$\begin{aligned} (\partial ^2/\partial t^2 + \widetilde{\Delta }_{E, \chi }^\#) u(t,\cdot ;\psi ) = 0, \quad u(0, \cdot ; \psi )=\psi , \, u_t(0, \cdot ;\psi )=0 \end{aligned}$$

splits as well and is given by

$$\begin{aligned} u(t, \cdot ;\psi ) = \left( \cos (t (\widetilde{\Delta }_E)^{1/2}) \otimes {\text {Id}}\right) \psi (\cdot ), \end{aligned}$$

where $\cos (t (\widetilde{\Delta }_E)^{1/2})$ is defined by the spectral theorem. Let $\varphi \in PW({\mathbb {C}})$ be even and let $k_\varphi (\tilde{x},\tilde{y})$ be the kernel of

$$\begin{aligned} \varphi \left( (\widetilde{\Delta }_E)^{1/2} \right) = \frac{1}{\sqrt{2\pi }} \int _{\mathbb {R}}\widehat{\varphi }(t) \cos (t (\widetilde{\Delta }_E)^{1/2}) \, dt. \end{aligned}$$

Then $H_\varphi$ from Lemma 25 is given by $H_\varphi (\tilde{x},\tilde{y}) = k_\varphi (\tilde{x}, \tilde{y}) \otimes {\text {Id}}$. Then it follows from (44) that the integral kernel of the operator $\varphi \left( (\Delta _{E,\chi }^\#)^{1/2}\right)$ is given by

$$\begin{aligned} K_\varphi (x,y)=\sum _{\gamma \in \Gamma } k_\varphi (\tilde{x},\gamma \tilde{y})\circ (R_\gamma \otimes \chi (\gamma )). \end{aligned}$$

Lemma 26 implies the following Lemma:

Lemma 27

Let F be a flat vector orbibundle over ${\mathcal {O}}$, associated to a finite-dimensional complex representation $\chi : \Gamma \rightarrow GL(V_\chi )$. Let $\Delta _{E, \chi }^\#$ be the twisted connection Laplacian from Definition 27 acting in $C^\infty ({\mathcal {O}}, E \otimes F)$. Let $\varphi \in PW({\mathbb {C}})$ be even (or $\varphi (\cdot ) = e^{- t (\cdot )^2}$) and denote by $k_\varphi (\tilde{x},\tilde{y})$ the kernel of $\varphi \left( (\widetilde{\Delta }_E)^{1/2} \right)$. Then we have

$$\begin{aligned} \sum _{\lambda \in spec(\Delta _{E,\chi }^\#)} m(\lambda )\varphi (\lambda ^{1/2}) = \sum _{\gamma \in \Gamma } {\text {tr}}\, \chi (\gamma ) \int _F {\text {tr}}(k_\varphi (\tilde{x},\gamma \tilde{y})\circ R_\gamma ) \, d\tilde{x}. \end{aligned}$$

7.3 Locally symmetric spaces and the pre-trace formula

In this subsection we apply Lemma 27 to the case when E is a locally homogeneous orbibundle.

Let $\chi : \Gamma \rightarrow GL(V_\chi )$ be a finite-dimensional (possibly non-unitary) complex representation and let $F \rightarrow {\mathcal {O}}$ be the associated flat vector bundle over ${\mathcal {O}}$ as in previous subsection. Let $\nu : K \rightarrow \text {GL}(V_\nu )$ be a unitary representation of K and let $E_\nu \rightarrow {\mathcal {O}}$ be the locally homogeneous orbibundle as in Definition 12.

Denote by $\Delta _{E_\nu , \chi }^\#$ be the twisted connection Laplacian acting on $C^\infty ({\mathcal {O}}, E_\nu \otimes F)$ as in Definition 27. To simplify notations, denote:

$$\begin{aligned} \Delta _{\nu , \chi }^\#:= \Delta _{E_\nu , \chi }^\#. \end{aligned}$$

(46)

Let $\widetilde{\Delta _\nu }$ be as in Definition 5. We are now interested in rewriting $k_\varphi$ in a different way with respect to the information that $E_\nu$ is a locally homogeneous orbibundle. Note that $\varphi (\widetilde{\Delta }_\nu ^{1/2})$ is a G-invariant operator. With respect to the isometry (2) it can be identified with a compactly supported $C^\infty$ function

$$\begin{aligned} h_\varphi : G \rightarrow \text {End}\, V_\nu , \end{aligned}$$

such that

$$\begin{aligned} h_\varphi (k_1 g k_2) = \nu (k_1) \circ h_\varphi (g) \circ \nu (k_2), \quad k_1, k_2 \in K. \end{aligned}$$

Then $\varphi (\widetilde{\Delta }_\nu ^{1/2})$ acts by convolution:

$$\begin{aligned} \left( \varphi (\widetilde{\Delta }_\nu ^{1/2}) f \right) (g_1) = \int _G h_\varphi (g_1^{-1} g_2)(f(g_2)) dg_2, \end{aligned}$$

(47)

and the kernel $K_\varphi$ of $\varphi ((\Delta _{\nu ,\chi }^{\#})^{1/2})$ is given by

$$\begin{aligned} K_\varphi (g_1 K, g_2 K) = \sum _{\gamma \in \Gamma } h_\varphi (g_1^{-1} \gamma g_2) \otimes \chi (\gamma ). \end{aligned}$$

(48)

By Lemma 27, we get

$$\begin{aligned} \sum _{\lambda \in \text {spec}(\Delta _{E, \chi }^\#)} m(\lambda ) \varphi (\lambda ^{1/2}) = \sum _{\gamma \in \Gamma } {\text {tr}}\, \chi (\gamma ) \, \int _{\Gamma \backslash G} {\text {tr}}\, h_\varphi (g^{-1} \gamma g) d\dot{g}. \end{aligned}$$

(49)

Definition 29

For $\gamma \in \Gamma$, denote by $\{ \gamma \}_\Gamma$ its $\Gamma$-conjugacy class.

Definition 30

For $\gamma \in \Gamma$, denote by $\Gamma _\gamma$ and $G_\gamma$ the centralizers of $\gamma$ in $\Gamma$ and G, respectively.

Collect the terms in the right hand side of (49) according to their conjugacy classes. Separating $\{ e\}_\Gamma$, we obtain a pre-trace formula.

Proposition 28

[Pre-trace formula] For all even $\varphi \in PW({\mathbb {C}})$ (or $\varphi (\cdot ) = e^{- t (\cdot )^2}$), we have:

$$\begin{aligned} \begin{aligned} \sum _{\lambda \in \text {spec}(\Delta _{E, \rho }^\#)} m(\lambda ) \varphi (\lambda ^{1/2}) =&\dim (V_\chi ) {\text {vol}}(\Gamma \backslash G / K)\, {\text {tr}}\, h_\varphi (e) \\&+ \sum _{\{\gamma \}_\Gamma \ne \{e\}} {\text {tr}}\, \chi (\gamma ) {\text {vol}}(\Gamma _\gamma \backslash G_\gamma ) \int _{G_\gamma \backslash G} {\text {tr}}\, h_\varphi (g^{-1} \gamma g) d\dot{g}. \end{aligned} \end{aligned}$$

(50)

We restrict ourselves to the case $G = \text {SO}_0(1,2n+1)$, $K=\text {SO}(2n+1)$. In order to make the formula more explicit, we need to evaluate the orbital integrals $\int _{G_\gamma \backslash G} {\text {tr}}\, h_\varphi (g^{-1} \gamma g) d\dot{g}$ on the right hand side of (50), that will be done in Sects. 7.4 and 7.5 for hyperbolic and for elliptic $\gamma$, respectively.

7.4 Orbital integrals for hyperbolic elements

Definition 31

For a hyperbolic $\gamma \in \Gamma$, define its primitive element as an element $\gamma _0 \in \Gamma$ such that $\gamma = \gamma _0^k$, and for any $\gamma '_0 \in \Gamma$ such that $\gamma = (\gamma '_0)^n$, it follows that $n\le k$.

A primitive element $\gamma _0$ is not necessarily unique. It is defined up to

$$\begin{aligned} \Gamma _\gamma ^1:= \Gamma \cap G_\gamma ^1, \end{aligned}$$

where $G_\gamma ^1$ is the maximal compact subgroup of $G_\gamma$.

Remark 8

Note that

$$\begin{aligned} {\text {vol}}(\Gamma _\gamma \backslash G_\gamma ) = l(\gamma _0) / |\Gamma _\gamma ^1|. \end{aligned}$$

Let $\pi _{\sigma , \lambda }$ be as in (10), and denote by $\Theta _{\sigma ,\lambda }$ the character of $\pi _{\sigma , \lambda }$. For hyperbolic $\gamma$, we slightly modify [25, Theorem 6.7] in order to get the following lemma:

Lemma 29

Let $\gamma \in \Gamma$ be a hyperbolic element. Then the following holds:

$$\begin{aligned} \begin{aligned}&{\text {vol}}(\Gamma _\gamma \backslash G_\gamma ) \, \int _{G_\gamma \backslash G} {\text {tr}}\, h_\varphi (g^{-1} \gamma g) d\dot{g} \\&\quad = \frac{ l(\gamma _0)}{2 \pi |\Gamma _\gamma ^1| D(\gamma )} \cdot \sum _{\sigma \in \widehat{SO(2n)}} \overline{{\text {tr}}\, \sigma (\gamma )} \int _{{\mathbb {R}}} \Theta _{\sigma , \lambda }(h_\varphi )\cdot e^{-il(\gamma ) \lambda } d \lambda , \end{aligned} \end{aligned}$$

where

$$\begin{aligned} D(\gamma ) = e^{-n l(\gamma )} \Big |\det (\text {Ad}(m_\gamma a_\gamma )|_{\bar{\mathfrak {n}}} - {\text {Id}})\Big | \end{aligned}$$

(51)

and $l(\gamma _0)$ is from Definition 15, $m_\gamma$ and $a_\gamma$ is from Lemma 6.

Proof

The modification of [25, Theorem 6.7] is as follows: we take ${\text {vol}}(\Gamma _\gamma \backslash G_\gamma )$ and change it to $l(\gamma _0) / |\Gamma _\gamma ^1|$ by Remark 8. $\square$

7.5 Orbital integrals for elliptic elements

In this subsection we evaluate the orbital integrals

$$\begin{aligned} E_\gamma (h_\varphi ):= \int _{G_\gamma \backslash G} {\text {tr}}\, h_\varphi (g^{-1} \gamma g) d\dot{g} \end{aligned}$$

(52)

for $\gamma \in \Gamma$ elliptic. We may assume $\gamma$ is as in (11). Note that in general $\gamma$ is not a regular element, e.g. if $k>1$ or some of the angles $\phi _{i}$ coincide; see Definition 17. For further use we want to approximate $\gamma$ by a sequence of regular elements $\gamma _j$, $j \in {\mathbb {N}}$:

$$\begin{aligned} \gamma _j = {\text {diag}} \left( \left( {\begin{matrix} 1 &{} 0 \\ 0 &{} 1 \end{matrix}} \right) , R_{\varepsilon _{2;j}}, \ldots , R_{\varepsilon _{n+1;j}} \right) \end{aligned}$$

where sequences $\varepsilon _{i;j} \in {\mathbb {R}}$, $i=2, \ldots , n+1$, $j \in {\mathbb {N}}$ are chosen in the following way: for fixed j, all $\varepsilon _{i;j}$ are pairwise distinct, that is $\varepsilon _{i_1;j} \ne \varepsilon _{i_2;j}$ if $i_1 \ne i_2$, and

$$\begin{aligned} \begin{aligned} \lim _{j \rightarrow \infty } \varepsilon _{i;j} = 0, \quad i \le k,\\ \lim _{j \rightarrow \infty } \varepsilon _{i;j} = \phi _i, \quad i > k.\\ \end{aligned} \end{aligned}$$

The strategy for the subsection is the following: first we recall how to calculate $E_{\gamma _j}(h_\varphi )$, second we apply a certain element of the symmetric algebra $S(\mathfrak {b}_{\mathbb {C}})$, and set $\varepsilon = 0$ to obtain $E_{\gamma }(h_\varphi )$. To calculate $E_{\gamma _j}(h_\varphi )$, we combine together an adjusted version of [9, Theorem 13.1] together with the following proposition:

Proposition 30

Let $\eta (\gamma ) \in {\text {Spin}}(1,2n+1)$ be a lift of $\gamma$. Let $\pi$ be a canonical projection $\pi : {\text {Spin}}(1,2n+1) \rightarrow \text {SO}_0(1,2n+1)$. For $\gamma \in G$, denote ${\text {Spin}}(1,2n+1)_{\eta (\gamma )}$ to be the centralizer of $\eta (\gamma )$ in ${\text {Spin}}(1,2n+1)$. Define

$$\begin{aligned}&\pi ^* h_\varphi : {\text {Spin}}(1,2n+1) \mapsto {\mathbb {C}}, \\&\pi ^* h_\varphi (x):= h_\varphi (\pi (x)). \end{aligned}$$

Then

$$\begin{aligned} \begin{aligned}&\int _{{\text {Spin}}(1,2n+1)_{\eta (\gamma )} \backslash {\text {Spin}}(1,2n+1) } \pi ^* h_\varphi (g^{-1} \eta (\gamma ) g) d g \\&\qquad \quad =\int _{\text {SO}_0(1,2n+1)_\gamma \backslash \text {SO}_0(1,2n+1)} h_\varphi (g^{-1} \gamma g) d g. \end{aligned} \end{aligned}$$

(53)

Proof

Note that ${\text {Spin}}(1,2n+1)_{\eta (\gamma )}$ is a 2-fold covering of $\text {SO}_0(1,2n+1)_\gamma$, hence

$$\begin{aligned} \pi \left( {\text {Spin}}(1,2n+1)_{\eta (\gamma )} \backslash {\text {Spin}}(1,2n+1)\right) \cong \text {SO}_0(1,2n+1)_\gamma \backslash \text {SO}_0(1,2n+1). \end{aligned}$$

Moreover,

$$\begin{aligned} h_\varphi (\pi (g)^{-1} \gamma \pi (g)) = h_\varphi (\pi (g^{-1} \eta (\gamma )g)) = \pi ^* h_\varphi (g^{-1} \eta (\gamma ) g). \end{aligned}$$

and hence

$$\begin{aligned} \begin{aligned}&\int _{ {\text {Spin}}(1,2n+1)_{\eta (\gamma )} \backslash {\text {Spin}}(1,2n+1)} \pi ^* h_\varphi \left( g^{-1} \eta (\gamma ) g\right) d g \\&\quad =\int _{{\text {Spin}}(1,2n+1)_{\eta (\gamma )} \backslash {\text {Spin}}(1,2n+1)} h_\varphi \left( \pi (g)^{-1} \gamma \pi (g)\right) d g \\&\quad =\int _{\pi ({\text {Spin}}(1,2n+1)_{\eta (\gamma )} \backslash {\text {Spin}}(1,2n+1))} h_\varphi \left( g^{-1} \gamma g \right) d g \\&\quad =\int _{SO_0(1,2n+1)_{\gamma } \backslash SO_0(1,2n+1)} h_\varphi \left( g^{-1} \gamma g \right) d g, \end{aligned} \end{aligned}$$

(54)

that proves the proposition. $\square$

Lemma 31

[9, Theorem 13.1] The orbital integral $E_{\gamma _j}(h_{\varphi })$ can be expressed as

$$\begin{aligned} E_{\gamma _j}(h_\varphi ) = C\cdot \sum _{\sigma \in \widehat{SO(2n)}} \int _{{\mathbb {R}}} \sum _{s \in W} \det (s) \left( \xi _{-s(\Lambda (\sigma ) + \rho _M)}\otimes e^{-\sqrt{-1} \lambda }(\gamma _j) \right) \cdot \Theta _{\sigma ,\lambda } (h_\varphi ) d \lambda ,\nonumber \\ \end{aligned}$$

(55)

where $C \in {\mathbb {R}}{\setminus }\{0\}$ does not depend on $\gamma _j$. The sum in (55) is finite, because $h_\varphi$ is K-finite. Above $\det (s)$ denotes the determinant of s, and for every $\gamma _j = \left( {\begin{matrix} \gamma ^{1} &{} 0 \\ 0 &{} \gamma ^{2}\end{matrix}} \right)$, where $\gamma ^{1} = \left( {\begin{matrix} e^t &{} 0 \\ 0 &{} e^{-t} \end{matrix}} \right) \in \text {SO}_0(1,1)$ and $\gamma ^{2} \in \text {SO}(2n)$, the tensor product acts as:

$$\begin{aligned} \xi _{-s(\Lambda (\sigma ) + \rho _M)}\otimes e^{-\sqrt{-1} \lambda }(\gamma _j):= \left[ \xi _{-s(\Lambda (\sigma ) + \rho _M)}(\gamma ^2)\right] \cdot [e^{-\sqrt{-1} \lambda t}], \end{aligned}$$

and W is the Weyl group of $\Delta (\mathfrak {m}_{\mathbb {C}}, \mathfrak {b}_{\mathbb {C}})$ as in (6).

Remark 9

For $\gamma _j$ elliptic, $\gamma _j^{1} = \left( {\begin{matrix} 1 &{} 0 \\ 0 &{} 1 \end{matrix}} \right)$ and hence

$$\begin{aligned} \xi _{-s(\Lambda (\sigma ) + \rho _M)}\otimes e^{-\sqrt{-1} \lambda }(\gamma _j) \end{aligned}$$

does not depend on $\lambda$.

Remark 10

Our notation differs from [9, Theorem 13.1], namely h, f and $F^T_f(h)$ corresponds to our $\gamma _\epsilon$, $h_\varphi$ and $E_{\gamma _j}(h_\varphi )$, respectively; $F^T_f(h)$ is defined in [9, p. 349].

Definition 32

For $\alpha \in \Delta ^+({\mathfrak {s}}{\mathfrak {o}}_0(1,2n+1)_{\mathbb {C}}, \mathfrak {h}_{{\mathbb {C}}})$, denote by $H_\alpha$ its coroot.

Without loss of generality assume that all $\phi _i$ from (11) are different, then the stabilizer $G_\gamma$ of $\gamma$ is equal to $\text {SO}(2)^k \times SO_0(1, 2k-1)$. The root system for $G_\gamma$ can be written as

$$\begin{aligned} \Delta _\gamma (\mathfrak {g}_{ {\mathbb {C}}}, \mathfrak {h}_{\mathbb {C}}) = \{\pm e_i \pm e_j, 1 \leqslant i < j \leqslant k \}. \end{aligned}$$

We can choose an ordering such that

$$\begin{aligned} \Delta ^+_\gamma (\mathfrak {g}_{ {\mathbb {C}}}, \mathfrak {h}_{\mathbb {C}}) = \{\pm (e_i + e_j), 1 \leqslant i < j \leqslant k \}. \end{aligned}$$

Lemma 32

[23, (5.2)] There exists $M_\gamma \in {\mathbb {R}}{\setminus } \{0\}$ such that

$$\begin{aligned} E_\gamma (h_\varphi ) = M_\gamma \cdot \lim _{\gamma _j \rightarrow \gamma } \left( \prod _{\alpha \in \Delta ^+_\gamma } H_\alpha \right) E_{\gamma _j}(h_\varphi ). \end{aligned}$$

We are ready to prove the main theorem in this subsection:

Theorem 33

There exists an even polynomial $P^\gamma _\sigma (\sqrt{-1} \lambda )$ such that

$$\begin{aligned} E_\gamma (h_\varphi ) = \sum _{\sigma \in \widehat{SO(2n)}} \int _{{\mathbb {R}}} P^\gamma _\sigma (\sqrt{-1} \lambda ) \Theta _{\sigma ,\lambda } (h_\varphi ) d \lambda . \end{aligned}$$

Proof

Theorem 33 holds with

$$\begin{aligned} P^\gamma _\sigma (\sqrt{-1} \lambda ) = \left. \prod _{\alpha \in \Delta ^+_\gamma } H_\alpha \left( \sum _{s \in W} \det (s) \left( \xi _{-s(\Lambda (\sigma ) + \rho _M)}\otimes e^{-\sqrt{-1} \lambda }(\gamma _j) \right) \right) \right| _{j \rightarrow \infty } \end{aligned}$$

(56)

by Lemmas  31 and 32. We need to show that $P^\gamma _\sigma (\sqrt{-1} \lambda )$ is an even polynomial. Note that every $\alpha \in \Delta ^+_\gamma$ is a root with $\langle \alpha , \alpha \rangle = 2$, hence

$$\begin{aligned} \begin{aligned}&\prod _{\alpha \in \Delta ^+_\gamma } H_\alpha \left( \xi _{-s(\Lambda (\sigma ) + \rho _M)}\otimes e^{-\sqrt{-1} \lambda }(\gamma _j) \right) \\&\quad =\left\{ \prod _{\alpha \in \Delta ^+_\gamma } \langle -s(\Lambda (\sigma ) + \rho _M) - \sqrt{-1} \lambda e_1, \alpha \rangle \right\} \left( \xi _{-s(\Lambda (\sigma ) + \rho _M)}\otimes e^{-\sqrt{-1} \lambda }(\gamma _j) \right) . \end{aligned} \end{aligned}$$

(57)

Let $s(\Lambda (\sigma ) + \rho _M) = \sum _{2 \leqslant i \leqslant n+1} k_i e_i$ with $\rho _M$ as in (8). For simplicity assume $k_1 = 0$ and denote by $\delta _{i,j}$ the Kronecker delta. Then

$$\begin{aligned}{} & {} \prod _{\alpha \in \Delta ^+_\gamma } \langle -s(\Lambda (\sigma ) + \rho _M) - \sqrt{-1} \lambda e_{1}, \alpha \rangle \nonumber \\{} & {} \quad = (-1)^{|\Delta _\gamma ^+|} \prod _{1 \leqslant i'< j' \leqslant k} \left[ \left\langle \sum _{2 \leqslant i \leqslant n+1} k_i e_i + \sqrt{-1} \lambda e_{1}, e_{i'} - e_{j'}\right\rangle \right. \nonumber \\{} & {} \qquad \left. \times \left\langle \sum _{2 \leqslant i \leqslant n+1} k_i e_i + \sqrt{-1} \lambda e_{1}, e_{i'} + e_{j'}\right\rangle \right] \nonumber \\{} & {} \quad = (-1)^{|\Delta _\gamma ^+|} \prod _{1 \leqslant i' < j' \leqslant k} \left[ \left( \sqrt{-1} \lambda (\delta _{i', 1} - \delta _{j', 1}) + (k_{i'}-k_{j'}) \right) \right. \nonumber \\{} & {} \qquad \left. \times \left( \sqrt{-1} \lambda (\delta _{i', 1} + \delta _{j',1}) + (k_{i'}+k_{j'}) \right) \right] . \end{aligned}$$

(58)

Note that above $\delta _{j', 1}$ is always equal to 0. Now we would like to study the dependance of (61) on $\lambda$, for this we split the product above as:

$$\begin{aligned} \prod _{1 \le i'< j' \le k} = \prod _{\begin{array}{c} i' = 1,\\ 2 \le j' \le k \end{array}} \cdot \prod _{2 \le i' < j' \le k}, \end{aligned}$$

and first notice that

$$\begin{aligned} \begin{aligned}&\prod _{2 \le i' \le j' \le k} \left( \sqrt{-1} \lambda \delta _{i', 1} + (k_{i'}-k_{j'}) \right) \cdot \left( \sqrt{-1} \lambda \delta _{i', 1} + (k_{i'}+k_{j'}) \right) \\&\quad =\prod _{2 \le i' < j' \le k} (k_{i'}^2-k_{j'}^2):= C(k) \end{aligned} \end{aligned}$$

(59)

does not depend on $\lambda$. Second,

$$\begin{aligned} \begin{aligned}&\prod _{\begin{array}{c} i' = 1,\\ 2 \le j' \le k \end{array}} \left( \sqrt{-1} \lambda \delta _{i', 1} + (k_{i'}-k_{j'}) \right) \cdot \left( \sqrt{-1} \lambda \delta _{i', 1} + (k_{i'}+k_{j'}) \right) \\&\quad =\prod _{2 \leqslant j' \le k} \left( \sqrt{-1} \lambda - k_{j'} \right) \cdot \left( \sqrt{-1} \lambda + k_{j'} \right) = - \prod _{2 \le j' \le k} (\lambda ^2+k_{j'}^2). \end{aligned} \end{aligned}$$

(60)

Putting together (58–60) gives us

$$\begin{aligned} \prod _{\alpha \in \Delta ^+_\gamma } \langle -s(\Lambda (\sigma ) + \rho _M) - \sqrt{-1} \lambda e_{1}, \alpha \rangle = (-1)^{|\Delta _\gamma ^+|+1} \cdot C(k) \cdot \prod _{2 \le j' \le k} (\lambda ^2+k_{j'}^2), \end{aligned}$$

(61)

where $(-1)^{|\Delta _\gamma ^+|+1} \cdot C(k)$ does not depend on $\lambda$.

Note that (61) is an even polynomial in $\lambda$ and by Remark 9, $\xi _{-s(\Lambda (\sigma ) + \rho _M)- \sqrt{-1} e_1 \lambda }(\gamma _\epsilon )$ does not depend on $\lambda$. Hence, (57) and (56) are even polynomials in $\lambda$ as well. $\square$

We would like to mention the resemblance of Theorem 33 to the following:

Proposition 34

[9, Theorem 13.2] There exists an even polynomial $P_{\sigma '}(\sqrt{-1} \lambda )$ such that

$$\begin{aligned} {\text {tr}}\,h_\varphi (e) = \sum _{\sigma ' \in \widehat{M}} \int _{{\mathbb {R}}} P_{\sigma '}(\sqrt{-1} \lambda ) \Theta _{\sigma , \lambda } (h_\varphi ) d \lambda . \end{aligned}$$

For further use we need to show one property of the polynomial $P_\sigma ^\gamma (\sqrt{-1} \lambda )$. Let $\sigma$ be a finite-dimensional representation of M with the highest weight

$$\begin{aligned} \Lambda (\sigma ) = \sum _{j=2}^{n+1} \lambda _j(\sigma ) e_j, \end{aligned}$$

(62)

then the highest weight of a representation $w_0 \sigma$, where $w_0$ is the non-identity element of W(A) from (9), equals

$$\begin{aligned} \Lambda (w_0 \sigma ) = \sum _{j=2}^{n} \lambda _j(\sigma ) e_j - \lambda _{n+1}(\sigma ) e_{n+1}. \end{aligned}$$

(63)

Lemma 35

The polynomial $P_\sigma ^{\gamma }$ is invariant under the action of W(A):

$$\begin{aligned} P^\gamma _\sigma (\sqrt{-1} \lambda ) = P^\gamma _{w_0 \sigma }(\sqrt{-1} \lambda ). \end{aligned}$$

Proof

Recall that $s \in W$ acts on the roots by even sign changes and the permutations. Then it follows from (62) and (63) that if $s(\Lambda (\sigma ) + \delta _M) = \sum _{2 \leqslant i \leqslant n+1} k_i e_i$ for some $k_i \in {\mathbb {Z}}$, then $s(\Lambda (\sigma ) + \delta _M) = \sum _{2 \leqslant i \leqslant n+1} \hat{k}_i e_i$, where $\hat{k}_i = - k_i$ for exactly one i and $\hat{k}_j = k_j$ for all $j \ne i$. It follows that $\hat{k}_i^2 = k_i^2$. By (61) the polynomial $P_\sigma ^\gamma (\sqrt{-1} \lambda )$ depends only on $k_i^2$ which completes the proof of Lemma 35. $\square$

8 Selberg zeta function

Let $\sigma \in \widehat{M}$, where $M=\text {SO}(2n)$.

Definition 33

The Selberg zeta function is defined as:

$$\begin{aligned} Z(s, \sigma , \chi ):= \exp \left( - \sum _{\{ \gamma \} \, \text {hyperbolic}} \frac{{\text {tr}}(\chi (\gamma )) \cdot {\text {tr}}(\sigma (m_\gamma )) \cdot e^{-(s+n) l(\gamma )}) }{ n_\Gamma \cdot |\Gamma ^1_\gamma |\cdot \det ({\text {Id}}- \text {Ad}(m_\gamma a_\gamma )|_{\mathfrak {\bar{n}}}) }\right) . \end{aligned}$$

(64)

Proposition 36

There exists a constant $c = C(\Gamma , \chi )>0$ such that $Z(s, \sigma , \chi )$ converges absolutely and locally uniformly for $\text {Re}(s)>c$.

Proof

Analogously to [22, Lemma 3.3], there exist $k,K \ge 0$ such that

$$\begin{aligned} {\text {tr}}(\chi (\gamma )) \le K e^{k l(\gamma )}. \end{aligned}$$

It follows by definition that $|\Gamma _\gamma ^1| \ge 1$, $n_\Gamma \ge 1$ and ${\text {tr}}(\sigma (m_\gamma )) \le \dim (\sigma )$. We need the following lemma to estimate the number of closed geodesics:

Lemma 37

There exists a constant $C_3 > 0$ such that for all $x \in {\mathbb {H}}^{2n+1}$, the following estimate holds:

$$\begin{aligned} \# \{\gamma \text { hyperbolic}, \gamma \in \Gamma : \rho (x, \gamma x) \le R \} \le C_3 e^{2n R}, \end{aligned}$$

(65)

where $\rho (x,y)$ denotes the hyperbolic distance between x and y.

Proof

Let $x \in {\mathbb {H}}^{2n+1}$, denote by $B_R(x)$ the hyperbolic ball around x of radius R; note that

$$\begin{aligned} {\text {vol}}(B_R(x)) \le C_2 \cdot e^{2 n R} \end{aligned}$$

for some $C_2 > 0$. Note that because $\Gamma$ is cocompact, there exists $\varepsilon > 0$ such that

$$\begin{aligned} B_\varepsilon (x) \cap \gamma B_\varepsilon (x) = \emptyset , \quad \gamma \in \Gamma ,\, \gamma \text { hyperbolic}, \, x\in {\mathbb {H}}^{2n+1}. \end{aligned}$$

Thus

$$\begin{aligned} \bigsqcup _{\gamma \in \Gamma ,\, \gamma \text { hyperbolic}, \, \rho (x, \gamma x)\le R} \gamma B_{\varepsilon } (x) \subseteq B_{R+\varepsilon }(x), \end{aligned}$$

that implies (65). $\square$

Moreover,

$$\begin{aligned} \det ({\text {Id}}- \text {Ad}(m_\gamma a_\gamma )|_{\mathfrak {\bar{n}}}) \ge (1-e^{-l(\gamma )})^n, \end{aligned}$$

hence there exists a constant $C_4$ such that for every $\gamma$ hyperbolic,

$$\begin{aligned} \frac{1}{\det ({\text {Id}}- \text {Ad}(m_\gamma a_\gamma )|_{\mathfrak {\bar{n}}})} < C_4. \end{aligned}$$

Collecting all together proves Proposition 36. $\square$

8.1 The symmetric Selberg zeta function

Let $\sigma \in \widehat{M}$ with the highest weight

$$\begin{aligned} k_2(\sigma ) e_2 + \cdots + k(\sigma )_{n+1} e_{n+1}. \end{aligned}$$

Definition 34

For $\text {Re}(s) > c$ with the constant c as in Proposition 36, we define the symmetric Selberg zeta function by

$$\begin{aligned} S(s, \sigma , \chi ) = {\left\{ \begin{array}{ll} Z(s, \sigma , \chi ) Z(s, w_0 \sigma , \chi ), &{} \text {if}\ \sigma \ne w_0 \sigma ; \\ Z(s, \sigma , \chi ), &{} \text {if}\ \sigma = w_0 \sigma , \end{array}\right. } \end{aligned}$$

(66)

where $w_0$ is the non-identity element of W(A) from (9).

In this subsection we prove the existence of the meromorphic continuation of the symmetric Selberg zeta function. We follow the approach of [17] which associates a vector bundle $E(\sigma )$ to every representation $\sigma \in \widehat{SO(2n)}$. This vector bundle is graded and there exists a canonical graded differential operator $A(\sigma , \chi )$ which acts on smooth sections of $E(\sigma )$. The next step is to apply the Selberg trace formula to $A(\sigma ,\chi )$ with a certain test function.

First, we construct the bundle $E(\sigma )$ and the operator $A(\sigma ,\chi )$.

Definition 35

Let R(K) and R(M) be the representation rings over ${\mathbb {Z}}$ of K and M, respectively.

Definition 36

Denote by $\iota ^*: R(K) \rightarrow R(M)$ the restriction map induced by the inclusion $\iota : M \hookrightarrow K$.

By [17, Proposition 2.17], there exist integers $m_\nu (\sigma ) \in \{ -1, 0, 1\}$ such that for $\sigma = w_0 \sigma$, one has $\sigma = \sum _{\nu \in \widehat{K}} m_\nu (\sigma ) \iota ^* \nu$ and for $\sigma \ne w_0 \sigma$, one has $\sigma + w_0 \sigma = \sum _{\nu \in \widehat{K}} m_\nu (\sigma ) \iota ^* \nu .$ Moreover, $m_\nu (\sigma )$ are zero except for finitely many $\nu \in \widehat{K}$.

Let $E_{\nu , \chi }$ be the orbibundle associated to $\nu \in \widehat{K}$, $\chi : \Gamma \rightarrow GL(V)$ as in Sect. 7.3.

Definition 37

Let $E(\sigma )$ be the orbibundle

$$\begin{aligned} E(\sigma ):= \bigoplus _{\nu : \, m_\nu (\sigma ) \ne 0} E_{\nu , \chi }. \end{aligned}$$

For every $\nu \in \widehat{K}$ let $A_{\nu , \chi }$ be the operator defined by

$$\begin{aligned} A_{\nu , \chi }:= \Delta _{\nu , \chi }^\# + c(\sigma ) - \nu (\Omega _K), \end{aligned}$$

where $\Delta _{\nu ,\chi }^\#$ is as in Sect. 7.2, $\nu (\Omega _K)$ is as in (3) and

$$\begin{aligned} c(\sigma ) = \sum _{j=1}^{n+1} (k_j(\sigma )+\rho _j)^2 - \sum _{j=1}^{n+1} \rho _j^2. \end{aligned}$$

Let $A(\sigma , \chi )$ be the operator acting on $C^\infty ({\mathcal {O}}, E(\sigma ))$ defined by

$$\begin{aligned} A(\sigma , \chi ):= \bigoplus _{\nu : \, m_\nu (\sigma ) \ne 0} A_{\nu ,\chi }. \end{aligned}$$

Let $\tilde{E}(\sigma ):=\bigoplus _{\nu :\, m_\nu (\sigma ) \ne 0} \tilde{E}_{\nu , \chi }$ be the lift of $E(\sigma )$ to ${\mathbb {H}}^{2n+1}$, and let $\tilde{A}(\sigma , \chi )$ be the lift of $A(\sigma ,\chi )$ to $\tilde{E}(\sigma )$. Note that by (45),

$$\begin{aligned} \tilde{A}(\sigma ,\chi ) = \bigoplus _{\nu : \, m_\nu (\sigma ) \ne 0} \left( \tilde{\Delta }_{\nu } + c(\sigma ) - \nu (\Omega _k)\right) \otimes {\text {Id}}_{E_\chi }. \end{aligned}$$

We want to apply the Selberg trace formula to $A(\sigma , \chi )$. For this put

$$\begin{aligned} h_t^\sigma (g):= e^{-t c(\sigma )} \sum _{\nu : \, m_\nu (\sigma ) \ne 0} m_\nu (\sigma ) h_t^\nu (g), \end{aligned}$$

(67)

where $h_t^\nu :={\text {tr}}\, H_t^\nu$, and $H_t^\nu$ is the integral kernel of $e^{-t (\tilde{\Delta }_{\nu } - \nu (\Omega _k))}$.

Lemma 38

[14, Sect. 4] $\Theta _{\sigma ', \lambda }(h_t^\sigma ) = e^{-t \lambda ^2}$ for $\sigma ' \in \{ \sigma , w_0 \sigma \}$ and equals zero otherwise.

We are almost ready to apply the Selberg trace formula.

Definition 38

Let $B_\nu$, $\nu \in \hat{K}$, be trace class operators acting on sections of $E_\nu$. Let

$$\begin{aligned} B=\bigoplus _{\nu : m_\nu (\sigma )\ne 0} B_\nu . \end{aligned}$$

Then define

$$\begin{aligned} {\text {Tr}}_s \, B:= \sum _{\nu : \, m_\nu (\sigma ) \ne 0} m_\nu (\sigma ) \, {\text {Tr}}\, B_\nu . \end{aligned}$$

(68)

Proposition 28, Lemma 29, Theorem 33 and Proposition 34 imply:

Theorem 39

We have

$$\begin{aligned} \begin{aligned}&\text {Tr}_s(e^{-t A(\sigma ,\chi )}) = {\text {vol}}({\mathcal {O}}) \dim (V_\chi )\sum _{\sigma ' \in \hat{M}} \int _{\mathbb {R}}P_{\sigma '} (i \lambda ) \Theta _{\sigma ', \lambda } (h_t^\sigma ) d\lambda \\&\quad +\sum _{\sigma ' \in \hat{M}} \sum _{\{\gamma \}\, \text {elliptic}} {\text {vol}}(\Gamma _\gamma \backslash G_\gamma ) {\text {tr}}(\chi (\gamma )) \sum _{\sigma ' \in \hat{M}} \int _{\mathbb {R}}P^\gamma _{\sigma '}(i \lambda ) \Theta _{\sigma ', \lambda } (h_t^\sigma ) d \lambda \\&\quad \sum _{\sigma ' \in \hat{M}} \sum _{\{\gamma \} \, \text {hyperbolic}} \frac{{\text {tr}}(\chi (\gamma )) \, l(\gamma _0)}{2 \pi D(\gamma ) |\Gamma ^1_\gamma |} \overline{{\text {tr}}(\sigma '(\gamma ))} \int _{\mathbb {R}}\Theta _{\sigma ', \lambda } (h_t^\sigma ) e ^{- l(\gamma ) \lambda }d\lambda . \end{aligned} \end{aligned}$$

Let

$$\begin{aligned} \epsilon (\sigma ) = {\left\{ \begin{array}{ll} 2, &{} \text {if}\ \sigma \ne w_0 \sigma ; \\ 1, &{} \text {if}\ \sigma = w_0 \sigma . \end{array}\right. } \end{aligned}$$

(69)

Denote

$$\begin{aligned} \begin{aligned} I(t)&:= \epsilon (\sigma ) {\text {vol}}({\mathcal {O}}) \dim (V_\chi ) \int _{{\mathbb {R}}} P_{\sigma }(i \lambda ) e^{-t \lambda ^2} dt,\\ E(t)&:= \epsilon (\sigma ) \sum _{\{\gamma \} \, \text {elliptic}} {\text {vol}}(\Gamma _\gamma \backslash G_\gamma ) {\text {tr}}(\chi (\gamma )) \int _{{\mathbb {R}}} P^\gamma _\sigma (i \lambda ) e^{-t \lambda ^2} dt,\\ H(t)&:= \sum _{\{\gamma \} \, \text {hyperbolic}} \frac{{\text {tr}}(\chi (\gamma )) \, l(\gamma _0)}{2 \pi D(\gamma ) |\Gamma ^1_\gamma |} \left( \overline{{\text {tr}}(\sigma (\gamma ))}+\overline{{\text {tr}}(w_0 \sigma (\gamma ))} \right) \int _{\mathbb {R}}e^{-t \lambda ^2} e ^{- l(\gamma ) \lambda }d\lambda . \end{aligned} \end{aligned}$$

(70)

Then Lemma 35, Theorem 39, Lemma 38 together with (69) and (70) imply

$$\begin{aligned} \text {Tr}_s (e^{-t A(\sigma , \chi )}) = I(t) + E(t) + H(t). \end{aligned}$$

(71)

Denote $(A(\sigma ,\chi )+s^2)^{-1} =: R(s^2)$ for $s\in {\mathbb {C}}$, $s \not \in \text {spec}(A(\sigma , \chi ))$. Note that for $\text {Re}( s^2 + A(\sigma ,\chi ) ) > 0$,

$$\begin{aligned} R(s^2)=\int _0^\infty e^{-ts^2}e^{-t A(\sigma ,\chi )} dt. \end{aligned}$$

(72)

The operator $R(s^2)$ is not a trace class operator, but we will now improve it.

Lemma 40

[1, Lemma 3.5] Let $s_1, \ldots , s_N \in {\mathbb {C}}$ such that $s_i^2 \ne s_j^2$ for $i \ne j$. Then for every $z \in {\mathbb {C}}{\setminus } \{ -s_1^2, \ldots , -s_N^2 \}$ one has

$$\begin{aligned} \sum _{i=1}^N \frac{1}{s_i^2 + z} \prod _{j = 1, j\ne i}^{N} \frac{1}{s_j^2 - s_i^2}= \prod _{i=1}^N \frac{1}{s_i^2+z}, \end{aligned}$$

hence for $c_i = \prod _{j = 1, j\ne i}^{N} \frac{1}{s_j^2 - s_i^2}$, $i=1, \ldots , N,$

$$\begin{aligned} \sum _{j=1}^N c_j R(s_j^2) = \prod _{j=1}^N R(s_j^2). \end{aligned}$$

(73)

Lemma 41

The operator $\prod _{j=1}^N R(s_j^2)$ is of trace class.

Proof

In [1] Lemma 40 was proven for manifolds by the following argument: each of the factors is a pseudodifferential operator of order $-2/(2n+1)$, hence their product is a pseudodifferential operator of order $-2 N / (2n+1)$ that is of trace class for sufficiently large N by the Weyl law.

Now, let R be the value of the resolvent of any self-adjoint Laplacian on $E(\sigma )$ at a point $x \in {\mathbb {R}}$, that is not in its spectrum and such that $R \ge 0$.

Lemma 42

The operator $R^{N}$ is of trace class for $N > (2n+1)/2$.

Proof

Let $\lambda _k$ be the k-th eigenvalue of R. Then by Theorem 19, $\lambda _k = O( k^{-2/(2n+1)})$ as $k \rightarrow \infty .$ Note that $\lambda _k ^N$ is the k-th eigenvalue of $R^{N}$, hence $\lambda _k^N = O(k^{-2N/(2n+1)})$ as $k \rightarrow \infty$. $\square$

Lemma 43

The operator $R^{-N}\cdot \prod _{j=1}^N R(s_j^2)$ is bounded.

Proof

It is a pseudodifferential operator of order 0 and hence bounded by Remark 5. $\square$

By the above two lemmas,

$$\begin{aligned} \prod _{j=1}^N R(s_j^2) = R^{N} \cdot \left( R^{-N}\cdot \prod _{j=1}^N R(s_j^2)\right) \end{aligned}$$

is of trace class. $\square$

From now on let all $s_j$, $j = 1,\ldots , N$ satisfy $\text {spec}(s_j + A(\sigma ,\chi )^2 ) \in \{z \in {\mathbb {C}}, \text {Re}(z)>0 \}$. We can choose such $s_j$, because the real parts of eigenvalues of $A(\sigma ,\chi )$ are bounded from below. Put $s:=s_1$ and $c_j:=c'_j/c_1$ for $1\le j \le N$, then by Lemmas 40 and 41, $R(s^2)+\sum _{j=2}^N c_j R(s_j^2)$ is of trace class. Moreover, by (72),

$$\begin{aligned} {\text {Tr}}_s \left( R(s^2)+\sum _{j=2}^N c_j R(s_j^2)\right) =\int _0^\infty \left( e^{-ts^2} + \sum _{j=2}^{N} c_j e^{-ts_j^2} \right) \cdot {\text {Tr}}_s (e^{-t A(\sigma ,\chi )}) dt. \end{aligned}$$

(74)

By analogy with [17, pp. 68–70], it follows from (71) and (70) that

$$\begin{aligned} \begin{aligned} \int _0^\infty \left( e^{-ts^2} + \sum _{j=2}^N c_j e^{-ts_j^2}\right) \cdot I(t) dt&= \epsilon (\sigma ) {\text {vol}}({\mathcal {O}}) \dim (V_\chi ) \\&\quad \times \left( \frac{\pi }{s} P_\sigma (s) + \sum _{j=2}^N \frac{c_j \pi }{s_j} P_\sigma (s_j) \right) ,\\ \int _0^\infty \left( e^{-ts^2} + \sum _{j=2}^N c_j e^{-ts_j^2}\right) \cdot E(t) dt&= \sum _{ \{\gamma \} \, \text {elliptic}}\epsilon (\sigma ) {\text {vol}}(\Gamma _\gamma \backslash G_\gamma ) {\text {tr}}(\chi (\gamma ))\\&\quad \times \left( \frac{\pi }{s} P_{\sigma }^\gamma (s) + \sum _{j=2}^N \frac{c_j \pi }{s_j} P_{\sigma }^\gamma (s_j) \right) ,\\ \int _0^\infty \left( e^{-ts^2} + \sum _{j=2}^N c_j e^{-ts_j^2}\right) \cdot H(t) dt&= \frac{1}{2s} \frac{S'(s,\sigma ,\chi )}{S(s,\sigma , \chi )} \\&\quad + \sum _{j=2}^N \frac{c_j}{2 s_j} \frac{S'(s_j,\sigma ,\chi )}{S(s_j,\sigma ,\chi )}. \end{aligned} \end{aligned}$$

(75)

Note that we are crucially using that $P_{\sigma }^\gamma (\nu )$ and $P_\sigma (\nu )$ are even polynomials in $\nu$. Thus we get

$$\begin{aligned} \begin{aligned}&{\text {Tr}}_s \big (R(s^2)+\sum _{j=2}^N c_j R(s_j^2)\big ) = \frac{1}{2s} \frac{S'(s,\sigma )}{S(s,\sigma )} + \sum _{j=2}^N \frac{c_j}{2 s_j} \frac{S'(s_j,\sigma )}{S(s_j,\sigma )} \\&\quad + \epsilon (\sigma ) {\text {vol}}({\mathcal {O}}) \dim (V_\chi ) \cdot \left( \frac{\pi }{s} P_\sigma (s) + \sum _{j=2}^N \frac{c_j \pi }{s_j} P_\sigma (s_j) \right) \\&\quad + \sum _{ \{\gamma \} \text { elliptic}}\epsilon (\sigma ) {\text {vol}}(\Gamma _\gamma \backslash G_\gamma ) {\text {tr}}(\chi (\gamma )) \cdot \left( \frac{\pi }{s} P_{\sigma }^\gamma (s) + \sum _{j=2}^N \frac{c_j \pi }{s_j} P_{\sigma }^\gamma (s_j) \right) . \end{aligned} \end{aligned}$$

Put

$$\begin{aligned} \begin{aligned} \Xi (s,\sigma ,\chi )&:= \exp \Bigg (- 2 \pi \epsilon (\sigma ){\text {vol}}({\mathcal {O}}) \dim (V_\chi ) \int _0^s P_\sigma (r) dr \\&\quad - 2 \epsilon (\sigma ) \sum _{\{ \gamma \} \text { elliptic}} {\text {tr}}(\chi (\gamma )) \int _0^s P^\gamma _{\sigma } (r) dr\Bigg ) \times S(s, \sigma ,\chi ) \end{aligned} \end{aligned}$$

(76)

Then (76) can be rewritten as

$$\begin{aligned} {\text {Tr}}_s \left( R(s^2)+\sum _{j=1}^N c_j R(s_j^2) \right) = \frac{1}{2s} \frac{\Xi '(s,\sigma ,\chi )}{\Xi (s,\sigma ,\chi )}+\sum _{j=1}^N \frac{c_j}{2s_j}\frac{\Xi '(s_j,\sigma ,\chi )}{\Xi (s_j, \sigma ,\chi )}, \end{aligned}$$

(77)

where $\Xi '(s,\sigma )$ denotes the differentiation with respect to the first variable. It follows from (76) and (77), that $S(s,\sigma )$ extends meromorphically to ${\mathbb {C}}$ if and only if $\Xi (s,\sigma )$ does, moreover, its singularities coincide. Let $\lambda _i$, $i=1,2, \ldots$ be the eigenvalues of $A(\sigma ,\chi )$. For each $\lambda _i$ let ${\mathcal {E}}(\lambda _i)$ be the eigenspace of $A(\sigma )$ with eigenvalue $\lambda _i$. Put

$$\begin{aligned} m_s(\lambda _i, \sigma ):=\sum _{\nu : \, m_\nu (\sigma ) \ne 0} (-1)^{m_\nu (\sigma )+1} \dim \mathcal {E_\nu }(\lambda _i), \end{aligned}$$

(78)

where $\mathcal {E_\nu }(\lambda _i)$ is the eigenspace of $A_{\nu ,\chi }$ with eigenvalue $\lambda _i$. Put

$$\begin{aligned} s_i^\pm = \pm \sqrt{-1} \cdot \sqrt{\lambda _i}, j \in {\mathbb {N}}, \end{aligned}$$

where $\sqrt{\lambda _i}$ is chosen to have the non-negative imaginary part. Note that $\frac{1}{\lambda _i + s^2}$ and $\frac{c_j}{\lambda _i+s_j^2}$ are the eigenvalues of $R(s^2)$ and $c_j R(s_j^2)$, hence by (78) and Lidskii’s theorem,

$$\begin{aligned} {\text {Tr}}_s \left( R(s^2)+\sum _{j=1}^N c_j R(s_j^2) \right) = \sum _{i=1}^\infty \left( \frac{m_s(\lambda _i, \sigma )}{\lambda _i + s^2} + \sum _{j=1}^N \frac{ c_j \cdot m_s(\lambda _i, \sigma )}{\lambda _i + c_j^2}\right) . \end{aligned}$$

(79)

Note that (79) and

$$\begin{aligned} \frac{2s}{\lambda _i + s^2} = \frac{1}{s + s_i^+} + \frac{1}{s - s_i^-} \end{aligned}$$

imply that all residues of $2\,s \cdot {\text {Tr}}_s \left( R(s^2)+\sum _{j=1}^N c_j R(c_j^2) \right)$ in s are integers. Hence by (77), $\Xi (s,\sigma )$ admits a meromorphic extension to ${\mathbb {C}}$. Together with (76) it implies

Theorem 44

The symmetric Selberg zeta function $S(\sigma ,s, \chi )$ has a meromorphic extension to ${\mathbb {C}}$. The set of singularities of $S(s,\sigma , \chi )$ equals $\{ s^{\pm }_i: i \in {\mathbb {N}}\}$. If $\lambda _i \ne 0$, then the orders of $S(s,\sigma , \chi )$ at both $s_i^+$ and $s_i^-$ are equal to $m_s(\lambda _i, \sigma )$. The order of the singularity at $s=0$ is $2m_s(0,\sigma )$.

8.2 Antisymmetric Selberg zeta function

Suppose that $\sigma \ne w_0 \sigma$, otherwise the symmetric Selberg zeta function equals the Selberg zeta function and this section can be skipped. For $\text {Re}(s) > c$ with the constant c as in Proposition 36 we define the antisymmetric Selberg zeta function as

$$\begin{aligned} S_a(s, \sigma ,\chi ):= Z(s, \sigma ,\chi ) / Z(s, w_0 \sigma ,\chi ). \end{aligned}$$

(80)

In this subsection we prove the meromorphic continuation of the antisymmetric Selberg zeta function $S_a(s, \sigma , \chi )$.

8.3 Dirac bundles and twisted Dirac operators

Let ${\text {Cl}}(\mathfrak {p})$ be the Clifford algebra of $\mathfrak {p}$ with respect to the scalar product on $\mathfrak {p}$. Let $\kappa$ be the spin-representation of K and put $\Delta _{2n}:= {\mathbb {C}}^{2^n}$; denote by $\tilde{S} = G \times _\kappa \Delta ^{2n}$ be the spinor bundle on ${\mathbb {H}}^{2n+1}$ and equip it with a connection $\nabla ^S$.

Let $\sigma \in \hat{M}$. By [1, Proposition 1.1], there a unique $\nu (\sigma ) \in R(K)$ such that

$$\begin{aligned} \nu (\sigma ) \otimes \kappa = \nu ^+(\sigma ) \oplus \nu ^-(\sigma ) =: \nu _\kappa (\sigma ), \end{aligned}$$

where $\nu ^{\pm }(\sigma ) \in \hat{K}$. Define $\widetilde{E}_{\nu _{\kappa }(\sigma )}$ to be the locally homogeneous vector bundle over ${\mathbb {H}}^{2n+1}$:

$$\begin{aligned} \widetilde{E}_{\nu _{\kappa }(\sigma )}:= G \times _{\nu _{\kappa }(\sigma )} ( V_{\nu (\sigma )} \otimes \Delta ^{2n}) \rightarrow {\mathbb {H}}^{2n+1}. \end{aligned}$$

Remark 11

Note that

$$\begin{aligned} \widetilde{E}_{\nu _{\kappa }(\sigma )} = \widetilde{E}_{\nu (\sigma )} \times \tilde{S}, \end{aligned}$$

that allows us to equip $\widetilde{E}_{\nu _{\kappa }(\sigma )}$ with the product connection $\nabla ^{\sigma }:= \nabla ^{\nu (\sigma )} \otimes 1 + 1 \otimes \nabla ^S$.

Define $E_\sigma := \Gamma \backslash \widetilde{E}_{\nu _\kappa (\sigma )}$ and denote the pull-back of $\nabla ^{\nu _{\kappa }(\sigma )}$ by the same symbol. Let $\chi : \Gamma \rightarrow GL(V_\chi )$ be a finite-dimensional (possibly non-unitary) complex representation and let $E_\chi \rightarrow {\mathcal {O}}$ be the associated flat vector bundle over ${\mathcal {O}}$ as in Sect. 7.2, equiped with the flat connection $\nabla ^\chi$. Define

$$\begin{aligned} E_{\sigma , \chi }:= E_{\sigma } \otimes E_{\chi }, \end{aligned}$$

equip it with the connection

$$\begin{aligned} \nabla ^{\sigma ,\chi }:= \nabla ^{\sigma } \otimes 1 + 1 \otimes \nabla ^\chi . \end{aligned}$$

Let

$$\begin{aligned} \cdot : \mathfrak {p} \otimes \Delta _{2n} \rightarrow \Delta _{2n} \end{aligned}$$

denote the Clifford multiplication. We extend its action to $V_{\nu (\sigma )} \otimes \Delta _{2n}$ and $(V_{\nu (\sigma )} \otimes \Delta _{2n} ) \otimes V_\chi$ as follows:

$$\begin{aligned} e \cdot _{\sigma } (w \otimes s){} & {} := w \otimes (e \cdot s), \quad w \in V_{\nu (\sigma )}, e \in {\text {Cl}}(\mathfrak {p}), s \in \Delta _{2n}. \\ e \cdot _{\sigma , \chi } ((w \otimes s) \otimes v){} & {} := (w \otimes (e \cdot s)) \otimes v, \quad w \in V_{\nu (\sigma )}, e \in {\text {Cl}}(\mathfrak {p}), v \in V_\chi , s \in \Delta _{2n}. \end{aligned}$$

Consider an open subset U of ${\mathcal {O}}$ such that $E_\chi |_{U}$ is trivial. Then $E_{\sigma , \chi }|_{U}$ is isomorphic to the direct sum of $\text {rank} (E_\chi )$ copies of $E_\sigma |_U$. Let $v_j$ be the basis of flat sections of $E_\chi |_U$, then each $\varphi \in C^\infty (U, E_{\sigma , \chi }|_{U})$ can be written as:

$$\begin{aligned} \varphi = \sum _{j=1}^{\text {rank}(E_\chi )} \phi _j \otimes v_j, \end{aligned}$$

where $\phi _j \in C^\infty (U, E_\sigma |_U)$. The Dirac operator $D(\sigma , \chi )$ acting on sections of $E_{\sigma , \chi }$ is defined as follows: for each $\varphi$ as above,

$$\begin{aligned} D(\sigma , \chi ) \varphi = \sum _{i=1}^{\dim {\mathcal {O}}} \sum _{j=1}^{\text {rank}(E_\chi )} e_i \cdot _{\sigma , \chi } \left( \nabla _{e_i}^\sigma \phi _j \otimes v_j\right) . \end{aligned}$$

The Dirac operator $\widetilde{D}(\sigma )$ acting on sections of $\widetilde{E}_{\nu _k(\sigma )}$ is defined as follow:

$$\begin{aligned} \widetilde{D}(\sigma ) f = \sum _{i = 1}^{\dim {\mathcal {O}}} e_i \cdot _{\sigma } \nabla ^\sigma _{e_i} f, \end{aligned}$$

where $f \in C^\infty ({\mathbb {H}}^{2n+1}, V_{\nu (\sigma )} \otimes \Delta _{2n}).$

Note that $D(\sigma , \chi )^2$ a second order elliptic differential operator and by Theorem 9, its spectrum is discrete and there exist $R \in {\mathbb {R}}$ and $\varepsilon > 0$ such that

$$\begin{aligned} \text {spec}(D(\sigma , \chi )^2) \in L:= \Lambda _{[-\varepsilon ,\varepsilon ]} \cup B(R). \end{aligned}$$

(81)

8.4 Selberg trace formula

In this subsection we verify that the Selberg trace formula can be applied to the operator $D(\sigma , \chi ) e^{-t D(\sigma , \chi )^2}$. We define the operator $D(\sigma , \chi )e^{-tD(\sigma , \chi )^2}$ via the integral

$$\begin{aligned} D(\sigma , \chi ) e^{-t D(\sigma , \chi )^2}:= \frac{i}{2 \pi }\int _B e^{-t \lambda } D(\sigma , \chi ) (D(\sigma , \chi )^2 - \lambda )^{-1} d \lambda \end{aligned}$$

(82)

for $B = \partial L$ with L as in (81).

Proposition 45

The right hand side of (82) converges.

Proof

Follows from Theorem 9. $\square$

Note that the lift of $D(\sigma , \chi )$ to ${\mathbb {H}}^{2n+1}$ splits into $\widetilde{D}(\sigma ) \otimes {\text {Id}}$ by the same arguments as in Sects. 7.2 and 8.1. Also the operator $D(\sigma , \chi ) e^{-t D(\sigma , \chi )}$ is an integral operator with smooth kernel, because $e^{-tD(\sigma , \chi )}$ is. By an analogy with the previous calculations we obtain:

Lemma 46

Denote by $k_t^\sigma (\cdot )$ the convolution kernel of $\widetilde{D}(\sigma ) e^{-t \widetilde{D}^2(\sigma ) }$. Then we have

$$\begin{aligned} \begin{aligned} \sum _{\lambda \in \text {spec}(D)} \lambda e^{- t \lambda ^2}&= \dim (V_\chi ) {\text {vol}}(\Gamma \backslash G / K) {\text {tr}}\, k_t^\sigma (e) \\&\quad +\sum _{\{\gamma \} \ne \{e\} } {\text {tr}}\chi (\gamma ) {\text {vol}}(\Gamma _\gamma \backslash G_\gamma ) \int _{G_\gamma \backslash G} {\text {tr}}k_t^\sigma (g^{-1} \gamma g) d\dot{g}. \end{aligned} \end{aligned}$$

(83)

Analogously to Theorem 39, we get

$$\begin{aligned} \begin{aligned}&\text {Tr}_s(D(\sigma , \chi ) e^{-t D(\sigma , \chi )^2}) = {\text {vol}}({\mathcal {O}})\dim (V_\chi ) \sum _{\sigma ' \in \hat{M}} \int _{\mathbb {R}}P_{\sigma '} (i \lambda ) \Theta _{\sigma ', \lambda } (k_t^\sigma ) d\lambda \\&\quad +\sum _{\sigma ' \in \hat{M}} \sum _{\{\gamma \} \text { elliptic}} {\text {vol}}(\Gamma _\gamma \backslash G_\gamma ) {\text {tr}}(\chi (\gamma )) \sum _{\sigma ' \in \hat{M}} \int _{\mathbb {R}}P^\gamma _{\sigma '}(i \lambda ) \Theta _{\sigma ', \lambda } (k_t^\sigma ) d \lambda \\&\quad +\sum _{\sigma ' \in \hat{M}} \sum _{\{\gamma \} \text { hyperbolic}} \frac{{\text {tr}}(\chi (\gamma )) l(\gamma _0)}{2 \pi D(\gamma ) |\Gamma ^1_\gamma |} \overline{{\text {tr}}(\sigma '(\gamma ))} \int _{\mathbb {R}}\Theta _{\sigma ', \lambda } (k_t^\sigma ) e ^{- l(\gamma ) \lambda }d\lambda . \end{aligned} \end{aligned}$$

(84)

Proposition 47

([15, 18, Proposition 8.2]) Let $\sigma \in \hat{M}$, $k_{n+1}(\sigma )>0$. Then for $\lambda \in {\mathbb {R}}$ one has

$$\begin{aligned} \Theta _{\sigma , \lambda } (k) = (-1)^n \lambda e^{-t\lambda ^2}, \quad \Theta _{w_0 \sigma , \lambda } (k) = (-1)^{n+1} \lambda e^{-t\lambda ^2}. \end{aligned}$$

Moreover, if $\sigma ' \in \hat{M}$ and $\sigma ' \ne \{ \sigma , w_0 \sigma \},$ for every $\lambda \in {\mathbb {R}}$ one has $\Theta _{ \sigma ', \lambda }(k) = 0.$

Applying Proposition 47 to (84), we get

$$\begin{aligned} \begin{aligned}&(-1)^n {\text {Tr}}_s (D e^{-t D^2}) = {\text {vol}}({\mathcal {O}}) \dim (V_\chi )\int _{\mathbb {R}}(P_\sigma (i \lambda ) - P_{w_0 \sigma } (i \lambda )) \lambda e^{- t\lambda ^2}d\lambda \\&\quad + \sum _{[\gamma ] \text { elliptic}} {\text {vol}}(\Gamma _\gamma \backslash G_\gamma ) \int _{\mathbb {R}}(P^\gamma _\sigma (i \lambda ) - P^\gamma _{w_0 \sigma }(i \lambda )) \lambda e^{- t\lambda ^2} d \lambda \\&\quad + \sum _{[\gamma ] \text { hyperbolic}} C_2(\gamma ) \frac{l(\gamma _0)}{2 \pi } \left( L(\gamma , \sigma ) - L(\gamma , w_0 \sigma ) \right) \int _{\mathbb {R}}\lambda e^{-t \lambda ^2} e ^{- l(\gamma ) \lambda }d\lambda , \end{aligned} \end{aligned}$$

(85)

Moreover, the first and the second summand in the right hand side of (85) vanish by the following two remarks.

Remark 12

By [14, (2.22)],

$$\begin{aligned} P_\sigma (i \nu ) - P_{w_0 \sigma }(i \nu ) = 0. \end{aligned}$$

Remark 13

By Lemma 35,

$$\begin{aligned} P^\gamma _\sigma (i \nu ) - P^\gamma _{w_0 \sigma }(i \nu ) = 0. \end{aligned}$$

We proceed as in Sect. 8.1. The operator $D(\sigma , \chi ) \cdot (D(\sigma , \chi )^2 + s^2)^{-1}$ is not of trace class, but we can choose coefficients $c_j$ and $s_j$ such that $D(\sigma , \chi ) \cdot (D(\sigma , \chi )^2 + s^2)^{-1} + \sum _j c_j D(\sigma , \chi ) \cdot (D(\sigma , \chi )^2 + s_j^2)^{-1}$ is of trace class. By the same arguments as in (74)–(76) and Remarks 12 and 13, we obtain

$$\begin{aligned} \begin{aligned}&{\text {Tr}}\left( D(\sigma , \chi ) \cdot (D(\sigma , \chi )^2 + s^2)^{-1} + \sum _j c_j D(\sigma , \chi ) \cdot (D(\sigma , \chi )^2 + s_j^2)^{-1} \right) \\&\quad = \frac{1}{2s} \frac{S_a'(s, \sigma , \chi )}{S_a(s, \sigma , \chi )} + \sum _j \frac{c_j}{2 s_j} \frac{S_a'(s_j, \sigma , \chi )}{S_a(s_j, \sigma , \chi )} \end{aligned} \end{aligned}$$

(86)

The theorem below follows.

Theorem 48

The antisymmetric Selberg zeta $S_a(s, \sigma , \chi )$ function has a meromorphic extension to ${\mathbb {C}}$. It has singularities at the points $\pm i \mu _k$ of order $\frac{1}{2} (d(\pm \mu _k, \sigma ) - d(\mp \mu _k, \sigma )),$ where $\mu _k$ is a non-zero eigenvalue of $D(\sigma , \chi )$ of multiplicity $d(\mu _k, \sigma )$.

Using that $Z(s, \sigma ,\chi ) = S(s, \sigma ,\chi ) S_a(s, \sigma ,\chi )$, we obtain

Theorem 49

The Selberg zeta function has an meromorphic extension to ${\mathbb {C}}$. It has the following singularities:

• If $\sigma = w_0 \sigma$, a singularity at the points $\pm i \sqrt{\lambda _k}$ of order $m_s(\lambda _k, \sigma )$, where $\lambda _k$ is a non-zero eigenvalue of $A(\sigma ,\chi )$ and $m_s(\lambda _k, \sigma )$ is the graded dimension of the corresponding eigenspace.

• If $\sigma \ne w_0 \sigma$, a singularity at the points $\pm i \mu _k$ of order $\frac{1}{2} (m_s(\mu _k^2, \sigma ) + d(\pm \mu _k, \sigma ) - d(\mp \mu _k, \sigma )).$ Here $\mu _k$ is a non-zero eigenvalue of $D(\sigma , \chi )$ of multiplicity $d(\mu _k, \sigma )$ and $m_s(\mu _k^2, \sigma )$ is the graded dimension of the eigenspace $A(\sigma , \chi )$ corresponding to the eigenvalue $\mu _k^2.$

• At the point $s=0$ a singularity of order $2m_s(0, \sigma )$ if $\sigma = w_0 \sigma$ and of order $m_s(0, \sigma )$ if $\sigma \ne w_0 \sigma .$