Analytic continuation of nonholomorphic discrete series for classical groups

Abstract

The question of unitarity of representations in the analytic continuation of discrete series from a Borel-de Siebenthal chamber is considered for those linear equal-rank classical simple Lie groups G that have not been treated fully before. Groups treated earlier by other authors include those for which G has real rank one or has a symmetric space with an invariant complex structure . Thus the groups in question are locally isomorphic to SO(2m,  n)₀ with m ≥ 2 and n ≥ 3, or to Sp(m, n) with m ≥ 2 and n ≥ 2.

The representations under study are obtained from cohomological induction. One starts from a finite-dimensional irreducible representation of a compact subgroup L of G associated to a Borel-de Siebenthai chamber, forms an upside-down generalized Verma module, applies a derived Bernstein functor, and passes to a specific irreducible quotient. Enright, Parthasarthy, Wallach, and Wolf had previously identified all cases where the representation of L is I-dimensional and the generalized Verma-like module is irreducible; for these cases they proved that unitarity is automatic. B. Gross and Wallach had proved unitarity for additional cases for a restricted class of groups when the representation of L is I-dimensional.

The present work gives results for all groups and allows higher-dimensional representations of L. In the case of I-dimensional representations of L, the results address unitarily and nonunitarity and are conveniently summarized in a table that indicates how close the results are to being the best possible. In the case of higher-dimensional representations of L, the method addresses only unitarity and in effect proceeds by reducing matters to what happens for a 1-dimensional representation of L and a lower-dimensional group G.