Rationally smooth elements of Coxeter groups and triangle group avoidance

Abstract

We study a family of infinite-type Coxeter groups defined by the avoidance of certain rank 3 parabolic subgroups. For this family, rationally smooth elements can be detected by looking at only a few coefficients of the Poincaré polynomial. We also prove a factorization theorem for the Poincaré polynomial of rationally smooth elements. As an application, we show that a large class of infinite-type Coxeter groups have only finitely many rationally smooth elements. Explicit enumerations and descriptions of these elements are given in special cases.

1 Introduction

Let W be a Coxeter group with finite generating reflection set S, and let ℓ and ≤ denote the length function and Bruhat order on W, respectively. Let e∈W denote the identity of W. By definition, W is the group generated by S satisfying relations $(st)^{m_{st}} = e$, where m _st ∈{1,2,3,…,∞} such that m _st =1 if and only if s=t. If m _st =∞, then by convention the relation (st)^∞=e is omitted. The Poincaré series

$$P_w(q) = \sum_{x \leq w} q^{\ell(x)} $$

of an element w∈W is a polynomial of degree ℓ(w). An element w is said to be palindromic (or rationally smooth) if the coefficients of P _w (q) are the same whether read from top degree to bottom degree, or in reverse.¹ In other words, if we write P _w (q)=∑a _i q ⁱ, then w is palindromic when a _i =a _ℓ(w)−i for all i.

An important question in the combinatorics of Coxeter groups is to describe the set of palindromic elements of W. This question stems from its connection with the topology of Schubert varieties. A Coxeter group is crystallographic if m _st ∈{2,3,4,6,∞} for all s≠t. If W is crystallographic, then it can be realized as the Weyl group of a Kac–Moody algebra. The Schubert subvarieties of the full flag variety corresponding to this algebra are indexed by the elements of W. Carrell and Peterson prove that the Schubert variety indexed by w is rationally smooth if and only if w is palindromic [9]. Furthermore, w is palindromic if and only if the Kazhdan–Lusztig polynomial indexed by (x,w) is equal to 1 for all x≤w [11, 12]. If W is crystallographic, then it is sufficient that the Kazhdan–Lusztig polynomial indexed by (e,w) be equal to 1 [9]. For Schubert varieties of simply laced types A,D, and E, the notion of smooth and rationally smooth are equivalent. For finite Weyl groups, the palindromic elements are well understood. In particular, they can be characterized using permutation pattern avoidance in classical types A,B,C, and D and using root system avoidance in all types [3–5, 13]. The characterization using permutation pattern avoidance has recently been extended to the affine type A case as well [2]. The generating series for the number of palindromic elements in A _n , as n varies, is also known [3, 8, 16].

While the theory of palindromic elements is well-developed for finite and affine Coxeter groups, the situation for general Coxeter groups is quite different. In particular, it seems to be quite difficult to determine whether or not an element of a general Coxeter group is rationally smooth. In this paper, we introduce a family of Coxeter groups (mostly) outside the finite and affine cases, for which it is possible to determine if an element is rationally smooth by looking at just a few coefficients of the Poincaré polynomial. The family in question is defined as the set of all Coxeter groups which do not contain certain triangle groups as standard parabolic subgroups. A triangle group is a Coxeter group with |S|=3. Triangle groups arise naturally in arithmetic geometry and the study of tessellations of triangles on Riemann surfaces, see e.g. [1]. We will denote a triangle group by the triple (m _rs ,m _rt ,m _st ) where S={r,s,t}. We say a Coxeter group W contains the triangle (a,b,c) if there exists a subset {r,s,t}⊆S such that (a,b,c)=(m _rs ,m _rt ,m _st ). If S contains no such subset, then we say W avoids the triangle (a,b,c). We are interested in the groups which avoid the following special set of triangle groups:

$$\operatorname{HQ}:=\bigl\{(2,b,c)\bigm|b,c\geq3\ \text{and}\ b<\infty\bigr\}. $$

The set $\operatorname{HQ}$ (Hecke quotients) is the set of quotients of the Hecke triangle group (2,p,∞), p≥3, which is a generalization of the well-known modular group (2,3,∞). Every finite Coxeter group of rank ≥3 contains a triangle in $\operatorname{HQ}$, and the same is true of affine Coxeter groups, with the exception of (3,3,3), which is the affine group $\tilde{A}_{2}$. However, there are many crystallographic Coxeter groups which do avoid $\operatorname{HQ}$; for example, any Coxeter group with no commuting relations (i.e. m _st ≥3 for all s≠t) avoids $\operatorname{HQ}$. Any Coxeter group defined by only by commuting and infinite relations also avoids $\operatorname{HQ}$.

To state our main theorem, we make the following definition:

Definition 1.1

Let w be an element of a Coxeter group W, and write P _w (q)=∑a _i q ⁱ for the Poincaré polynomial of w. We say that w is k-palindromic if a _i =a _ℓ(w)−i for all 0≤i<k.

Note that if k=∞, then we recover the usual notion of palindromic elements, and that every element is 1-palindromic with a ₀=a _ℓ(w)=1. If W is crystallographic, then k-palindromicity can be detected from the Kazhdan–Lusztig polynomial. Let T _e,w =1+∑ _i≥0 b _i q ⁱ be the Kazhdan–Lusztig polynomial indexed by (e,w). A theorem of Bjorner and Ekedahl states that, for crystallographic groups, an element w∈W is k-palindromic if and only if b _i =0 for 0≤i<k [7] (note that b ₀=0 always).

We now state the main theorem:

Theorem 1.2

Let W be a Coxeter group which avoids all triangle groups in $\operatorname{HQ}$. Then every 4-palindromic w∈W is palindromic.

Furthermore, if W avoids all triangle groups (3,3,c) where 3<c<∞, then every 2-palindromic w∈W is palindromic.

Given a Coxeter group, it is natural to ask whether there is a number k such that every k-palindromic element is palindromic. This question appears to be open in general. Billey and Postnikov have conjectured that if W is a finite simply laced Weyl group with n generators, then every (n+1)-palindromic element of W is palindromic [4]. In type A _n , it is known that every (n−1)-palindromic element is palindromic [4].

The proof of Theorem 1.2 is based on a factorization theorem for the Poincaré polynomial of 2-palindromic elements in Coxeter groups which avoid $\operatorname{HQ}$. In the classical groups of finite type A,B,C, and D, it is known that the Poincaré polynomial of a rationally smooth element factors into a product of q-integers (see Eq. (2)) [5, 10]. In fact, it is possible to see this factorization combinatorially, writing each palindromic element w as a reduced product w ₁⋯w _|S|, such that each q-integer factor of the Poincaré polynomial equals the (relative) Poincaré polynomial of the w _i ’s. We prove a similar result for 2-palindromic elements in Coxeter groups which avoid $\operatorname{HQ}$. This result has a number of applications. For example, we show there are many infinite Coxeter groups with only a finite number of palindromic elements. We also give explicit descriptions of palindromic elements in special cases. In the case of uniform Coxeter groups W(m,n), defined by m _st =m for all s≠t and |S|=n, we calculate the generating series for the number of palindromic elements weighted by length. Formulas for these generating series are stated in Propositions 3.8 and 3.9. We also observe that the $\operatorname{HQ}$-avoiding groups form the largest class of Coxeter groups for which our factorization theorem can hold.

1.1 Organization

Section 2 contains some background material and elementary lemmas used to state the factorization theorem. Section 3 states the main factorization theorem and its consequences, including the proof of Theorem 1.2 and enumerative results. In Sect. 4, we consider triangle groups in the set $\operatorname{HQ}$ and prove the main results cannot hold for any Coxeter group containing these triangle groups. Section 5 gives some elementary lemmas on the descent sets of Coxeter groups avoiding $\operatorname{HQ}$. Finally, Sect. 6 proves the main factorization theorem.

2 Background and terminology

Let W be a Coxeter group with simple generator set S. For basic facts on Coxeter groups, we refer the reader to [6]. Let ℓ(w) denote the length of w∈W. We say w=uv∈W is a reduced factorization if ℓ(w)=ℓ(u)+ℓ(v). A special type of reduced factorization can be constructed from any subset J⊆S. Let W _J denote the standard parabolic subgroup of W generated by J. Let W ^J denote the set of minimal length coset representatives of W _J ∖W. Every element w∈W can be written uniquely as w=uv where u∈W _J , v∈W ^J and ℓ(w)=ℓ(u)+ℓ(v). We call this reduced factorization of w the parabolic decomposition with respect to J.

Let ≤ denote the Bruhat order on W. If u≤v∈W, then the interval [u,v] denotes the set of elements x∈W such that u≤x≤v. For any w∈W we can define the Poincaré polynomial

$$P_w(q):= \sum_{x \in[e,w]} q^{\ell(x)}. $$

The Poincaré polynomial relative to J⊆S of an element w∈W is defined to be

$$P_w^J(q) := \sum_{x \in[e,w] \cap W^J} q^{\ell(x)}. $$

If w∈W ^J, then $P_{w}^{J}(q)$ is a polynomial of degree ℓ(w). If J=∅, then $P_{w}^{J}(q)=P_{w}(q)$. Recall that for any J, the poset [e,w]∩W _J has a unique maximal element. The following proposition is due to Billey and Postnikov in [4, Theorem 6.4].

Proposition 2.1

[4]

Let J⊆S and let w=uv be a parabolic decomposition with respect to J. Then u is the unique maximal element of [e,w]∩W _J if and only if

$$P_w(q) = P_u(q) \cdot P_v^J(q). $$

While the proof of Proposition 2.1 given in [4] is stated only for finite Weyl groups, it easily extends to all Coxeter groups. A parabolic decomposition w=uv is called a BP-decomposition of w if u is the unique maximal element of [e,w]∩W _J .

For any w∈W, define the sets

$$\begin{aligned} \mathsf{S}(w)&:=\bigl\{u\leq w\bigm|\ell(u)=1\bigr\}, \\ \mathsf{D}(w)&:=\bigl\{u\leq w\bigm|\ell(u)=\ell(w)-1\bigr\}, \\ D_R(w)&:=\bigl\{s\in S\bigm|\ell(ws)<\ell(w)\bigr\}, \\ D_L(w)&:=\bigl\{s\in S\bigm|\ell(sw)<\ell(w)\bigr\}. \end{aligned}$$

The sets S(w) and D(w) are known as the support and divisor sets of w. The sets D _R (w) and D _L (w) are called the right and left descent sets of w respectively and are contained in S(w). We use these sets to give an equivalent characterization of a BP-decomposition.

Lemma 2.2

A parabolic decomposition w=uv is a BP-decomposition if and only if S(v)∩J⊆D _R (u).

Proof

If w=uv is a BP-decomposition, then u is the unique longest element of [e,w]∩W _J . If there exist x∈S(v)∩J and x∉D _R (u), then ℓ(ux)=ℓ(u)+1 and ux∈[e,w]∩W _J which is a contradiction.

Conversely, assume that S(v)∩J⊆D _R (u) and let $\bar{u}$ denote the maximal element in [e,w]∩W _J . Since $\bar{u}$ is unique, we have $u\leq \bar{u}$. We now show that $\bar{u} \leq u$. Let

$$\bar{u}=u'v' $$

be a reduced factorization which maximizes ℓ(u′) under the conditions that u′≤u and v′≤v. Suppose that v′≠e. Then there exists y∈D _L (v′)∖D _R (u′). By assumption, we have y∈D _R (u). Taking a reduced decomposition for u with y appearing at the end, we see that u′≤uy, and hence u′ can be extended, a contradiction. □

We remark that one direction of Lemma 2.2 is proved in [15, Lemma 10]. Another property of BP-decompositions is the following lemma.

Lemma 2.3

Let J ₁⊆J ₂⊆S and let v ₁ v ₂ v ₃ be a reduced factorization such that v ₁ v ₂ and (v ₁ v ₂)(v ₃) are BP-decompositions with respect to J ₁ and J ₂ respectively. Then v ₁(v ₂ v ₃) is a BP-decomposition with respect to J ₁.

Proof

By definition, we have v ₁ v ₂ is maximal in $[e, v_{1}v_{2}v_{3}]\cap W_{J_{2}}$. In particular, if u denotes the maximal element in $[e,v_{1}v_{2}v_{3}]\cap W_{J_{1}}$, then u≤v ₁ v ₂ since $W_{J_{1}}\subseteq W_{J_{2}}$. But now u is maximal in $[e,v_{1}v_{2}]\cap W_{J_{1}}$, which implies that u=v ₁. □

Clearly, if P _w (q)=∑a _i q ⁱ, then |S(w)|=a ₁ and |D(w)|=a _ℓ(w)−1. We now consider a special class of parabolic decompositions.

Definition 2.4

We say that w=uv, a parabolic decomposition with respect to J, is a Grassmannian factorization if J=S(u) and |S(w)|=|S(u)|+1.

It is easy to see that every element w∈W of length ≥2 has a Grassmannian factorization. The term “Grassmannian” comes from the fact that v is a Grassmannian element of W which, by definition, has |D _L (v)|=1. Note that a Grassmannian factorization is not necessarily a BP-decomposition. Although elementary, this concept is quite useful. For example, we can use it to prove:

Lemma 2.5

|D(w)|≥|S(w)|.

Proof

We proceed by induction on ℓ(w). The proposition is true if ℓ(w)=1, so suppose ℓ(w)≥2. Let w=uv be a Grassmannian factorization with respect to J. By induction, |D(u)|≥|S(u)|.

If u′∈D(u), then u′v∈D(w), since v∈W ^J. Now v is not the identity, so we can write v′=vs∈W ^J with s∈S and ℓ(v′)=ℓ(v)−1. Consequently uv′∈D(w). Moreover, uv′≠u′v for any u′∈D(u) since they are both parabolic decompositions with respect to J and u≠u′. Hence

$$ \bigl|\mathsf{D}(w)\bigr|\geq\bigl|\mathsf{D}(u)\bigr|+1\geq \bigl|\mathsf{S}(u)\bigr|+1=\bigl| \mathsf{S}(w)\bigr|. $$

(1)

This completes the proof. □

We remark that, for crystallographic Coxeter groups, Bjorner and Ekedahl prove a much stronger version of Lemma 2.5 concerning all the coefficients of P _w (q) [7, Theorem A].

We can continue to decompose any Grassmannian factorization w=uv by taking a Grassmannian factorization of u. We say that

$$w=v_1v_2\cdots v_{|\mathsf{S}(w)|} $$

is a complete Grassmannian factorization of w if for every i<|S(w)|, we see that (v ₁⋯v _i )(v _i+1) is a Grassmannian factorization. Observe that if each (v ₁⋯v _i )(v _i+1) is also a BP-decomposition, then by Lemma 2.3, we have $(v_{1}\cdots v_{i}) (v_{i+1}\cdots v_{k})$ is a BP decomposition for any i<k≤|S(w)|.

By definition, w is 2-palindromic if and only if |D(w)|=|S(w)|. The following lemma gives an inductive characterization of the 2-palindromic property.

Lemma 2.6

Suppose that w=uv is a Grassmannian factorization. Then w is 2-palindromic if and only if u is 2-palindromic and |u⋅D(v)∩D(w)|=1.

Proof

Equality holds in Eq. (1) if and only if |D(u)|=|S(u)| and u⋅D(v)∩D(w)={uvs} where s∈D _R (v). □

3 The factorization theorem

The main technical theorem of this paper is the following:

Theorem 3.1

Suppose that W avoids all triangle groups in $\operatorname{HQ}$. Let w∈W be 2-palindromic and fix a Grassmannian factorization w=uv with respect to J⊆S. Then w=uv is a BP-decomposition with respect to J such that |S(v)|≤3.

Moreover, if |S(v)|=3 and S(v)={r,s,t}, then one of the following is true:

• (1) v=trv′ with $v'=\underbrace{stst\ldots}_{m_{st}-1}$ where S(v) generates the triangle group (3,m _rs ,m _st ) with m _rt =3 and 3≤m _st <∞, 3≤m _rs ≤∞.

• (2) v=rstrv′ with $v'=\underbrace{stst\ldots}_{m_{st}-1}$ where S(v) generates the triangle group (3,3,m _st ) with 3<m _st <∞.

• (3) v=strstr⋯ is a spiral word ² of even length where S(v) generates the triangle group (3,3,3).

Theorem 3.1 says that if W avoids triangle groups in $\operatorname{HQ}$, then the Poincaré polynomial P _w (q) of a 2-palindromic element w∈W factors along any Grassmannian factorization of w=uv. Moreover, the possibilities for the factor $P^{J}_{v}(q)$ is limited by the fact that |S(v)|≤3. Note that parts (1) and (3) of the theorem overlap when m _rs =m _st =3. The proof of this theorem is the focus of Sect. 6. The remainder of this section is devoted to consequences of Theorem 3.1.

Fix a 2-palindromic element w∈W and a Grassmannian factorization w=uv with respect to J⊆S. Theorem 3.1 can be used, together with Lemma 2.6, to completely determine the polynomial P _w (q). By Theorem 3.1 and Proposition 2.1, we have

$$P_w(q) = P_u(q) \cdot P_v^J(q), $$

so it suffices to characterize all possible polynomials $P_{v}^{J}(q)$. For any integer k≥1 define the q-integer

$$ [k]_q:=1+q\cdots+q^{k-1}. $$

(2)

If |S(v)|≤2, then any v′≤v where v′∈W ^J is given by a prefix of the unique reduced word of v. This implies

$$ P_v^J(q)=\bigl[\ell(v)+1 \bigr]_q. $$

(3)

If |S(v)|=3, it suffices to compute $P_{v}^{J}(q)$ in all the cases of Theorem 3.1. We have the following lemma.

Lemma 3.2

Suppose we have w=uv as in Theorem 3.1 with |S(v)|=3. Then the following are true:

• (1) If v satisfies the conditions in Theorem 3.1 part (1), then

  $$P_v^J(q)=\bigl[\ell(v)+1\bigr]_q+q^2 \bigl[\ell(v)-3\bigr]_q. $$

• (2) If v satisfies the conditions in Theorem 3.1 part (2), then

  $$P_v^J(q)=\bigl[\ell(v)+1\bigr]_q+q^2 \bigl[\ell(v)-3\bigr]_q+q^4\bigl[\ell(v)-6 \bigr]_q. $$

• (3) If v satisfies the conditions in Theorem 3.1 part (3) with $k= \lfloor\frac{\ell(v)}{4} \rfloor$, then

  $$P_v^J(q)=\sum_{i=0}^k q^{2i}\bigl[\ell(v)-4i+1\bigr]_q. $$

Proof

Part (3) is proved in [14, Proposition 2.4] where certain Poincaré polynomials of Schubert varieties in the affine Grassmannian of type A are calculated. Parts (1) and (2) can be deduced from elementary counting arguments of the sets

$$\bigl\{v'\in W^J\cap[e,v]\bigm|\ell \bigl(v'\bigr)=i\bigr\}. $$

In particular, for part (1), there are two q-integer contributions from reduced subwords of the form

$$tr\underbrace{stst\ldots}_k\quad\text{and}\quad\underbrace{tstst \ldots}_k. $$

For part (2) there are three q-integer contributions from reduced subwords of the form

$$r\underbrace{tsts\ldots}_k\quad\text{and}\quad r\underbrace{stst \ldots}_k\quad\text{and}\quad rstr\underbrace{stst \ldots}_k. $$

 □

The polynomials in parts (1) and (3) of the lemma are palindromic, while the polynomial is part (2) is 3-palindromic but not 4-palindromic. We now prove the theorem stated in the introduction.

Proof of Theorem 1.2

Suppose that W avoids all triangles in $\operatorname{HQ}$. Let w=v ₁ v ₂⋯v _|S(w)|∈W be a complete Grassmannian factorization. Then by Theorem 3.1 and Proposition 2.1, we have

$$P_w(q)=\prod_{i=1}^{|\mathsf{S}(w)|} P_{v_i}^{J_i}(q), $$

where J _i :=S(v ₁)∪⋯∪S(v _i−1) and J ₁:=∅. Moreover, the factors $P_{v_{i}}^{J_{i}}(q)$ are given by either Eq. (3) or by parts (1)–(3) of Lemma 3.2. Now P _w (q) is 4-palindromic if the polynomial in Lemma 3.2 part (2) does not appear as one of the factors $P_{v_{i}}^{J_{i}}(q)$. Since all other possible choices for $P_{v_{i}}^{J_{i}}(q)$ are palindromic, we see that P _w (q) is 4-palindromic if and only if it is palindromic. This proves part (1) of Theorem 1.2.

If W also avoids the triangles of the form (3,3,c), then Lemma 3.2 part (2) is never an option for $P_{v_{i}}^{J_{i}}(q)$. Hence every 2-palindromic w∈W is palindromic. This completes the proof. □

3.1 Examples

Consider the Coxeter group W with S={s ₁,s ₂,s ₃,s ₄} defined by the Dynkin diagram in Fig. 1.

Fig. 1

Dynkin diagram of W in Examples 3.3 and 3.4

Unlabeled edges are assumed to have label m _st =3 and if there is no edge between s and t, then m _st =2. Clearly, W avoids all triangle groups in $\operatorname{HQ}$ and hence we can apply Theorem 3.1 to compute Poincaré polynomials.

Example 3.3

Let w=s ₁ s ₂ s ₁ s ₃ s ₂ s ₁ s ₃ s ₂ s ₁ s ₄. Then w is 2-palindromic with |S(w)|=|D(w)|=4. The following is a complete Grassmannian factorization:

$$w=\underbrace{(s_1)}_{v_1}\underbrace{(s_2s_1)}_{v_2} \underbrace{(s_3s_2s_1s_3s_2s_1)}_{v_3} \underbrace{(s_4)}_{v_4}. $$

The corresponding Poincaré polynomial factorization is

$$\begin{aligned} P_w(q)&=[2]_q[3]_q\bigl([7]_q+q^2[3]_q \bigr)[2]_q \\ &=(1+q) \bigl(1+q+q^2\bigr) \bigl(1+q+2q^2+2q^3+2q^4+q^5+q^6 \bigr) (1+q), \end{aligned}$$

so P _w (q) is palindromic.

Example 3.4

Let w=s ₂ s ₄ s ₂ s ₄ s ₁ s ₂ s ₄ s ₁ s ₂ s ₄ s ₂. Then w is 2-palindromic with |S(w)|=|D(w)|=3. A complete Grassmannian factorization of w is

$$w=\underbrace{(s_2)}_{v_1}\underbrace{(s_4s_2s_4)}_{v_2} \underbrace{(s_1s_2s_4s_1s_2s_4s_2)}_{v_3}. $$

The corresponding Poincaré polynomial factorization is

$$\begin{aligned} P_w(q)&=[2]_q[4]_q\bigl([8]_q+q^2[4]_q+q^4[1]_q \bigr) \\ &=(1+q) \bigl(1+q+q^2+q^3\bigr) \bigl(1+q+2q^2+2q^3+3q^4+2q^5+q^6+q^7 \bigr). \end{aligned}$$

Note that {s ₁,s ₂,s ₄} generates the triangle group (3,3,4). Since v ₃=s ₁ s ₂ s ₄ s ₁ s ₂ s ₄ s ₂, we find that w is 3-palindromic but not 4-palindromic.

An example of a $\operatorname{HQ}$-avoiding Coxeter group W with commuting relations is given by the Dynkin diagram in Fig. 2 where p≥3.

Fig. 2

Dynkin diagram of a HQ-avoiding Coxeter group with commuting relations

Observe that W also avoids all triangle groups of the form (3,3,c). Hence every 2-palindromic element is palindromic by Theorem 1.2. Moreover, every palindromic polynomial factors into a product of q-integers. We also remark that W is indecomposable with respect to products and free products of Coxeter groups.

3.2 Enumeration and description of palindromic elements

Theorem 3.1 gives a description of the set of palindromic (resp. 2-palindromic) elements of any $\operatorname{HQ}$-avoiding Coxeter group. Specifically, the palindromic (resp. 2-palindromic) elements are those with a certain Grassmannian factorization. In this section we provide some applications of this idea. We start by proving a corollary of Theorem 3.1 on the finiteness of the number of palindromic elements for all $\operatorname{HQ}$-avoiding Coxeter groups.

Corollary 3.5

Let (W,S) be a Coxeter group that avoids all triangle groups in $\operatorname{HQ}$. Then W has a finite number of palindromic elements if and only if m _st <∞ for all s,t∈S and W avoids the triangle group (3,3,3).

Proof

Theorem 3.1 part (3) implies that the triangle group (3,3,3) contains an infinite number of palindromic elements. Also, if m _rs =∞, then W _{r,s} is infinite and every element is palindromic.

Let m ₀ denote the largest value of m _st for s,t∈S. Suppose that W avoids (3,3,3) and m ₀<∞. Let w∈W be palindromic with complete Grassmannian factorization w=v ₁⋯w _|S(w)|. By Theorem 3.1, we find that each factor v _i has length at most m ₀+3, so

$$\ell(w)<\bigl|\mathsf{S}(w)\bigr|(m_0+3)\leq|S|(m_0+3) $$

and hence the number of palindromic elements in W is finite. □

Corollary 3.5 also holds if palindromic is replaced by 2-palindromic.

Note that the Grassmannian factorization of an element provided by Theorem 3.1 is not necessarily unique. When m _st ≥3 for all s≠t, we give a modified factorization which does not have this problem. To state the modified factorization we need the following definition.

Definition 3.6

We say a reduced factorization w=u ₁ u ₂⋯u _d is separable if S(u _i )∩S(u _j )=∅ for all i≠j. If no such non-trivial factorization exists, then we say that w is inseparable.

Given any complete Grassmannian factorization of a palindromic element w=v ₁⋯v _|S(w)|, there is a simple method for constructing a separable factorization. Let (i ₁,…,i _d ) denote the subsequence of integers for which $\ell (v_{i_{j}})=1$. Then w=u ₁⋯u _d is a separable factorization where

$$u_j:=v_{i_j}v_{i_j+1}\cdots v_{i_{j+1}-1} $$

and i _d+1:=|S(w)|+1. We remark that ℓ(v ₁)=1 and hence the sequence (i ₁,…,i _d ) is nonempty. Furthermore, each factor u _j is inseparable. For example, let W be defined by the Dynkin diagram in Fig. 1 and w=s ₄ s ₂ s ₄ s ₂ s ₃ s ₁ s ₃. Then w=u ₁ u ₂ given by

$$ w=\overbrace{\underbrace{s_4}_{v_1} \underbrace{s_2s_4s_2}_{v_2}}^{u_1} \overbrace{\underbrace{s_3}_{v_3}\underbrace {s_1s_3}_{v_4}}^{u_2} $$

(4)

is a separable factorization. The following corollary follows from Theorem 3.1.

Corollary 3.7

Let W be a Coxeter group with m _st ≥3 for all s≠t, and let w∈W be palindromic. Then w has a unique separable factorization w=u ₁⋯u _d where each u _i is inseparable and palindromic. Moreover, any complete Grassmannian factorization $u_{i}=v_{1}\cdots v_{|\mathsf{S}(u_{i})|}$ is unique up to choice of v ₁.

Proof

Any element w has a separable factorization w=u ₁⋯u _d where each u _i is inseparable. Since S(u _i ) is distinct and W has no commuting braid relations, the factorization is unique. If w is palindromic, then every u _i is palindromic since (u ₁⋯u _i )(u _i+1) is a BP-decomposition with respect to J=S∖S(u _i ).

Let $u_{i}=v_{1}\cdots v_{|\mathsf{S}(u_{i})|}$ be a complete Grassmannian factorization, and let s _j be the unique element of D _L (v _j ). Note that v ₁=s ₁. As mentioned above, since u _i is inseparable, we must have |S(v _j )|≥2 for j=2,…,|S(u _i )|. Indeed, if S(v _j )={s _j } then s _j is the unique right descent of v ₁⋯v _j , since s _j ∉S(v ₁⋯v _j−1). But by Lemma 2.3, $(v_{1} \cdots v_{j}) (v_{j+1} \cdots v_{|\mathsf{S}(u_{i})|})$ is a BP decomposition, so

$$\mathsf{S}(v_{j+1} \dots v_{|\mathsf{S}(u_i)|}) \cap\mathsf{S}(v_1 \cdots v_j) \subset\{s_j\}. $$

Thus $(v_{1} \cdots v_{j-1}) (v_{j} \cdots v_{|\mathsf{S}(u_{i})|})$ is a separable factorization, which is a contradiction.

We now show that s _j is the unique left descent of $v_{j} \cdots v_{|\mathsf{S}(u_{i})|}$, for j≥2. Indeed, looking ahead to Lemma 5.3, and using the fact that |S(v _j )|≥2, we see that $D_{L}(v_{j} \cdots v_{|\mathsf{S}(u_{i})|})$ is a subset of S(v _j )∖S(v ₁⋯v _j−1)={s _j }. Hence the sequence $(s_{2},\ldots,s_{|\mathsf {S}(u_{i})|})$ is uniquely determined given the choice of v ₁=s ₁, and the v _j ’s are uniquely determined from the corresponding parabolic decomposition. □

Note that there are at most two complete Grassmannian factorizations of each u _i in Corollary 3.7. For example, taking u ₁ in Eq. (4), we have

$$u_1=\underbrace{s_4}_{v_1} \underbrace{s_2s_4s_2}_{v_2}= \underbrace{s_2}_{v_1}\underbrace{s_4s_2s_4}_{v_2} $$

as the only two complete Grassmannian factorizations.

Corollary 3.7 implies that to count the number of palindromic elements of W, it is sufficient to enumerate elements of W which are inseparable and palindromic. When m _st is constant we compute an exponential generating series for the number of palindromic elements. Specifically, let W(m,n) denote the uniform Coxeter group such that |S|=n and m _st =m for all s≠t. Uniform Coxeter groups satisfy the property that every 2-palindromic element w is palindromic by Theorem 3.1. Define the generating series

$$\varPhi_m(q,t):=\sum_{n,k\geq0}P_{n,k} \frac{q^kt^n}{n!} $$

where P _n,k denotes the number of palindromic w∈W(m,n) of length k. In the case that m=2, we have $W(2,n)\simeq(\mathbb{Z}/2\mathbb{Z})^{n}$ with every element palindromic, so $P_{n,k}=\binom{n}{k}$. Hence the generating series

$$\varPhi_2(q,t)=\exp(qt+t). $$

For m≥3, define

$$\phi_m(q,t):=\sum_{n,k\geq1}I_{n,k} \frac{q^kt^n}{n!} $$

where I _n,k denotes the number of palindromic w∈W(m,n) of length k that are inseparable with |S(w)|=n. Note that Φ _m and ϕ _m are exponential in t and ordinary in q. Corollary 3.7 implies

Proposition 3.8

For any 3≤m≤∞, the series

$$\varPhi_m(q,t)=\frac{\exp(t)}{1-\phi_m(q,t)}. $$

The following proposition completes the calculation.

Proposition 3.9

The exponential generating series for the number of inseparable palindromic elements in W(m,n) is

$$ \phi_m(q,t)= \begin{cases}\frac {(2q-2q^3)t-(3q^3+q^5)t^2}{2-2q^2-4q^2t} & \mathit{for}\ m=3,\\ \frac {2qt-3q^{m}t^2-q^{m+2}[m-3]_qt^3}{2-2q^2t([m-2]_q+q^{m-3})} & \mathit{for}\ 4\leq m<\infty,\\ \frac {qt-q^2t}{1-q-q^2t} & \mathit{for}\ m=\infty. \end{cases} $$

(5)

Proof

By Theorem 3.1, |D _R (w)|≤2 for any palindromic w∈W(m,n). Hence we can partition the set of inseparable palindromic elements into those with |D _R (w)|=1,2 respectively. For notation, let A _n,k be the number of inseparable palindromic w∈W(m,n) of length k with |S(w)|=n and D _R (w)=1. Let B _n,k be the number of those same elements with D _R (w)=2. We have I _n,k =A _n,k +B _n,k . Consider the polynomials

$$A_n(q):=\frac{1}{n!}\sum_{k\geq1} A_{n,k} q^k\quad\text{and}\quad B_n(q):= \frac{1}{n!}\sum_{k\geq1} B_{n,k} q^k. $$

If n=1, then

$$A_1(q) = q\quad\text{and}\quad B_1(q) = 0. $$

If 3≤m<∞, then for n=2, the inseparable elements have the form s ₁ s ₂ s ₁⋯ or s ₂ s ₁ s ₂⋯ where the length is at least 3. There is also a unique longest element $w_{0}:=\underbrace {s_{1}s_{2}\cdots}_{m}$ with |D _R (w ₀)|=2. This gives

$$A_2(q) =q^3[m-3]_q \quad\text{and}\quad B_2(q) =\frac{q^m}{2}. $$

For the remainder of the proof, let w=v ₁⋯v _|S(w)|∈W(m,n) be a complete Grassmannian factorization. We first consider the case when m=3. If w is palindromic and inseparable, then by Theorem 3.1, each v _i is a spiral word as in Theorem 3.1 part 3. In particular, for each even length, there is a unique v _i of up to S ₃ permutation symmetry on the generators {r,s,t}. Moreover, if |S(w)|≥3, then |D _L (w)|=2. Thus for all n≥3, we have A _n (q)=0 and

$$B_n(q)= \biggl(\frac{2q^2}{1-q^2} \biggr)B_{n-1}(q)= \frac{q^3}{2} \biggl(\frac{2q^2}{1-q^2} \biggr)^{n-2}. $$

Hence

$$\phi_3(q,t)=qt+\frac{q^3t^2}{2}+\frac{q^5}{1-q^2}\sum _{n\geq 3} \biggl(\frac{2q^2}{1-q^2} \biggr)^{n-3}t^n. $$

This proves the first equation in (5).

Now suppose 4≤m<∞. In this case, if w is palindromic and inseparable, then Theorem 3.1 implies |S(v _i )|≤2. Hence each factor v _i has a reduced expression stst⋯ where t∈D _L (v ₁⋯v _i−1). In particular, when constructing w=v ₁⋯v _|S(w)|, there are exactly twice as many choices for v _i if D _L (v ₁⋯v _i−1)=2 than if D _L (v ₁⋯v _i−1)=1. This yields, for n≥3, the polynomials A _n (q) and B _n (q) satisfy the first order recurrence

$$\begin{aligned} A_n(q)&=q^2[m-3]_q \bigl(A_{n-1}(q)+2B_{n-1}(q) \bigr), \\ B_n(q)&=q^{m-1} \bigl(A_{n-1}(q)+2B_{n-1}(q) \bigr). \end{aligned}$$

This implies that

$$\begin{aligned} \left[ \begin{matrix} A_n(q) \\ B_n(q) \end{matrix} \right]&= \left[ \begin{matrix} q^2[m-3]_q & 2q^2[m-3]_q \\ q^{m-1} & 2q^{m-1} \end{matrix} \right]^{n-2}\left[ \begin{matrix} A_2(q) \\ B_2(q) \end{matrix} \right] \\ &= q^5[m-2]_q\bigl(q^2[m-3]_q+2q^{m-1} \bigr)^{n-3}\left[ \begin{matrix} [m-3]_q \\ q^{m-3} \end{matrix} \right] \\ &= q^7[m-2]_q\bigl([m-2]_q+q^{m-2} \bigr)^{n-3}\left[ \begin{matrix} [m-3]_q \\ q^{m-3} \end{matrix} \right]. \end{aligned}$$

Thus

$$\begin{aligned} \phi_m(q,t)&=q t+ \biggl(q^3[m-3]_q+ \frac{q^m}{2} \biggr) t^2+q^7[m-2]^2 \sum_{n\geq 3} \bigl([m-2]_q+q^{m-3} \bigr)^{n-3} t^n \end{aligned}$$

which proves the second equation in (5).

Finally, we compute the exponential generating series for the uniform Coxeter group W(∞,n) by taking the limit of ϕ _m in the second equation of (5) as m→∞. This is equivalent to taking

$$q^m\rightarrow0\quad\text{and}\quad[m]_q\rightarrow \frac{1}{1-q} $$

which yields the third equation in (5). □

The following equations are the first few terms in the Taylor expansion of Φ _m (q,t) for m=3,4,∞. These calculations were computed using the combinat package for Mupad.

$$\begin{aligned} \varPhi_3(q,t)=&1+(1+q) t+\bigl(1+2q+2q^2+q^3 \bigr) \frac{t^2}{2} \\ &+\bigl(1+3q+6q^2+9q^3+6q^4+6q^5+6q^7+O \bigl(q^9\bigr)\bigr) \frac{t^3}{6} \\ &+\bigl(1+4q+12q^2+30q^3+48q^4+60q^5+54q^6+O \bigl(q^7\bigr)\bigr) \frac{t^4}{24}+O\bigl(t^5 \bigr), \end{aligned}$$

$$\begin{aligned} \varPhi_4(q,t)=&1+(1+q) t+\bigl(1+2q+2q^2+2q^3+q^4 \bigr) \frac{t^2}{2} \\ &+\bigl(1+3q+6q^2+12q^3+15q^4+12q^5+12q^6+6q^7 \bigr) \frac{t^3}{6} \\ &+\bigl(1+4q+12q^2+36q^3+78q^4+120q^5 \\ & \quad+ 156q^6+168q^7+150q^8+120q^9+48q^{10} \bigr) \frac{t^4}{24}+O\bigl(t^5\bigr), \end{aligned}$$

$$\begin{aligned} \varPhi_{\infty}(q,t)=&1+(1+q) t+\bigl(1+2q+2q^2+2q^3+2q^4+2q^5+O \bigl(q^6\bigr)\bigr) \frac{t^2}{2} \\ &+\bigl(1+3q+6q^2+12q^3+18q^4+24q^5+O \bigl(q^6\bigr)\bigr) \frac{t^3}{6} \\ &+ \bigl(1+4q+12q^2+36q^3+84q^4+156q^5+O \bigl(q^6\bigr)\bigr) \frac{t^4}{24}+O\bigl(t^5 \bigr). \end{aligned}$$

By evaluating Φ _m (q,t) at q=1, we can recover the total number of palindromic elements in W(m,n). By Corollary 3.5, this value is finite only when 4≤m<∞. We list these values for 4≤m≤8 and 1≤n≤7 in Fig. 3.

Fig. 3

Number of palindromic elements in W(m,n)

4 Properties of triangle groups in $\operatorname{HQ}$

We discuss a few properties of triangle groups in $\operatorname{HQ}$. The first property is that there are k-palindromic Poincaré polynomials which are not palindromic for large k:

Proposition 4.1

Let W be the triangle group (2,b,c) with S={r,s,t} such that

$$(rs)^2=(rt)^b=(st)^c=e $$

where b,c≥3 and c is finite. Then there exist elements w∈W which are (c−2)-palindromic but not palindromic.

Proof

Consider w=uv where

$$ u^{-1}=stst\ldots\quad\text{and}\quad v=rtstst \ldots $$

(6)

with ℓ(u)<c and ℓ(v)≤c. Calculation of the polynomial P _w (q) reduces to determining the cardinality of the sets

$$M_k:=\bigl\{w'\leq w\bigm|\ell\bigl(w' \bigr)=k\bigr\}. $$

First we partition

$$M_k=(M_k\cap W_{\{s,t\}}) \sqcup (M_k\cap W\backslash W_{\{s,t\}}). $$

If w′∈W _{s,t}, then w′ has the form sts⋯ or tst⋯. Hence

$$|M_k\cap W_{\{s,t\}}|= \begin{cases} 2\quad\text{if}\ k<\min\{c,\ell(w)-2\},\\ 1\quad\text{if}\ k=c\ \text{or}\ \ell(w)-1,\\ 0\quad\text{if}\ k>c. \end{cases} $$

If w′∈W∖W _{s,t}, then it is uniquely determined by its parabolic decomposition w′=u′v′ where u′≤u,v′≤v and v′ is non-trivial in W ^{s,t}. Hence

$$W\backslash W_{\{s,t\}}\simeq[e,u]\times\bigl([r,v]\cap W^{\{s,t\}} \bigr). $$

This gives

$$|M_k\cap W\backslash W_{\{s,t\}}|= \begin{cases} 2k-1 & \text{if}\ k\leq\min\{\ell(u),\ell(v)\},\\ 2\ell(v) & \text{if}\ \ell(v)<k\leq\ell(u),\\ 2\ell(u) & \text{if}\ \ell(u)<k\leq\ell(v),\\ 2\ell(w)-2k+1 & \text{if}\ k\geq\max\{\ell(u),\ell(v)\}. \end{cases} $$

In the case that ℓ(u)=ℓ(v)=c−1, we have

$$P_w(q)=\bigl[\ell(w)+1\bigr]_q+q^{c+1}+\sum _{k=1}^{c} 2q^{k}\bigl[ \ell(w)-2k+1\bigr]_q. $$

In particular, if we write P _w (q)=∑a _i q ⁱ, then we have

$$a_i=a_{\ell(w)-i}=2i+1 $$

for i≤c−3 and

$$a_{c-2}=2c-1,\qquad a_{c}=2c-2. $$

Hence P _w (q) is (c−2)-palindromic but not palindromic. For example, if we take c=4 and w=uv=(sts)(rts), then

$$[e,w]\cap W_{\{s,t\}}=\{e,s,t,st,ts,tst,sts,stst\} $$

and

$$[e,w]\cap W\backslash W_{\{s,t\}}=[e,u]\cdot\bigl([r,v]\cap W^{\{s,t\}}\bigr)=\{ e,s,t,st,ts,sts\}\cdot\{r,rt,rts\}. $$

In this case, the Poincaré polynomial P _w (q)=1+3q+5q ²+7q ³+6q ⁴+3q ⁵+q ⁶ is 2-palindromic but not 3-palindromic. □

It is tempting to conjecture that, for the triangle groups (2,b,c) as in Proposition 4.1, all (c−1)-palindromic elements are palindromic. However, for triangle group (2,3,5) (Coxeter type H ₃ with c=5) there is a unique length 14 element which is 4-palindromic but not palindromic given by w=tsrtsrtsrtsrtr.

Theorem 3.1 states that any Grassmannian factorization of a 2-palindromic element w∈W is also a BP-decomposition if W avoids triangles in $\operatorname{HQ}$. This statement is not true for Coxeter groups which contain triangles in $\operatorname{HQ}$.

Proposition 4.2

Let W be a Coxeter group. Then W avoids all triangle groups in $\operatorname{HQ}$ if and only if every Grassmannian factorization w=uv where w is palindromic is a BP-decomposition.

Proof

By Theorem 3.1, it suffices to show that for triangle groups (2,b,c) as in Proposition 4.1 there are Grassmannian factorizations w=uv of palindromic w which are not BP-decompositions. Consider w=uv as in Eq. (6) with ℓ(u)=2 and ℓ(v)=c=m _st . It is easy to check that w is palindromic and that w=uv is a Grassmannian factorization with respect to J={s,t} but not a BP-decomposition. □

5 Descent sets of triangle avoiding groups

In this section, we prove several basic properties of Coxeter groups which avoid triangle groups in $\operatorname{HQ}$. We begin with a lemma that holds for all Coxeter groups:

Lemma 5.1

Let W be a Coxeter group and u∈W. If s∉D _L (u), then D _L (su)∖{s} consists of the elements t∈D _L (u) such that u has a reduced factorization starting with a braid tsts⋯ of length m _st −1. (If m _st =2 then this braid consists of only one element.) In other words,

$$ D_L(su) = \{s\} \cup\bigl\{t \in D_L(u) : u = u_0 u_1, u_0 \in W_{\{s,t\}}, u_1 \in W^{\{s,t\}}, \ell(u_0) = m_{st} - 1\bigr\}. $$

Proof

Let J=D _L (su). Then by [6], W _J is a finite Coxeter group and su has a reduced factorization beginning with the maximal element w ₀ of W _J . If t is an element of J∖{s}, then m _st <∞ and w ₀ has a reduced decomposition starting with the longest element of W _{s,t}. □

We now consider Coxeter groups which avoid triangle groups in $\operatorname{HQ}$.

Lemma 5.2

If W is a Coxeter group which avoids all triangle groups in $\operatorname{HQ}$, then the only finite parabolic subgroups of W are products of rank 2 Coxeter groups.

In other words, if J⊂S is such that W _J is finite, then J can be written as a disjoint union

$$J=\bigsqcup_{i} J_i, $$

where |J _i |≤2 for all i, and m _st =2 if s∈J _i , t∈J _j , i≠j.

Proof

Using the classification of finite Coxeter groups, we see that every finite irreducible Coxeter group of rank ≥3 contains a triangle group in $\operatorname{HQ}$. □

If J=D _L (w), then W _J is a finite Coxeter group. In particular, Lemma 5.2 applies to the parabolic subgroups generated by descent sets of $\operatorname{HQ}$-avoiding Coxeter groups. The following lemma is the main result of this section.

Lemma 5.3

Let (W,S) be a Coxeter group which avoids triangle groups in $\operatorname{HQ}$. Let r,s∈S such that 3≤m _rs ≤∞, and suppose u is an element of W such that (rs)u is a reduced factorization. Then

$$ D_L(rsu) \setminus\{r,s\} = \bigl\{t \in D_L(u) : m_{rt} = m_{st} = 2\bigr\}. $$

Proof

The proposition is obviously true if u=e. We proceed by induction on the length of u. Let J=D _L (su), and write J=⨆J _i as in Lemma 5.2. We can further assume that if J _i ={x,y}, then m _xy ≥3, and that s∈J ₀.

Now if t∈D _L (rsu)∖{r,s}, then by Lemma 5.1 we must have m _rt <∞ and rsu must have a reduced decomposition starting the longest element in W _{r,t}. If t∉J ₀ then m _st =2. Since W avoids triangle groups in $\operatorname {HQ}$, we have m _rt =2 as well.

This leaves the possibility that t∈J ₀, in which case m _st ≥3. Once again, since W avoids triangle groups in $\operatorname{HQ}$, we conclude that m _rt ≥3. Thus rsu has a reduced factorization rsu=(rtr)u′, where ℓ(u′)=ℓ(u)−1. Now tru′=su, so s∈D _L (tru′). But by induction, this implies that m _ts =m _rs =2, which is a contradiction. Hence t∉J ₀. □

6 Proof of Theorem 3.1

We now prove Theorem 3.1. The following assumptions are fixed for the remainder of the section. Let W be a Coxeter group that avoids all triangle groups in $\operatorname {HQ}$. Let w∈W be 2-palindromic with a Grassmannian factorization w=uv with respect to J=S(u). By Lemma 2.6 we have

$$ \bigl|u\cdot\mathsf{D}(v)\cap\mathsf{D}(w)\bigr|=1. $$

(7)

This implies

$$ \bigl|W^J\cap\mathsf{D}(v)\bigr|=1, $$

(8)

and in particular, |D _R (v)|=1. Let z∈D _R (v) denote this unique simple reflection. The element vz is the unique element in W ^J∩D(v).

We divide the proof into three steps. The first step is to prove that S(v) has at most three elements. Second, we prove the characterization of v when S(v) has exactly three elements. For the last step, we show that w=uv is BP-decomposition. We begin with the following technical lemma.

Lemma 6.1

Let s ₁,…,s _k be the longest sequence of distinct simple reflections such that v has a reduced decomposition

$$ v = s_{1} \cdots s_{k} v', $$

and for all j<k, $m_{s_{j} s_{j+1}} \geq3$. For any 1≤j≤k, define the set I _j :={s ₁,…,s _j }. Then:

• (1) $s_{j}\cdots s_{k}v'\in W^{I_{j-1}}$ for all j≤k, and

• (2) S(v′)⊆{s ₁,…,s _k }.

Proof

Clearly the lemma is true if ℓ(v)=1 and hence we assume that ℓ(v)≥2. Observe that k≥2, otherwise v∉W ^J. For any j≤k, let

$$v=v_jv_j' $$

be a parabolic decomposition with respect to $W_{I_{j}}$. It is easy to see that v ₁=s ₁ and hence $v_{1}'=s_{2}\cdots s_{k}v'\in W^{I_{1}}$.

Now let j≥2 and suppose that ℓ(v _j )>j. Then there exists s∈D _R (v _j ) such that

$$\mathsf{S}(v_js)=\mathsf{S}(v_j). $$

By Lemma 5.3, we have the left descent sets

$$D_L\bigl(v_jsv_j' \bigr)=D_L(v_js)\cup\bigl\{t\in D_L \bigl(v_j'\bigr)\bigm|m_{ts_i}=2\ \text{for}\ i \leq j\bigr\} $$

and

$$D_L(v)=D_L\bigl(v_jv_j' \bigr)=D_L(v_j)\cup\bigl\{t\in D_L \bigl(v_j'\bigr)\bigm|m_{ts_i}=2\ \text{for}\ i\leq j\bigr\}. $$

Since S(v _j s)=S(v _j ), the descent sets above are equal. Hence $v_{j}sv_{j}'\in W^{J}\cap\mathsf{D}(v)$. If j<k, then $v_{j}'\neq e$ and consequently $v_{j}sv_{j}'\neq vz$, contradicting Eq. (8). Thus ℓ(v _j )=j which implies that v _j =s ₁⋯s _j and $v_{j}'=s_{j+1}\cdots s_{k}v'\in W^{I_{j}}$. This proves part (1) of the lemma.

For part (2), suppose that S(v′)⊈{s ₁,…,s _k }. Then |S(v)|>k and $v_{k}'\neq e$. We get v _k =s ₁⋯s _k and

$$D_L\bigl(v_k'\bigr)\cap \{s_1,\ldots,s_k\}=\emptyset $$

since $v_{k}'\in W^{I_{k}}$. By the maximality of k, we have $m_{s_{k},r}=2$ for all $r\in D_{L}(v_{k}')$. We claim that

$$D_L\bigl(s_1\cdots s_{k-1}v' \bigr)=D_L(v). $$

Indeed, if k≥3, this follows from Lemma 5.3. Otherwise, if k=2, then

$$m_{s_1s_2}=m_{s_1r}=\infty $$

since W avoids all triangle groups in $\operatorname{HQ}$. This proves the claim when k=2. In either case we have

$$s_1\cdots s_{k-1}v'\in W^J\cap \mathsf{D}(v) $$

which contradicts Eq. (8). Therefore S(v′)⊆{s ₁,…,s _k }. □

The following proposition completes the first step of in the proof of Theorem 3.1.

Proposition 6.2

We have that |S(v)|≤3. Furthermore, if |S(v)|=3, then S(v) generates a triangle group (a,b,c) with a,b,c≥3.

Proof

Suppose |S(v)|≥4 and let v=s ₁⋯s _k v′ as in Lemma 6.1. We first show by induction on j that

• (1) D _L (s _j ⋯s _k v′)={s _j },

• (2) $m_{s_{i} s_{j}}=2$ for i∈{1,…,j−2}.

Indeed, part (1) is trivial for j=k. Suppose part (1) is true for some j≤k. Now by Lemma 6.1, s ₁⋯s _j−2 s _j ⋯s _k v′ is reduced, and therefore is not an element of W ^J. So by Lemma 5.3, we have s _j ∈D _L (s ₁⋯s _j−2 s _j ⋯s _k v′). Moreover, if j≥4, then $m_{s_{i} s_{j}} = 2$ for all 1≤i≤j−2. If j=3, then s ₁ s ₃⋯s _k v′ has a reduced expression beginning with a braid s ₁ s ₃ s ₁⋯ of length $m_{s_{1} s_{3}} < \infty$. Since s ₁∉D _L (s ₄⋯s _k v′), we conclude that $m_{s_{1} s_{3}} = 2$. Hence part (2) holds for j.

Now suppose part (2) holds for all j>j ₀. Since |D _L (v)|=1, we have $s_{j}\notin D_{L}(s_{j_{0}} \cdots s_{k} v')$ for any j>j ₀. Thus part (1) holds for j ₀. Hence (1) and (2) hold for all j.

Now part (2), combined with the $\operatorname{HQ}$-avoiding condition, implies that

$$ m_{s_i s_j} = \begin{cases} \infty& |i-j| = 1, \\ 2 & |i-j| \geq2. \end{cases} $$

In other words, if |S(v)|≥4 then W _S(v) is defined entirely by commuting relations. We show that this hypothesis implies that |S(v)|≤2. Indeed, suppose |S(v)|≥3, and let u=u ₁ u ₀, where u ₁∈^S(v) W and u ₀∈W _S(v). Here the set ^S(v) W denotes the minimal length representatives of the left cosets W/W _S(v). By Eq. (7), the product u ₀ s ₂⋯s _k v′ must not be reduced. We conclude that D _R (u ₀)∩D _L (s ₂⋯s _k v′)={s ₂} since for any s _i ,s _j ∈S(u ₀)∪S(v)=S(v) we have $m_{s_{i}s_{j}}= 2$ or ∞. Moreover, since D _L (s ₁ s ₃⋯s _k v′)={s ₁,s ₃}, the same argument shows that s ₃∈D _R (u ₀). But now we have {s ₂,s ₃}⊆D _R (u ₀) which implies that the $m_{s_{2},s_{3}}$ is finite. This contradicts the fact that $m_{s_{2},s_{3}} =\infty$. Hence, |S(v)|≤3.

Finally, if |S(v)|=3, then by Lemma 6.1, $m_{s_{1}s_{2}},m_{s_{2}s_{3}}\geq3$. If $m_{s_{1}s_{3}}=2$, then the $\operatorname{HQ}$-avoiding condition implies $m_{s_{1}s_{2}}=m_{s_{2}s_{3}}=\infty$. We can now apply the previous argument as above to show that {s ₂,s ₃}⊆D _R (u ₀) and hence $m_{s_{2},s_{3}}$ is finite. Thus we must have $m_{s_{1}s_{3}}\geq3$. This completes the proof. □

For the next step in the proof of Theorem 3.1, suppose that |S(v)|=3 with S(v)={r,s,t}. By Proposition 6.2, we have m _rs ,m _rt ,m _st ≥3. Consider the reduced factorization v=xy where

$$x^{-1}:=tsrtsr\cdots $$

is the largest spiral word prefix of v. In other words, we can write

$$ v=xy=(\cdots rstrst)\cdot y. $$

(9)

Define x′:=xtst. It is easy to see that ℓ(x′)=ℓ(x)−1 and that x′ equals x with the second to last reflection s removed. For any 0≤k≤ℓ(x′) define a length k suffix $x_{k}'$ of x′ by

$$x_k':=\underbrace{\cdots rstrt}_k. $$

Lemma 6.3

For any 0≤k≤ℓ(x′), the following are true:

• (1) The product $x_{k}'y$ is a reduced factorization.

• (2) If k is even, then $|D_{L}(x_{k}'y)|=1$. If k is odd, then $|D_{L}(x_{k}'y)|\leq2$.

• (3) If $|D_{L}(x_{k}'y)|=2$ and k≥5, then $|D_{L}(x_{k-2}'y)|=2$.

Proof

If k=0, then r,t∉D _L (y) since x is a maximal length spiral word. This implies that D _L (y)={s}. If k=1, then ty is reduced and D _L (ty)⊆{s,t}. Moreover, rty is reduced and by Lemma 5.3, we have D _L (rty)={r} since r∉D _L (y). This proves the lemma for k≤2.

We proceed with the proof by induction on k. Suppose k≥3. Without loss of generality, we can assume $r\in D_{L}(x_{k}')$, so that s is the first element of $x_{k-1}'$. We first consider the case where k is odd. Then by the inductive assumption, we have $D_{L}(x_{k-1}'y)=\{s\}$. Hence

$$x_k'y=rx_{k-1}'y $$

is reduced and $D_{L}(x_{k}'y)\subseteq\{r,s\}$. If k is even, then s and t are the first two elements of $x_{k-1}'$; in particular, r is not one of the first two elements. Therefore

$$D_L\bigl(x_{k-2}'y\bigr)=\{t\}\quad \text{and}\quad D_L\bigl(x_{k-1}'y\bigr) \subseteq\{s,t\}. $$

So $x_{k}'y$ is reduced and $D_{L}(x_{k}'y)=\{r\}$. This proves parts (1) and (2) of the lemma.

To prove part (3), suppose that k≥5 is odd with

$$t\in D_L\bigl(x_{k-2}'y\bigr)\subseteq\{t,r \}\quad\text{and}\quad r\in D_L\bigl(x_k'y \bigr)\subseteq\{r,s\}. $$

If $|D_{L}(x_{k}'y)|=2$, then $r\in D_{L}(x_{k-2}'y)$ since 3≤m _rs <∞. Hence $|D_{L}(x_{k-2}'y)|= 2$.  □

One immediate consequence of Lemma 6.3 is that x′y is a reduced factorization and that if ℓ(x′) is even, then x′y∈W ^J∩D(v) which is a contradiction to Eq. (8). Hence ℓ(x′) is odd (i.e. ℓ(x) is even). The following lemma is a preliminary characterization of v.

Lemma 6.4

The spiral word x satisfies one of the following conditions:

• (1) m _rt =3 and ℓ(x)=4.

• (2) m _rt =m _rs =3 and ℓ(x)=6.

• (3) m _rt =m _rs =m _st =3 and ℓ(x)≥8.

Proof

Since x′y is reduced, we have x′y∉W ^J and ℓ(x′)≥3. Furthermore, by Lemma 6.3 part (3), $|D_{L}(x_{k}'y)|=2$ for all k≥3. In particular the following statements are true:

• (1) For k=3, we have |D _L (trty)|=2 if and only if m _rt =3.

• (2) For k=5, we have |D _L (rstrty)|=2 if and only if m _rt =m _rs =3.

• (3) For k=7, we have |D _L (strstrty)|=2 if and only if m _rt =m _rs =m _st =3.

This completes the proof. □

Now we consider the reduced factorization

(10)

where k is the length of the longest possible prefix of sty the form stst⋯.

Lemma 6.5

With v as Eq. (10), the following are true:

• (1) $\bar{y}=e$.

• (2) k=m _st −1.

Proof

Suppose that $\bar{y}\neq e$. Then $D_{L}(\bar{y})=\{r\}$ by the maximality of k. If k=2, then x is not a maximal length spiral, and hence $\bar{y}= e$. Now assume that k≥3 and let $v=\bar{x}\bar{z}\bar{y}$ be the reduced factorization given in (10) where $\bar{z}\in W_{\{s,t\}}$ is of length k. Without loss of generality, let $t\in D_{R}(\bar{z})$ and define $\bar{z}':=\bar{z}t$. Since k≥3, we have $\ell(\bar{z}')\geq2$ and thus $\bar{x}\bar{z}'\bar{y}$ is a reduced factorization. Likewise, since $\ell(\bar{z}')\geq2$ and $D_{R}(\bar{x})=\{r\}$, we have $\bar{x}\bar{z}'\bar{y}\in W^{J}$ and hence $\bar{x}\bar{z}'\bar{y}\in W^{J}\cap\mathsf{D}(v)$. But this contradicts Eq. (8). Therefore $\bar{y}= e$ and part (1) of the lemma is proved.

Since ℓ(x) is even, we have k<m _st , otherwise v∉W ^J. This completes the proof in the case of x as in Lemma 6.4 part (3). Now suppose that k≤m _st −2. If x satisfies the condition in Lemma 6.4 part (1), then J=S(w)∖{t} and we can write

$$v=tr\underbrace{stst\cdots}_{\text{length $k$}}=tr\bar{z}. $$

But then $t\bar{z}\in W^{J}\cap\mathsf{D}(v)$ which contradicts Eq. (8). If x satisfies the condition in Lemma 6.4 part (2), then J=S(w)∖{r} and

$$v=rstr\underbrace{stst\cdots}_{\text{length $k$}}=rstr\bar{z}. $$

But then $rst\bar{z}\in W^{J}\cap\mathsf{D}(v)$ which also contradicts Eq. (8). Hence k>m _st −2 and part (2) of the lemma is proved. □

It is easy to see that Lemmas 6.3, 6.4 and 6.5 prove the characterization v when |S(v)|=3 in Theorem 3.1.

The final step in the proof is to show that w=uv is a BP-decomposition. In this step, we do not assume that |S(v)|=3.

Lemma 6.6

For any s ₀∈S(v)∩J, there exists v″∈W ^J of length ℓ(v″)=ℓ(v)−2 such that s ₀ v″∈D(v).

Proof

If |S(v)|≤2, then the lemma is obvious. If |S(v)|=3, then we can write v=xy as in Eq. (9) with the notational change that

$$x=rstrst\cdots. $$

In other words, we let r,s,t denote the first three simple reflections appearing in x, rather than the last three. We want to find v″ for s ₀∈S(v)∩J={s,t}. Note that with the change in notation, we have m _rs =3. Recall the definition of x′ given after Eq. (9). By Lemma 6.3 part (1) we find that x′y is reduced and hence x′y∈D(v)∩W ^{r,s}. Thus we have a reduced factorization

$$x'y=(srs)y' $$

for some y′. For s ₀=s, we set v″=rsy′. Then v″∈W ^J since D _L (v″)={r}.

We now find a v″ for s ₀=t. Consider the reduced factorization

$$v=(rs) \bigl(ty''\bigr). $$

Clearly r∉D _L (ty″), otherwise v∉W ^J. Hence rty″∈D(v) and rty″∉W ^J. This implies that t∈D _L (rty″) and we can write a reduced factorization

$$rty''=(trt)y''' $$

for some y‴. We set v″=rty‴ for s ₀=t. Since D _L (v″)={r}, we get v″∈W ^J. This completes the proof. □

If s ₀∈S(v)∩J and v″∈W ^J, such that s ₀ v″∈D(v), then s ₀∈D _R (u). Otherwise us ₀ v″∈u⋅D(v)∩D(w) which contradicts Eq. (7). Applying Lemma 2.2, we find that w=uv is a BP-decomposition. This completes the proof of Theorem 3.1.