1 Introduction

Motivated by the “constrained compositions” introduced by Andrews–Paule–Riese [1], Beck–Gessel–Lee–Savage [3] enumerated symmetrically constrained compositions, i.e., compositions of an integer M into n nonnegative parts

$$M = \lambda_1 + \lambda_2 + \cdots+ \lambda_n , $$

where the sequence λ:=(λ 1,λ 2,…,λ n ) satisfies the symmetric system of linear inequalities

$$a_1 \lambda_{ \pi(1) } + a_2 \lambda_{ \pi(2) } + \cdots+ a_n \lambda_{ \pi(n) } \ge0 \quad\text{for\ all}\ \pi\in S_n . $$

Specifically, [3] discusses various approaches to compute, for a fixed set of parameters a 1,a 2,…,a n , the generating functions

$$F ( z_1, z_2, \dots, z_n ) :=\sum _\lambda z_1^{ \lambda_1 } z_2^{ \lambda_2 } \cdots z_n^{ \lambda_n } $$

and

$$F(q) :=F(q,q,\dots,q) = \sum_\lambda q^{ \lambda_1 + \lambda_2 + \cdots+ \lambda_n } , $$

where both sums extend over all symmetrically constrained compositions λ. One viewpoint of [3] is geometric: The compositions (λ 1,λ 2,…,λ n ) are interpreted as integer lattice points in the cone

$$ \bigl\{ x \in\mathbf{R}^n \bigm| \forall\sigma\in W : ( \sigma x, a) \ge0 \bigr\} , $$
(1)

where W is the image of the permutation representation of S n , a=(a 1,…,a n ), and ( , ) is the standard inner product on R n. This viewpoint together with permutation statistics of S n gave rise to explicit (and in some instances surprising) generating function formulas.

Our goal is to generalize the results in [3] to cones of the form (1) where W is another reflection group. In addition to obtaining general multivariate generating function identities, we obtain several applications of these results for hyperoctahedral groups. These applications are similar in spirit to the applications in the symmetric-group case found in [3].

The outline of our paper is as follows. The general setup for our approach is discussed in the next section, which also contains our central result, Theorem 2.8. Section 3 illustrates our approach by re-deriving the main result in [3]. Sections 4 and 5 consider cones constrained by reflection groups of type B and D, respectively. Further, Sect. 4 contains applications obtained through specializing our generating functions in the type-B case.

2 General theory

Our goal in this section is to study integer points in cones that are constrained by the orbit of a single linear constraint under an appropriate group action on real space. This goal is realized in Theorem 2.8, where the multivariate generating function encoding the integer points in such a cone is expressed as a sum of simpler generating functions. Theorem 2.8 is an algebraic consequence of a geometric triangulation of the symmetric cone, which we obtain in Lemma 2.5. Proposition 2.6 makes the triangulation disjoint by using combinatorics of Coxeter groups as a tiebreaker for the walls separating the maximal cones in the triangulation. This is critical for our subsequent applications.

2.1 Almost irreducible finite reflection groups, Coxeter groups, and descents

In the following, we will consider finite reflection groups (i.e., finite subgroups of O(V) for some Euclidean space V that are generated by reflections—see, e.g., [10] for background) that act on the underlying Euclidean space in a restricted fashion. Namely, a finite reflection group W⊂O(V) acting on a Euclidean vector space V is called almost irreducible if V decomposes into W-invariant subspaces V=V 1⊕V 2 such that W acts irreducibly and nontrivially on V 1 and trivially on V 2, and that V 2 is 1-dimensional.

Example 2.1

S n acts almost irreducibly on R n by permutation of the components. The irreducible summand consists of all vectors with component sum 0, and the trivial summand consists of all vectors with equal components. This is the case considered in [3].

Example 2.2

Let V 1 be a Euclidean vector space and W⊂O(V 1) a nontrivial irreducible reflection group, i.e., a nontrivial reflection group such that V 1 does not contain any nontrivial proper W-invariant subspaces. Let W act trivially on R and set V=V 1⊕R. Then W acts almost irreducibly on V.

A Coxeter group of rank r is a group admitting a presentation with generators s 1,…,s r and relations \((s_{j}s_{k})^{m_{jk}} = 1\) for m jk ∈{1,2,3,…}∪{∞} subject to the conditions that m jk =m kj and m jk =1⇔j=k. Here, a value of m jk =∞ is to be understood as the absence of the corresponding relation. Such generators are called simple generators. For each Coxeter group considered, we will suppose that simple generators have been fixed once and for all. We refer the reader to [5] or [10] for further information regarding Coxeter groups and their relation to reflection groups.

The length l(σ) of an element σ∈W of a Coxeter group W is the smallest integer such that there is a decomposition \(\sigma= s_{j_{1}} \cdots s_{j_{l(\sigma)}}\) of σ as a product of l(σ) not necessarily distinct simple generators. For any σ∈W, the right descent set of σ is

$$ D_\mathrm{r}(\sigma) :=\bigl\{ j \in\{ 1, \ldots , r \} \mid l(\sigma s_j) < l(\sigma) \bigr\} . $$
(2)

Remark 2.3

Propositions 3.1, 4.1, and 5.1 review the connection between the definition of descent given here and definitions of descent for Coxeter groups of types A, B, and D in terms of the one-line notation.

Recall that if W is a finite reflection group, it is automatically a Coxeter group. Simple generators can be found as follows. Let \(\mathcal{H}\) be the union of all reflection hyperplanes for W; denote by F the closure of a connected component in \(V\backslash \mathcal{H}\). It is immediate that F is a convex polyhedral cone. Let H 1,…,H r be the facet hyperplanes of F and let s i be the reflection at H i . Then s 1,…,s r are simple generators of the Coxeter group W. See [6, V.3.2, Theorem 1] for the proof of these statements.

A subset F⊂V is a fundamental domain for W if F is the closure of an open set and each W-orbit intersects F in exactly one point. By [10, Sect. I.12], every such F is polyhedral, and is bounded by hyperplanes fixed by a set of simple reflections in W. Through the rest of this paper, when given a set of simple generators s 1,…,s r of a reflection group W, we denote by F a fixed fundamental domain with bounding hyperplanes corresponding to s 1,…,s r .

2.2 Triangulations of monoconditional cones

Denote the value of a linear form φ∈V ∗ on a vector x∈V by 〈x,φ〉. Let W⊂O(V) be an almost irreducible reflection group. A symmetric cone C⊂V is a convex polyhedral cone that is W-invariant. A symmetric cone is called monoconditional if there is a linear form φ∈V ∗, such that \(V_{1}, V_{2} \not\subset\ker (\varphi )\) and

$$ C = \bigl\{ x \in V \mid\forall\sigma\in W : \langle\sigma x, \varphi\rangle\geq0 \bigr\} . $$
(3)

This generalizes (1).

Example 2.4

The positive orthant \(\mathbf{R}_{\geq0}^{n}\) is a monoconditional symmetric cone for the almost irreducible action of S n on R n by permutation of the components. A possible linear form defining it is the projection on the first component.

Recall that a convex polyhedral cone is pointed if it does not contain a line, and it is simplicial if it is the set of linear combinations with nonnegative coefficients of a set of linearly independent vectors. A triangulation of a cone C is a finite collection T of simplicial cones such that C is the union of the elements of T and for any Δ1,Δ2∈T, Δ1∩Δ2 is a face common to both Δ1 and Δ2.

Lemma 2.5

Let W⊂O(V) be an almost irreducible reflection group. Let C⊂V be a monoconditional symmetric cone. Then C is pointed. Let F⊂V be a fundamental domain for the action of W on V. Then the cone C +:=C∩F is simplicial. In particular, C admits the triangulation

$$C = \bigcup_{\sigma\in W} \sigma C_+ . $$

Proof

Some of the notation used in this proof is shown in Fig. 1 for convenience. Let φ∈V ∗ be a linear form defining C as in (3); note that ( , ) is a W-invariant inner product on V. Let x φ ∈V such that 〈x,φ〉=−(x,x φ ) for all x∈V. Let x 0 be the unique element of Wx φ ∩F. Then

$$C_+ = \bigl\{ x \in F \mid\forall\sigma\in W : (\sigma x, x_0) \leq0 \bigr\} . $$

Let V=V 1⊕V 2 be the decomposition of V into the irreducible and trivial component. Let \(x_{0} = x_{0}' + x_{0}''\) with \(x_{0}' \in V_{1}\) and \(x_{0}'' \in V_{2}\). By definition \(V_{2} \not\subset\ker(\varphi)\), and so x φ ∉V 1, thus x 0∉V 1, and hence \(x_{0}'' \neq0\). As φ is only determined up to multiplication by a positive scalar, suppose without loss of generality that \((x_{0}'', x_{0}'') = 1\). Note that every nonzero element of C + has a nonzero V 2-component; otherwise, for some x∈C + it would hold that \((\sigma x, x_{0}') \leq(\sigma x,x_{0}'')=0\) for all σ∈W. This would imply that \((x,\sigma x_{0}')\leq0\) for all σ∈W, which leads to a contradiction since x≠0 and \(\sum_{\sigma\in W}\sigma x_{0}' = 0\) by the irreducibility of the action of W on V 1.

Fig. 1
figure 1

Notation used in the proof of Lemma 2.5. The cyclic group of order 2 acts almost irreducibly on R 2 by sign change in the first component

Full size image

This implies that every nonzero element of C has a nonzero V 2-component, in particular, C is pointed.

Let

$$P_+ = \bigl\{ x \in F \cap V_1 \mid\forall\sigma\in W : \bigl( \sigma x, x_0'\bigr) \leq1 \bigr\} . $$

Consider the reflections at the facet hyperplanes of F as simple generators of W. Let l denote the corresponding length function. Let x∈F∩V 1 and σ∈W. Let H be a facet hyperplane of σF, such that l(sσ)>l(σ) for the reflection s at H. We claim that in this situation

$$ \bigl(s \sigma x, x_0'\bigr) \leq \bigl(\sigma x, x_0'\bigr) . $$
(4)

Indeed, consider the decomposition V 1=(H∩V 1)⊕H ⊥. According to this decomposition, write \(x_{0}' = v_{0} + w_{0}\) and σx=v 1+w 1; then sσx=v 1−w 1. We have \((\sigma x, x_{0}') = (v_{1}, v_{0}) + (w_{1}, w_{0})\) and \((s \sigma x, x_{0}') = (v_{1}, v_{0}) - (w_{1}, w_{0})\). Hence

$$ \bigl(s \sigma x, x_0'\bigr) = \bigl(\sigma x, x_0'\bigr) - 2(w_1,w_0) . $$
(5)

Generally, if τ∈W, then l(τ) equals the number of reflection hyperplanes between F and τF. As l(sσx)>l(σx), this implies that x 0 and σx lie on the same side of H. Hence (w 1,w 0)≥0, and so (5) implies the claim (4).

By induction on l(σ), (4) implies that

$$P_+ = \bigl\{ x \in F \cap V_1 \bigm|\bigl(x, x_0' \bigr) \leq1 \bigr\} . $$

The cone C + is the cone over \(P - x_{0}''\), thus C +={x∈F∣(x,x 0)≤0}. The cone F∩V 1 is a fundamental domain for the action of W on V 1. Hence F∩V 1 is simplicial, as it is defined by the hyperplanes corresponding to the simple generators of W, and there are dim(V 1) many such hyperplanes. We have F=(F∩V 1)+V 2, and so dim(F)=dim(F∩V 1)+1. The cone C + is defined in F by the single additional inequality (x,x 0)≤0, thus C + is simplicial. □

Consider the situation of Lemma 2.5. Choose an order H 1,…,H n−1 of the facet hyperplanes of F with the corresponding simple reflections s 1,…,s n−1. For any subset J⊂{1,…,n−1}, let

$$C_J :=C_+ \Big\backslash\bigcup_{j \in J} H_j . $$

For example, C ∅=C +. If J≠∅ some of the facets of C + are removed.

Proposition 2.6

In the situation of Lemma 2.5, C decomposes as a disjoint union

$$C = \biguplus_{\sigma\in W} \sigma C_{D_\mathrm{r}(\sigma)} . $$

Proof

For x∈C, let

$$W(x) :=\{ \sigma\in W \mid x \in\sigma C_+ \} . $$

Let x +∈Wx∩C + be the (unique) W-conjugate of x contained in the fundamental domain C +. Let \(\tilde{W} := \{ \sigma\in W \mid\sigma x_{+} = x_{+} \}\) be the isotropy subgroup of x + in W. By Lemma 2.5, C=⋃ σ∈W σC +, and so the set W(x) is nonempty. Fix any σ∈W(x).

We claim that \(W(x) = \sigma\tilde{W}\). Indeed, it is obvious that \(W(x) \supseteq\sigma\tilde{W}\). For the opposite inclusion, let \(\bar{\sigma}\in W(x)\). Then \(\bar{\sigma}^{-1} x \in C_{+}\), so \(\bar{\sigma}^{-1} = x_{+}\). By the same argument σ −1 x=x +, so \(\sigma^{-1} \bar{\sigma}\in\tilde{W}\). Hence \(\bar{\sigma}= \sigma\sigma^{-1} \bar{\sigma}\in\sigma\tilde{W}\), proving the claim.

By [10, Theorem 1.12(c)], the isotropy subgroup \(\tilde{W}\) is generated by the reflections it contains. So \(\tilde{W}\) is a parabolic subgroup of W. Hence \(W(x) = \sigma\tilde{W}\) contains a unique element of minimal length [10, §1.10], denoted by σ x .

Assume that \(\sigma_{x}^{-1}x \in H_{j}\) for some j∈D r(σ x ). Then \(s_{j} \sigma_{x}^{-1} x = \sigma_{x}^{-1}x\), and so \(x = \sigma_{x} s_{j} \sigma_{x}^{-1} x\). As \(\sigma_{x}^{-1} x \in C_{+}\), this implies that σ x s j ∈W(x). On the other hand l(σ x s j )<l(σ x ), a contradiction. Hence \(\sigma_{x}^{-1} x \notin H_{j}\) for all j∈D r(σ x ). Hence \(\sigma_{x}^{-1} x \in C_{D_{\mathrm{r}}(\sigma_{x})}\), and so \(x \in\sigma_{x} C_{D_{\mathrm{r}}(\sigma_{x})}\). This proves that \(C = \bigcup_{\sigma\in W} \sigma C_{D_{\mathrm{r}}(\sigma)}\).

To prove disjointness, let \(x \in\sigma C_{D_{\mathrm{r}}(\sigma)}\) for some σ∈W. We have to show that σ=σ x . Clearly σ∈W(x). It remains to show that σ has minimal length in W(x). Assume that σ has not minimal length in W(x). Then there is j∈{1,…,n−1} such that l(σs j )<l(σ) and σs j ∈W(x) [10, §1.10]. From σ∈W(x) we conclude that σ −1 x∈C + and from σs i ∈W(x) that s j σ −1 x∈C +. Hence σ −1 x∈H j . Since l(σs j )<l(σ) we have j∈D r(σ). Hence \(\sigma^{-1}x \notin C_{D_{\mathrm{r}}(\sigma)}\), and so \(x \notin \sigma C_{D_{\mathrm{r}}(\sigma)}\), a contradiction. □

Our proposition above is reminiscent of the theory of P-partitions [12, 13]. For a given finite poset P, one can produce a cone of P-partitions. The standard approach to studying a P-partition cone, originating in the work of Stanley referenced above, is to recognize that each such cone is a disjoint union of various chambers of the type A braid arrangement where each chamber has some of its facets removed. The removal of facets in a given chamber is controlled by the descent statistic for the permutation indexing that chamber. Thus, each P-partition cone admits a unimodular triangulation of form similar to Proposition 2.6.

2.3 Generating functions for monoconditional cones

Let \(V^{*}_{\mathbf{C}} = V^{*} \otimes_{\mathbf{R}} \mathbf{C}\). Extend 〈 , 〉 to \(V \times V^{*}_{\mathbf{C}}\) by C-linearity in the second argument. Let Γ⊂V be a lattice and S⊂V. Suppose that there is a nonempty open subset \(B \subset V^{*}_{\mathbf{C}}\) such that the series ∑ x∈S∩Γ e −〈x,φ〉 converges for φ∈B and has a meromorphic continuation to \(V^{*}_{\mathbf{C}}\). We denote this continuation by f S and call it the generating function of S with respect to Γ.

Example 2.7

If C⊂V is a cone, let

$$C^\vee:=\bigl\{ \varphi\in V^* \bigm|\forall x \in C : \langle x, \varphi \rangle\geq0 \bigr\} \subset V^* $$

be its dual cone. The complexified dual of C is defined as

$$C^\vee_\mathbf{C} :=\bigl\{ \varphi\in V^*_\mathbf{C} \bigm|\forall x \in C : \Re\bigl(\langle x, \varphi\rangle\bigr) \geq0 \bigr\} = C^\vee+ i V^* . $$

Let C⊂V be a pointed cone, rational with respect to Γ. Then ∑ x∈C∩Γ e −〈x,φ〉 converges on the interior of \(C^{\vee}_{\mathbf{C}}\) and has a meromorphic continuation f C to \(V^{*}_{\mathbf{C}}\); see, e.g., [2, Chap. 13].

From now on, suppose that W is crystallographic, i.e., that there is a W-invariant lattice Γ in V. A full-dimensional simplicial cone C⊂V is called unimodular (with respect to Γ) if it is generated by a basis of Γ. These generators are called primitive.

Theorem 2.8

In the situation of Lemma 2.5, suppose that C + is unimodular with respect to Γ. Let b 1,…,b n be the primitive generators of C +, enumerated in the unique way such that b j ∉H j for j∈{1,…,n−1}. Then the generating function of C is

$$f_C(\varphi) = \sum_{\sigma\in W} \frac{\prod_{j \in D_\mathrm{r}(\sigma)} e^{- \langle\sigma b_j, \varphi\rangle}}{(1-e^{- \langle\sigma b_1, \varphi\rangle}) \cdots(1-e^{- \langle\sigma b_n, \varphi\rangle })} . $$

In practice, Γ is often endowed with a distinguished basis. In this case, it is often more convenient to work with the following formulation.

Corollary 2.9

In the situation of Theorem 2.8, let e 1,…,e n be a basis of Γ. Define coordinates z j on \(V^{*}_{\mathbf{C}}\) by \(z_{j}(\varphi) := e^{- \langle e_{j}, \varphi\rangle}\). For a=a 1 e 1+⋯+a n e n ∈Γ, let \(z^{a} := z_{1}^{a_{1}} \cdots z_{n}^{a_{n}}\). Then

$$f_C = \sum_{\sigma\in W} \frac{\prod_{j \in D_\mathrm{r}(\sigma)} z^{\sigma b_j}}{(1-z^{\sigma b_1}) \cdots(1-z^{\sigma b_n})} . $$

Proof of Theorem 2.8

Since C is pointed (by Lemma 2.5), its generating series converges on a nonempty domain and the generating series of all W-conjugates of C + converge there. As C + is unimodular with primitive generators b 1,…,b n , its generating function is

$$f_{C_+}(\varphi) = \frac{1}{(1-e^{- \langle b_1, \varphi\rangle}) \cdots (1-e^{- \langle b_n, \varphi\rangle})} . $$

With unimodularity it also follows that each generator of C + is only one lattice hyperplane away from the opposite facet. Hence

$$C_{\{j\}} \cap\varGamma= (C_+ \backslash H_j) \cap\varGamma= (C_+ \cap\varGamma) + b_j $$

for all j∈{1,…,n−1}. More generally, C J ∩Γ=(C +∩Γ)+∑ j∈J b j for any J⊂{1,…,n−1}. Applying this observation to J=D r(σ) for a σ∈W and rephrasing it in terms of generating functions, one obtains

$$f_{C_{D_\mathrm{r}(\sigma)}}(\varphi) = \frac{\prod_{j \in D_\mathrm{r}(\sigma)} e^{- \langle b_j, \varphi\rangle}}{(1-e^{- \langle b_1, \varphi\rangle}) \cdots(1-e^{- \langle b_n, \varphi\rangle})} . $$

Hence for all σ∈W it follows that

$$f_{\sigma C_{D_\mathrm{r}(\sigma)}}(\varphi) = \frac{\prod_{j \in D_\mathrm{r}(\sigma )} e^{- \langle\sigma b_j, \varphi\rangle}}{(1-e^{- \langle\sigma b_1, \varphi\rangle}) \cdots(1-e^{- \langle\sigma b_n, \varphi\rangle })} . $$

By Proposition 2.6, \(f_{C} = \sum_{\sigma\in W} f_{\sigma C_{D_{\mathrm{r}}(\sigma)}}\), which proves the formula. □

3 Cones with the symmetry of a simplex

Theorem 2.8 specializes to more concrete identities once we fix a particular almost irreducible reflection group W. The case of W being the group of symmetries of a simplex has been treated in [3]. We include this case here to show how the result can be derived from Theorem 2.8.

Let S n denote the group of permutations of the set {1,…,n}. For π∈S n , we define the descent set of π as

$$ D(\pi) :=\bigl\{ j \in\{ 1, \ldots, n-1 \} \mid \pi(j) > \pi(j+1) \bigr\} . $$
(6)

This is the standard definition used in the literature on permutations.

The group S n acts on R n by permutation of the components. For π∈S n , let σ π ∈O(R n) denote the transformation by which π acts on R n. Let W={σ π ∣π∈S n }⊂O(R n). Then W is the group of symmetries of the (n−1)-dimensional standard simplex. For j=1,…,n−1, let s j ∈W be the transposition of the jth and (j+1)st component in R n. Then s 1,…,s n−1 are simple generators of W.

The following shows that the definitions of descent given in (2) and (6) agree.

Proposition 3.1

[5, Proposition 1.5.3]

D r(σ π )=D(π) for all π∈S n .

Our main result in this section is the following.

Proposition 3.2

[3, Theorem 1]

Fix integers a 1≤⋯≤a n such that a 1+⋯+a n =1. Let

$$C :=\bigl\{ x \in\mathbf{R}^n \bigm|\forall\pi\in S_n : a_1 x_{\pi(1)} + \cdots+ a_n x_{\pi(n)} \geq0 \bigr\} . $$

Let Σ j :=a 1+⋯+a j for j∈{1,…,n−1}. The generating function of C with respect to Z n is

$$f_C = \frac{1}{1-z_1 \cdots z_n} \sum_{\pi\in S_n} \frac{\prod_{j \in D(\pi)} (z_1 \cdots z_n)^{-\varSigma_j} \prod_{i=1}^j z_{\pi (i)}}{\prod_{j=1}^{n-1} (1 - (z_1 \cdots z_n)^{-\varSigma_j} \prod _{i=1}^j z_{\pi(i)} )} . $$

Note that the condition on the a i to be increasing is a normalization rather than a restriction.

Proof

The cone C is symmetric and monoconditional for W. Let F={x∈R n∣x 1≥⋯≥x n }, a fundamental domain for W. Then our chosen simple generators s 1,…,s n−1 of W are the reflections at the facet hyperplanes of F. Let x 0=(−a 1,…,−a n )∈F. By the proof of Lemma 2.5,

where

The determinant of A is a 1+⋯+a n =1, i.e., A is unimodular and so is C +. Let b 1,…,b n be the primitive generators of C +, enumerated in the unique way such that b j ∉H j for j∈{1,…,n−1}. Then by Corollary 2.9, the generating function of C is

$$f_C = \sum_{\sigma\in W} \frac{\prod_{j \in D_\mathrm{r}(\sigma)} z^{\sigma b_j}}{(1-z^{\sigma b_1}) \cdots(1-z^{\sigma b_n})} . $$

Proposition 3.2 follows once we describe D r(σ), b j , and the action of W explicitly. The inverse of A is

where \(\varSigma_{j}' :=1 - \varSigma_{j}\). Then b j is the jth column vector of B. Let

$$b_{ij} :=\begin{cases} 1 & \text{if}\ j = n, \\ 1-\varSigma_j & \text{if}\ i \leq j < n, \\ -\varSigma_j & \text{if}\ j < i \end{cases} $$

be the ith component of b j , i.e., the (i,j)th component of B. As defined above, with π∈S n we associate σ π ∈O(n) by σ π e i =e π(i). Then W={σ π ∣π∈S n } and we have D r(σ π )=D(π). Hence

 □

4 Cones with hyperoctahedral symmetry

We now consider the case of cones which are symmetric under the action of a hyperoctahedral group. Let W⊂O(n) be the hyperoctahedral group on the first n−1 components of R n. Let s 1∈W be the sign change in the first component and, for j=2,…,n−1, let s j ∈W be the transposition of the (j−1)st and jth component in R n. Then s 1,…,s n−1 are simple generators of W.

For combinatorial (as opposed to geometric) arguments, it is often more convenient to use the following parameterization of the hyperoctahedral group: For π∈S n−1 and ε∈{±1}n−1, define σ π,ε ∈O(n) by

$$ \sigma_{\pi,\varepsilon}e_i = \varepsilon_i e_{\pi(i)} , $$
(7)

where we use the convention that π(n):=n for π∈S n−1 and ε n :=1 for ε∈{±1}n−1. Then W={σ π,ε ∣π∈S n−1,ε∈{±1}n−1}. Let B n−1 denote the set S n−1×{±1}n−1, endowed with the group structure such that σ:B n−1→W becomes an isomorphism of groups.

In terms of this parameterization, the right descent set of W can be expressed more explicitly. For (π,ε)∈B n−1 let

$$ D(\pi, \varepsilon) :=\bigl\{ j \in\{1, \ldots, n-1\} \mid \varepsilon_{j-1} \pi(j-1) > \varepsilon_j \pi(j) \bigr\} $$
(8)

with the convention that ε 0 π(0):=0. Then the following holds.

Proposition 4.1

[5, Proposition 8.1.2]

For all (π,ε)∈B n−1, we have

$$D_\mathrm{r}(\sigma_{\pi,\varepsilon}) = D(\pi,\varepsilon) . $$

Note that the descent set defined in (8) is translated by +1 with respect to definitions found in the literature on signed permutations. This is because to have a consistent setup in Sect. 2, we always start the enumeration of the simple reflections with 1, whereas from a signed permutations perspective it is convenient to start this enumeration with 0.

We define the descent statistic on the hyperoctahedral group by setting the descent number

$$\operatorname{des}(\pi,\varepsilon):=\bigl\lvert D(\pi,\varepsilon) \bigr\rvert $$

for (π,ε)∈B n−1. Similarly, the major index is

$$\operatorname{maj}(\pi,\varepsilon):=\sum_{j \in D(\pi,\varepsilon )} (j-1) $$

and the comajor index is

$$\operatorname{comaj}(\pi,\varepsilon) :=\sum_{j \in D(\pi ,\varepsilon)} (n-j) $$

for (π,ε)∈B n−1. It follows that we have the relationship

$$ \operatorname{comaj}(\pi,\varepsilon )=(n-1)\operatorname{des}(\pi,\varepsilon)-\operatorname{maj}(\pi, \varepsilon) . $$
(9)

4.1 The multivariate generating function

In this situation, Corollary 2.9 specializes as follows.

Proposition 4.2

Fix integers 0≤a 1≤⋯≤a n−1≠0. Let

The generating function of C with respect to Z n is

$$f_C = \frac{1}{1-z_n} \sum_{\pi\in S_{n-1}} \sum_{\varepsilon\in\{ \pm1\}^{n-1}} \frac{\prod_{j \in D(\pi, \varepsilon)} \prod _{i=j}^{n-1} z_{\pi(i)}^{\varepsilon_i} z_n^{a_i}}{\prod_{j=1}^{n-1} (1 - \prod_{i=j}^{n-1} z_{\pi(i)}^{\varepsilon_i} z_n^{a_i} )} . $$

Note that the condition on the a i to be nonnegative and increasing is a normalization rather than a restriction.

Proof

The cone C is symmetric and monoconditional for W. Let

$$F :=\bigl\{ x \in\mathbf{R}^n \bigm| 0 \leq x_1 \leq \cdots\leq x_{n-1} \bigr\} , $$

a fundamental domain for W. The s 1,…,s n−1 defined previously are the simple generators of W corresponding to F. Let x 0:=(a 1,…,a n−1,−1)∈F. By the proof of Lemma 2.5,

where

The matrix A and hence C + is unimodular. Let b 1,…,b n be the primitive generators of C +, enumerated in the unique way such that b j ∉H j for j<n. Then by Corollary 2.9, the generating function of C is

$$f_C = \sum_{\sigma\in W} \frac{\prod_{j \in D_\mathrm{r}(\sigma)} z^{\sigma b_j}}{(1-z^{\sigma b_1}) \cdots(1-z^{\sigma b_n})} . $$

The inverse of A is

where Σ j :=a j +⋯+a n−1. Then b j is the jth column vector of B. Let

$$b_{ij} :=\begin{cases} 0 & \text{if}\ i < j , \\ 1 & \text{if}\ j \leq i < n\ \text{or}\ i = j = n , \\ \varSigma_j & \text{if}\ j < i = n \end{cases} $$

be the ith component of b j , i.e., the (i,j)th component of B. By Proposition 4.1 and using our notation introduced at the beginning of this section,

 □

4.2 Hyperoctahedral Eulerian polynomials

In the remainder of Sect. 4, we provide applications of Proposition 4.2 with connections to permutation statistics and Ehrhart theory. Our first application is well known, going back to [8] and [14]; the polyhedral perspective of the following identity was first established in [14], also using Ehrhart theory.

Corollary 4.3

[8, 14]

The hyperoctahedral Eulerian polynomials are given by

$$\sum_{(\pi, \varepsilon) \in B_{n-1}} t^{\operatorname{des}(\pi ,\varepsilon)} = (1-t)^n \sum_{k=0}^\infty(2k+1)^{n-1} t^k . $$

Proof

Let

$$P = [-1,1]^{n-1} . $$

be the (n−1)-dimensional hypercube. Our strategy to prove Corollary 4.3 is to compute the Ehrhart series

$$\mathrm{Ehr}_P(t) :=\sum_{k \geq0} \bigl\lvert kP \cap\mathbf{Z}^{n-1} \bigr\rvert\cdot t^k $$

of P in two different ways and to conclude by comparing the results.

On the one hand, note that the cone C over P,

$$C = \bigl\{ x \in\mathbf{R}^n \bigm| \forall j < n : \lvert x_j \rvert\leq x_n \bigr\} , $$

is the cone considered in Proposition 4.2 for a 1=⋯=a n−2=0, a n−1=1, so by Proposition 4.2 its generating function is

$$f_C = \frac{1}{1-z_n} \sum_{(\pi, \varepsilon) \in B_{n-1}} \frac {\prod_{j \in D(\pi, \varepsilon)} ( z_n \prod_{i=j}^{n-1} z_{\pi(i)}^{\varepsilon_i} )}{\prod_{j=1}^{n-1} (1 - z_n \prod_{i=j}^{n-1} z_{\pi(i)}^{\varepsilon_i} )} . $$

Since Ehr P (t) is obtained by evaluating f C at z 1=⋯=z n−1=1, z n =t, we obtain

On the other hand, by definition

$$\mathrm{Ehr}_P(t) = \sum_{k \geq0} (2k+1)^{n-1} t^k . $$

Together, we obtain

$$\frac{1}{(1-t)^n} \sum_{(\pi, \varepsilon) \in B_{n-1}} t^{\operatorname{des}(\pi ,\varepsilon)} = \sum_{k \geq0} (2k+1)^{n-1} t^k $$

and Corollary 4.3 follows. □

4.3 The distribution of the comajor index

For k∈N and a variable t, let

$$[k]_t :=1 + t + t^2 + \cdots+ t^{k-1} \quad\text{and} \quad[k]_t! :=[1]_t [2]_t \cdots[k]_t . $$

We show here how to derive the distribution of the comajor index. This is likely well known, as for example it follows from (10) below, but we could not find an explicit statement in the literature. It is worth comparing Corollary 4.4 with the distribution

$$\sum_{(\pi, \varepsilon) \in B_{n-1}} t^{\operatorname{maj}(\pi ,\varepsilon)} = 2^{n-1} [n-1]_t! , $$

which can also be derived from (10).

Corollary 4.4

The distribution of the comajor index on the hyperoctahedral group is given by

$$\sum_{(\pi, \varepsilon) \in B_{n-1}} t^{\operatorname{comaj}(\pi ,\varepsilon)} = (1+t)^{n-1} [n-1]_t! . $$

Proof

Let

$$P = \bigl\{ x \in\mathbf{R}^{n-1} \bigm| \lvert x_1 \rvert+ \cdots+ \lvert x_{n-1} \rvert\leq1 \bigr\} $$

be the (n−1)-dimensional cross-polytope. Our strategy to prove Corollary 4.4 is to compute the Ehrhart series

$$\mathrm{Ehr}_P(t) :=\sum_{k \geq0} \bigl\lvert kP \cap\mathbf{Z}^{n-1} \bigr\rvert\cdot t^k $$

of P in two different ways and to conclude by comparing the results.

On the one hand, note that the cone C over P,

$$C = \bigl\{ x \in\mathbf{R}^n \bigm|\lvert x_1 \rvert+ \cdots+ \lvert x_{n-1} \rvert\leq x_n \bigr\} , $$

is the cone considered in Proposition 4.2 for a 1=⋯=a n−1=1, so by Proposition 4.2 its generating function is

$$f_C = \frac{1}{1-z_n} \sum_{(\pi, \varepsilon) \in B_{n-1}} \frac {\prod_{j \in D(\pi, \varepsilon)} \prod_{i=j}^{n-1} z_{\pi (i)}^{\varepsilon_i} z_n}{\prod_{j=1}^{n-1} (1 - \prod_{i=j}^{n-1} z_{\pi(i)}^{\varepsilon_i} z_n )} . $$

Since Ehr P (t) is obtained by evaluating f C at z 1=⋯=z n−1=1, z n =t, we obtain

On the other hand, it is known [4, Theorem 2.7] that

$$\mathrm{Ehr}_P(t) = \frac{(1+t)^{n-1}}{(1-t)^n} . $$

Together, we obtain

$$\frac{\sum_{(\pi, \varepsilon) \in B_{n-1}} t^{\operatorname {comaj}(\pi,\varepsilon )}}{(1-t) \prod_{j=1}^{n-1} (1 - t^{n-j} )} = \frac {(1+t)^{n-1}}{(1-t)^n} , $$

so

 □

Remark 4.5

The distributions for the descent and comajor index statistics on B n arise from studying simple choices of the a i from the set of 0/1-vectors. It would be interesting to determine the structure of the multivariate generating functions (or their specializations) when other 0/1-vectors are used; due to the hyperoctahedral symmetry of our cones, this amounts to studying the case

$$(a_1,a_2,\ldots,a_{n-1})=(0,\ldots,0,1, \ldots,1) . $$

The resulting cones interpolate naturally between cones over hypercubes and cones over cross-polytopes. While we were not able to treat this family of polytopes using the methods exposed in this article, the following section shows how to do so for a different interpolation.

4.4 Almost constant coefficients

In this section, we show how to use Proposition 4.2 to obtain a closed form expression for the Ehrhart series of a family of rational polytopes interpolating between the hypercube and the cross-polytope (considered in Sects. 4.2 and 4.3, respectively). These are the polytopes P such that the cone over P is of the form considered in Proposition 4.2 such that a 1 through a n−2 coincide. An example of such a polytope is shown in Fig. 2.

Fig. 2
figure 2

The rational polytope P={x∈R 3∣|x 1|+|x 2|+|x 3|+max{|x 1|,|x 2|,|x 3|}≤1}. Its Ehrhart series can be computed by Corollary 4.6

Full size image

Corollary 4.6

Let b,c≥0, not both 0. Let

$$P = \bigl\{ x \in\mathbf{R}^{n-1} \bigm| c \cdot\bigl(\lvert x_1 \rvert+ \cdots+ \lvert x_{n-1} \rvert\bigr) + b\cdot\max\bigl\{ \lvert x_1 \rvert, \ldots, \lvert x_{n-1} \rvert\bigr\} \leq1 \bigr\} . $$

Then the Ehrhart series of P is

$$\mathrm{Ehr}_P(t) = \begin{cases} [c]_t(1+t^c)^{n-1}/(1-t^c)^n & \text{\textit{if}\ $b = 0$}, \\ [b]_t \sum_{k \geq0} ([k+1]_{t^c}+t^c[k]_{t^c} )^{n-1} t^{bk} & \text{\textit{if}\ $b \geq1$}. \end{cases} $$

Proof

In Proposition 4.2, we set a 1=⋯=a n−2=c and a n−1=c+b. Then Ehr P (t)=f C (t):=f C (1,…,1,t), the generating function of C evaluated at z 1,…,z n−1=1, z n =t.

For b>0, the generating function f C (t) becomes

For c≥1, if b=0 then f C (t)=f P (t c)[c] t where P is the (n−1)-dimensional cross-polytope discussed in Sect. 4.3; this can be seen by direct computation using Proposition 4.2, and is also a consequence of the fact that cross-polytopes are reflexive [4, Chap. 4]. Otherwise, to simplify, we make use of a result of Chow and Gessel [9, Eq. (26)] to compute the joint distribution of descent and comajor index over B n , namely

$$ \sum_{k \geq0}\bigl([k+1]_q+[k]_q \bigr)^n x^k = \frac{\sum_{(\pi,\varepsilon) \in B_{n}} x^{\operatorname{des}(\pi,\varepsilon)} q^{\operatorname{maj}(\pi,\varepsilon)}}{ \prod_{j=0}^{n}(1-xq^i)}. $$
(10)

Observe that for (π,ε)∈B n , using (9),

$$q^{\operatorname{comaj}(\pi,\varepsilon)} = \bigl(q^n\bigr )^{\operatorname{des}(\pi,\varepsilon)} ({1}/{q})^{\operatorname{maj}(\pi,\varepsilon)}. $$

Thus, substituting into (10), we get

To get the numerator of f C (t), we set n=n−1, x=t b and q=t c in the last line above and get

 □

4.5 Coefficients in arithmetic progression and lecture hall partitions

If we further generalize the results of the previous subsections to allow a i to be a linear function of i, then f C (t) can be expressed in terms of lecture hall partitions.

Lecture hall partitions, introduced by Bousquet-Mélou and Eriksson [7], are elements of the set

$$L_n = \biggl\{ \lambda\in\mathbf{Z}^n \biggm|0 \leq \frac{\lambda_1}{1} \leq\frac{\lambda_2}{2} \leq\cdots\leq \frac{\lambda_n}{n} \biggr \} . $$

The following relationship between statistics on lecture hall partitions and statistics on signed permutations follows from work of Pensyl and Savage [11]. Define the lecture hall polytope P n,2 by

$$ P_{n,2} = \biggl\{ \lambda\in\mathbb{R}^n \biggm| 0 \leq \frac{\lambda_{1}}{{2}} \leq\frac{\lambda_{2}}{{4}} \leq\cdots \leq\frac{\lambda_{n}}{{2n}} \leq1 \biggr\} . $$

For λ∈L n , (π,ε)∈B n , and ⌈x⌉=inf([x,∞)∩Z), set

Setting k=2, w=1, and z=y in [11, Theorem 6], and using the bijection of [11, Theorem 3], the following holds. For a positive integer n, using the notation

$$ f_{n,2}(t;q,y) = \sum_{\lambda\in tP_{n,2}\ \cap\mathbb{Z}^n} q^{\operatorname{stat}_1(\lambda)} y^{\operatorname{stat}_2(\lambda )} , $$
(11)

we have that

$$ \sum_{t \geq0} f_{n,2}(t;q,y) x^t = \frac{\sum_{(\pi,\varepsilon) \in B_{n}}q^{\operatorname {comaj}(\pi,\varepsilon )}x^{\operatorname{des}(\pi,\varepsilon)}(y^2)^{\operatorname {cobin}(\pi,\varepsilon)}}{ (1-x)\prod_{i=0}^{n-1}(1-xq^{n-i}y^{2((i+1) + \cdots+ n)})} . $$
(12)

Thus,

from which it follows that

$$ \sum_{\lambda\in L_n} x^{ \lceil\frac{\lambda _n}{2n} \rceil} q^{\operatorname{stat}_1(\lambda)} y^{\operatorname{stat}_2(\lambda )} = \frac {\sum_{(\pi,\varepsilon) \in B_{n}}q^{\operatorname{comaj}(\pi ,\varepsilon)}x^{\operatorname{des} (\pi,\varepsilon)}(y^2)^{\operatorname{cobin}(\pi,\varepsilon)}}{ \prod_{i=0}^{n-1}(1-xq^{n-i}y^{2((i+1) + \cdots+ n)})} . $$
(13)

We can apply Proposition 4.2, with z 1=⋯=z n−1=1, z n =t and appropriate choices of a i , to establish a surprising connection between lecture hall partitions and type B symmetrically constrained cones.

Corollary 4.7

Let d≥0, c≥−2d, and b≥0, not all 0. Define a 1,…,a n−1 by

$$a_i = 2di + c \quad(i = 1, \ldots, n-2), \qquad a_{n-1}=2d(n-1)+c+b . $$

Then

$$f_C(t) = \frac{1}{1-t} \sum_{\lambda\in L_{n-1}} t^{\sum_{i=1}^{n-1} a_i \lceil\frac{\lambda_{i}}{2i} \rceil} . $$

Proof

Observe that for 1≤i≤n−2, substituting the values of a i into Proposition 4.2 with z 1=⋯=z n−1=1, z n =t, and then using (13), we obtain

 □

As an example, let n=3, a 1=2 and a 2=4. Then

$$C = \bigl\{ x \in\mathbf{R}^3 \bigm|{} \forall\pi\in S_{2},\ \varepsilon\in\{\pm1 \}^2 : 2 \varepsilon_1 x_{\pi(1)} + 4 \varepsilon_{2} x_{\pi(2)} \leq x_3 \bigr\} , $$

and from Proposition 4.2,

On the other hand, checking the corollary, we have

Remark 4.8

It would be interesting to find a direct correspondence between lecture hall partitions and the points in the cone considered in Corollary 4.7.

5 Cones with symmetry of type D

In this section, we consider the case of monoconditional cones with symmetry given by a Coxeter group of type D. Unsurprisingly, much of the setup in this section is similar to the hyperoctahedral case; the most notable new feature is that we consider lattice point enumeration with respect to a sublattice of the standard integer lattice.

Throughout this section, let W be the finite reflection group of type D n−1 on the first n−1 components of R n. Specifically, let s 1∈O(n) be the reflection at the hyperplane {x∈R n∣x 1+x 2=0}. For j=2,…,n−1, let s j ∈O(n) be the transposition of the (j−1)st and jth component in R n. Then s 1,…,s n−1 are the simple generators of W.

We next describe D r(σ) and the action of W explicitly. Let

$$E_{n-1} :=\bigl\{ \varepsilon\in\{\pm1\}^{n-1} \bigm| \varepsilon_1 \cdots\varepsilon_{n-1} = 1 \bigr\} . $$

For π∈S n−1 and ε∈E n−1, define σ π,ε ∈O(n) by σ π,ε e i =ε i e π(i) for i<n and σ π,ε e n =e n . Then W={σ π,ε ∣π∈S n−1,ε∈E n−1}. We use the convention that π(n):=n for π∈S n−1 and ε n :=1 for ε∈E n−1.

For π∈S n−1 and ε∈E n−1 let

$$D(\pi, \varepsilon) :=\bigl\{ j \in\{ 1, \ldots, n-1 \} \mid \varepsilon_{j-1} \pi(j-1) > \varepsilon_j \pi(j) \bigr\} $$

with the convention that ε 0 π(0):=−ε 2 π(2).

Proposition 5.1

[5, Proposition 8.2.2]

For all σ π,ε ∈W we have

$$D_\mathrm{r}(\sigma_{\pi,\varepsilon}) = D(\pi,\varepsilon) . $$

For a proposition P, we use the symbol

$$[P] :=\begin{cases} 1 & \text{if $P$ is true,} \\ 0 & \text{if $P$ is false.} \end{cases} $$

Proposition 5.2

Fix integers a 1,…,a n−1 such that |a 1|≤a 2≤⋯≤a n−1≠0. Let

Let

$$\varGamma:=\bigl\{ x \in\mathbf{Z}^n \bigm| x_1 \equiv\cdots\equiv x_{n-1} \mod(2) \bigr\} . $$

The generating function of C with respect to Γ is

$$f_C = \frac{1}{1-z_n} \sum_{\pi\in S_{n-1}} \sum_{\varepsilon\in E_{n-1}} \frac{ \prod_{j \in D(\pi,\varepsilon)} ( z_{\pi(1)}^{-\varepsilon_1} z_n^{-a_1} )^{[j=2]} ( \prod_{i=j}^{n-1} z_{\pi(i)}^{\varepsilon_i} z_n^{a_i} )^{1 + [j \geq3]} }{ \prod_{j=1}^{n-1} ( 1 - ( z_{\pi(1)}^{-\varepsilon_1} z_n^{-a_1} )^{[j=2]} ( \prod_{i=j}^{n-1} z_{\pi(i)}^{\varepsilon_i} z_n^{a_i} )^{1 + [j \geq3]} ) } , $$

where z 1,…,z n are the coordinates corresponding to the standard lattice Z n⊂R n.

Note that the conditions on the a i are normalizations rather than restrictions.

Proof

The cone C is symmetric and monoconditional for W. Let

$$F :=\bigl\{ x \in\mathbf{R}^n \bigm|\lvert x_1 \rvert\leq x_2 \leq\cdots\leq x_{n-1} \bigr\} , $$

a fundamental domain for W. Our simple generators s 1,…,s n−1 defined at the beginning of this section are the simple generators of W corresponding to the facets of F. Let x 0:=(a 1,…,a n ,−1)∈F. By the proof of Lemma 2.5,

where

The inverse of A is

where Σ j :=a j +⋯+a n−1 and \(\varSigma_{2}' := \varSigma_{2} - a_{1}\). Hence the Γ-primitive generators of C + are the column vectors b 1,…,b n of the matrix

As det(B)=2n−2=|Z n/Γ| it follows that C + is unimodular. Note that b 1,…,b n are enumerated in the unique way such that b j ∉H j for j<n, where H j is the reflection hyperplane for the reflection s j . Hence by Corollary 2.9 the generating function of C is

$$f_C = \sum_{\sigma\in W} \frac{\prod_{j \in D_\mathrm{r}(\sigma)} z^{\sigma b_j}}{(1-z^{\sigma b_1}) \cdots(1-z^{\sigma b_n})} . $$

Let

$$b_{ij} :=\begin{cases} -1 & \text{if}\ i = 1,\ j = 2, \\ 1 & \text{if}\ j \leq i < n\ \text{or}\ i = j = n, \\ 0 & \text{if}\ i < j \geq2, \\ \varSigma_1 & \text{if}\ i = n,\ j = 1, \\ \varSigma_2' & \text{if}\ i = n,\ j = 2, \\ 2\varSigma_j & \text{if}\ i = n,\ 2 < j < n \end{cases} $$

be the ith component of b j , i.e., the (i,j)th component of B. Thus

 □