Counting matrices over finite fields with support on skew Young diagrams and complements of Rothe diagrams

Abstract

We consider the problem of finding the number of matrices over a finite field with a certain rank and with support that avoids a subset of the entries. These matrices are a q-analogue of permutations with restricted positions (i.e., rook placements). For general sets of entries, these numbers of matrices are not polynomials in q (Stembridge in Ann. Comb. 2(4):365, 1998); however, when the set of entries is a Young diagram, the numbers, up to a power of q−1, are polynomials with nonnegative coefficients (Haglund in Adv. Appl. Math. 20(4):450, 1998).

In this paper, we give a number of conditions under which these numbers are polynomials in q, or even polynomials with nonnegative integer coefficients. We extend Haglund’s result to complements of skew Young diagrams, and we apply this result to the case where the set of entries is the Rothe diagram of a permutation. In particular, we give a necessary and sufficient condition on the permutation for its Rothe diagram to be the complement of a skew Young diagram up to rearrangement of rows and columns. We end by giving conjectures connecting invertible matrices whose support avoids a Rothe diagram and Poincaré polynomials of the strong Bruhat order.

1 Introduction

We study certain q-analogues of permutations with restricted positions or, equivalently, of placements of nonattacking rooks. The q-analogue of permutations we work with is invertible n×n matrices over the finite field F _q with q elements, as in [17, Chap. 1]. Then the analogue of permutations with restricted positions is invertible matrices over F _q with some entries required to be zero.

Specifically, given a subset S of {1,2,…,n}×{1,2,…,n}, let $\operatorname {mat}_{q}(n,S,r)$ be the number of n×n matrices over F _q with rank r, none of whose nonzero entries lie in S. This is clearly an analogue (in the plain English meaning) of the problem of counting permutations whose permutation matrix has no 1 in the position of any entry of S, but actually much more can be said. In [11, Proposition 5.1] it was shown that $\operatorname {mat}_{q}(n,S,r)/(q-1)^{r}$ is in fact an enumerative q-analogue of permutations with restricted positions; that is, its value, modulo (q−1), counts the placements of r nonattacking rooks on the complement of S.

The function $\operatorname {mat}_{q}(n,S,r)$ can exhibit a variety of different behaviors, as seen in the following three examples.

Examples 1.1

• 1. When S=∅, $\operatorname {mat}_{q}(n,\varnothing,n)$ is the number of n×n invertible matrices over F _q , which is

  $$\bigl(q^{n}-1\bigr) \bigl(q^{n}-q\bigr)\cdots \bigl(q^n-q^{n-1}\bigr)=q^{\binom{n}{2}}(q-1)^n \prod_{i=1}^{n} \bigl(1+q+ \cdots+q^{i-1}\bigr). $$

  The term $\prod_{i=1}^{n}(1+q+\cdots+q^{i-1})$ in the product is a polynomial with positive coefficients and in fact is the generating series for permutations in $\mathfrak{S}_{n}$ by number of inversions.

• 2. When n=3 and S is the diagonal {(1,1),(2,2),(3,3)}, we have

  $$\operatorname {mat}_q\bigl(3,\bigl\{(1,1),(2,2),(3,3)\bigr\},3\bigr)=(q-1)^3 \bigl(q^3+2q^2-q\bigr). $$

  The number of invertible n×n matrices for general n over F _q with zero diagonal was computed in [11, Proposition 2.2]; as in this example, it is of the form (q−1)ⁿ f(q) for a polynomial f with both positive and negative coefficients.

• 3. When n=7, Stembridge [18] found a set F with 28 elements (shown in Fig. 1) such that $\operatorname {mat}_{q}(7,F,7)$ is given by a quasi-polynomial in q, that is, by two distinct polynomials depending on whether q is even or odd. The set F is the complement of the incidence matrix of the Fano plane.

  Fig. 1

  

  A representative matrix counted in $\operatorname {mat}_{q}(7,F,7)$ where F is the complement of the point-line incidence matrix of the Fano plane, shown at right. Stembridge [18] showed this to be the smallest example of the form $\operatorname {mat}_{q}(n, S, n)$ that is not a polynomial in q

From the examples above we see that $\operatorname {mat}_{q}(n,S,r)$ is not necessarily a polynomial in q, and if it is a polynomial in q, it might or might not be of the form (q−1)^r f(q) where f(q) is a polynomial with nonnegative integer coefficients. Then a natural question to ask is the following:

Question 1.2

What families of sets S are there such that $\operatorname {mat}_{q}(n,S,r)/(q-1)^{r}$ is (i) not a polynomial in q, (ii) a polynomial in q, or (iii) a polynomial in q with nonnegative integer coefficients?

In the remainder of this introduction, we give a summary of our progress toward answering this question.

1.1 Outline and summary of results

In Sect. 2, we give the definitions and notation that will be used throughout the paper including the definition and some properties of q-rook numbers.

In Sect. 3, we address general conditions on r and S under which the function $\operatorname {mat}_{q}(n,S,r)$ is always a polynomial in q. We show that if r=1, then $\operatorname {mat}_{q}(n, S, 1)$ is a polynomial in q for any set S, though not necessarily with nonnegative coefficients. (It is an open question whether there is a set S such that $\operatorname {mat}_{q}(n, S, 2)$ is nonpolynomial in q.) Our main result of this section is to extend work of Stembridge [18] to give reductions to compute $\operatorname {mat}_{q}(n, S, r)$ in terms of smaller instances of similar problems when S has a row or column with very few or very many entries.

In the rest of the paper, we discuss special families of sets S such that $\operatorname {mat}_{q}(n,S,r)/(q-1)^{r}$ is a polynomial in q with nonnegative integer coefficients. Haglund [7] showed that if the set S is a straight shape, then $\operatorname {mat}_{q}(n,\overline{S},r)/(q-1)^{r}$ is a polynomial with nonnegative integer coefficients. Our second main result, proved in Sect. 4, is to extend this to complements of skew shapes.

Corollary 4.6

For any skew shape S _λ/μ ,

$$\operatorname {mat}_q(n,\overline{S_{\lambda/\mu}},r) = (q-1)^r f(q), $$

where f(q) is a polynomial with nonnegative integer coefficients.

In fact, we show that this is true for an even larger class of shapes than skew shapes, namely those that have what we call the North–East Property. Also, because $\operatorname {mat}_{q}(n,S,r)$ is invariant under permuting rows and columns, we have that $\operatorname {mat}_{q}(n,\overline{S},r)/(q-1)^{r}$ is a polynomial with nonnegative integer coefficients for any set S that is a straight or skew shape after permuting rows and columns.

Another natural family of diagrams is the collection of Rothe diagrams of permutations, which appear in the study of Schubert calculus. The Rothe diagram R _w of a permutation w is a subset of {1,2,…,n}×{1,2,…,n} whose cardinality is equal to the number of inversions of w; it is given by

$$R_w = \bigl\{(i,j) \mid 1\leq i, j\leq n,\,\, w(i)>j,\,\, w^{-1}(j)>i\bigr\}. $$

See Fig. 5 for some examples of Rothe diagrams. Lascoux and Schützenberger [10] showed that the Rothe diagram R _w of a permutation w is a straight shape up to permutation of rows and columns if and only if w, written as a word w ₁ w ₂⋯w _n , avoids the permutation pattern 2143 (i.e., there is no sequence i<j<k<l such that w _j <w _i <w _l <w _k ). Our third main result gives an analogous criterion for the case of complements of skew shapes.

Theorem 5.4

The Rothe diagram R _w of a permutation w is, up to permuting its rows and columns, the complement of a skew shape if and only if w can be decomposed as w=a ₁ a ₂⋯a _k b ₁ b ₂⋯b _n−k where a _i <b _j for all i and j, and both a ₁ a ₂⋯a _k and b ₁ b ₂⋯b _n−k are 2143-avoiding.

We also show that this condition is equivalent to the statement that w avoids the nine patterns 24153, 25143, 31524, 31542, 32514, 32541, 42153, 52143, and 214365, and we express the generating series for these permutations in terms of the generating series for 2143-avoiding permutations.

By Corollary 4.6, if w satisfies the condition above, then $\operatorname {mat}_{q}(n,R_{w},r)/(q-1)^{r}$ is a polynomial with nonnegative integer coefficients. Surprisingly, computer calculations for n≤7 and 0≤r≤n [9] suggest that $\operatorname {mat}_{q}(n,R_{w},r)/(q-1)^{r}$ is a polynomial with nonnegative integer coefficients for all permutations w (see Conjecture 5.1). Moreover, computer calculations also suggest that when w avoids the permutation patterns 1324,24153,31524, and 426153, we have that $\operatorname {mat}_{q}(n,R_{w},n)/(q-1)^{n}$ is (up to a power of q) the Poincaré polynomial $P_{w}(q) = \sum_{u\geq w} q^{\operatorname {inv}(u)}$, where the sum is over all permutations u of n above w in the strong Bruhat order (see Conjecture 6.6). Interestingly, these four patterns have appeared in related contexts [5, 8, 14, 15].

Supplementary code for calculating $\operatorname {mat}_{q}(n,S,r)$ and other related objects and data generated by this code to test the conjectures in Sect. 6 are available at the following website: http://sites.google.com/site/matrixfinitefields/.

2 Definitions

We denote [n]={1,2,…,n}. The support of a matrix A is the set of indices (i,j) of the nonzero entries a _ij ≠0. Fix integers n and r such that n≥1 and n≥r≥0, and let S be a subset of [n]×[n]. We define $\operatorname {mat}_{q}(n,S,r)$ to be the number of n×n matrices over F _q with rank r and support contained in $\overline{S}$, the complement of S. That is, $\operatorname {mat}_{q}(n, S, r)$ counts matrices A of rank r such that if (i,j)∈S, then a _ij =0. We consider the problem of computing $\operatorname {mat}_{q}(n,S,r)$.

We now define several special types of diagrams that will be important to us in what follows. Examples of these diagrams are given in Fig. 2. We say that S⊆[n]×[n] is a straight shape if its elements form a Young diagram of a partition. (Throughout this paper we use English notation and matrix coordinates for partitions.) Thus, to every integer partition λ with at most n parts and with largest part at most n (i.e., to each sequence of integers (λ ₁,λ ₂,…,λ _n ) such that n≥λ ₁≥λ ₂≥⋯≥λ _n ≥0), there is an associated set S=S _λ ={(i,j)∣1≤j≤λ _i }. We denote by |λ| the size λ ₁+λ ₂+⋯ of the shape λ. This is also the number of entries in S _λ . Similarly, if λ and μ are partitions such that S _μ ⊆S _λ , then we say that the set S _λ ∖S _μ is a skew shape, and we denote it by S _λ/μ . Lastly, we say that a set S⊆[n]×[n] has the North–East (NE) Property if for all i,i′,j,j′∈[n] such that i′<i and j<j′, we have that if (i,j),(i′,j), and (i,j′) are in S, then so is (i′,j′). Note that for any partitions λ and μ, $S_{\lambda}, \overline{S_{\lambda}}$, and S _λ/μ have the NE Property. But $\overline{S_{\lambda/\mu}}$ in general does not have this property.

Fig. 2

Representative matrices from $\operatorname {mat}_{q}(5,S,r)$ when S is (i) a straight shape, (ii) a skew shape, (iii) a set with the NE Property; and their respective complements (iv), (v), (vi)

We denote by $\mathfrak{S}_{n}$ the group of permutations on [n]. We write permutations as words w=w ₁ w ₂⋯w _n where w _i is the image of w at i. Let $\operatorname {inv}(w)$ denote the number of inversions #{(i,j)∣i<j,w _i >w _j } of w. We also identify each permutation w with its permutation matrix, the n×n 0–1 matrix with 1s in positions (i,w _i ).

We think of the 1s in a permutation matrix as n nonattacking rooks on [n]×[n]. In this case, the number of inversions of the permutation is exactly the number of elements in [n]×[n] that are not directly below/south (in the same column) or to the right/east (in the same row) of any placed rook. We generalize this as follows. Given a subset B of [n]×[n] (sometimes called a board) and a rook placement C of r nonattacking rooks on B, the SE-inversion number $\operatorname {inv}_{\textsf{SE}}(C,B)$ is the number of elements in B not directly south (in the same column) or to the east (in the same row) of any placed rook. Then the rth (SE) q-rook number of Garsia and Remmel [4] is

$$R^\textsf{(SE)}_r(B,q) = \sum_{C} q^{\operatorname {inv}_{\textsf{SE}}(C,B)}, $$

where the sum is over all rook placements C of r nonattacking rooks on B. We define the north–east inversion number $\operatorname {inv}_{\textsf{NE}}(C,B)$ and rook polynomial $R^{\textsf{(NE)}}_{r}(B,q)$ analogously.

Proposition 2.1

[4] Given an integer partition λ=(λ ₁,λ ₂,…,λ _n ) such that n≥λ ₁≥λ ₂≥⋯≥λ _n ≥0, set S _λ ={(i,j)∣1≤i≤n,1≤i≤λ _j }. The Garsia–Remmel q-rook number $R_{n}^{\textsf{(SE)}}(S_{\lambda},q)$ is

$$ R_n^\textsf{(SE)}(S_{\lambda},q) = \prod _{i=1}^n [\lambda_{n-i+1}-i+1]_q, $$

(2.2)

where [m] _q =1+q+q ²+⋯+q ^m−1.

Remark 2.3

We will see as a corollary of Theorems 4.1 and 4.2 that for a straight shape λ, $R_{r}^{\textsf{(SE)}}(S_{\lambda},q)= R_{r}^{\textsf{(NE)}}(S_{\lambda},q)$. However, this is not true for all skew shapes. For example, if λ/μ=4432/31, we have $R_{3}^{\textsf{(SE)}}(S_{4432/31},q)=1+6q^{2}+5q^{3}+3q^{4}+2q^{5}+q^{6}$ and $R_{3}^{\textsf{(NE)}}(S_{4432/31},q)=2q+8q^{2} + 7q^{3}+q^{4}$. But for skew shapes in the case of n rooks we do have an analogous relation, as the following result shows.

Proposition 2.4

For a skew shape S _λ/μ ⊆[n]×[n] we have

$$R_n^\textsf{(SE)}(S_{\lambda/\mu},q) =q^{\binom{n}{2}-|\mu|}\cdot R_n^{\sf(NE)}\bigl(S_{\lambda/\mu},q^{-1}\bigr). $$

Proof

For each rook placement of n rooks on S _λ/μ , the number of SE-inversions is equal to the number of inversions of the associated permutation w minus the size of μ. On the other hand, the number of NE-inversions of this rook placement on S _λ/μ is $\binom{n}{2}$ minus the number of inversions of w. The result follows. □

3 General polynomiality results

In this section, we give some general conditions under which $\operatorname {mat}_{q}(n, S, r)$ is a polynomial. In Sect. 3.1, we show that for any n and S, the function $\operatorname {mat}_{q}(n, S, 1)$ is polynomial in q. In Sect. 3.2, we give reduction formulas for computing $\operatorname {mat}_{q}(n, S, r)$ in terms of smaller instances when S has a row or column with either very few or very many entries.

Throughout this section we work with rectangular matrices of any dimensions rather than just square matrices. Thus, in this section, for integers m, n and r and a subset S of [m]×[n], we denote by $\operatorname {mat}_{q}(m \times n, S, r)$ the number of m×n matrices of rank r over F _q whose support avoids S.

3.1 Polynomial formula for the rank-one case mat _q (n,S,1)

In Fig. 1 we showed an example by Stembridge [18] of a set S⊆[7]×[7] such that $\operatorname {mat}_{q}(7,S,7)$ is not a polynomial in q. In this paper, we mainly focus on studying certain families of sets S where $\operatorname {mat}_{q}(n,S,r)$ is a polynomial in q. But before looking at particular sets S, we consider the rank r=1 case for an arbitrary set S.

Proposition 3.1

For any m and n and any set S⊆[m]×[n], $\operatorname {mat}_{q}(m \times n,S,1)$ is a polynomial in q.

Proof

Fix m, n, and S. We count matrices with a given collection of nonzero rows. Given a nonempty subset T⊆[m] of rows, let a _S (T) be the number of columns with no entries which are both in one of the rows of T and in S. Then there are exactly $(q^{a_{S}(T)} - 1)(q - 1)^{\# T - 1}$ matrices of rank 1 over F _q whose support avoids S and whose nonzero rows are exactly those in T. It follows immediately that

$$ \operatorname {mat}_q(m \times n, S, 1) = \sum _{\substack{T \subseteq [m] \\ \text{nonempty}}} \bigl(q^{a_S(T)} - 1\bigr) (q - 1)^{\# T - 1} $$

(3.2)

is a polynomial in q. □

Example 3.3

Take the 4×4 shape S={(i,i)∣1≤i≤4}. Then

(In fact, one can show that if S is the diagonal {(i,i)∣1≤i≤n}, then $\operatorname {mat}_{q}(n\times n,S,1) = \frac{1}{q-1}((2q-1)^{n} - 2q^{n}+1)$.)

Remark 3.4

In later sections of this paper, we show that for certain diagrams S (straight shapes, skew shapes, and conjecturally Rothe diagrams of permutations), the function $\operatorname {mat}_{q}(n, S, r)/(q - 1)^{r}$ is not only a polynomial in q but also has nonnegative coefficients. However, this is not the case for matrices of rank 1: although each summand is a power of q−1 times a polynomial with positive coefficients, the powers of q−1 differ. So, as in Example 3.3, negative coefficients can turn up for certain choices of S. Interestingly, if we substitute t=q−1 into (3.2), we obtain a polynomial in t with nonnegative coefficients. Is this true more generally?

3.2 Reduction formulas when S has rows with few or many entries

In [18, Theorem 8.2], Stembridge gave some structural restrictions on a minimal set S such that $\operatorname {mat}_{q}(n, S, n)$ is nonpolynomial. In particular, he showed that in such a minimal example, every row contains at least three entries of $\overline{S}$ and at least two entries of S. In this section, we push his results slightly further: we show that for any rank r, if either S or $\overline{S}$ has a row with at most two entries, then we can express $\operatorname {mat}_{q}(m \times n, S, r)$ as a linear combination with polynomial coefficients of similar expressions for matrices of strictly smaller size. (Of course, the same arguments apply to columns as well as to rows.) Furthermore, we explain why this approach cannot be used in the case where S contains three entries in some row. This does not settle the question of whether there are some m and n and a set S⊆[m]×[n] with only three entries per row such that $\operatorname {mat}_{q}(m \times n, S, r)$ is nonpolynomial in q, though we conjecture that such examples exist; in Stembridge’s example [18] (see Fig. 1), the set S has four entries per row, and $\overline{S}$ has three.

For convenience, throughout this section we write expressions like $\operatorname {mat}_{q}(a \times b, S, r)$ without worrying whether S⊆[a]×[b], where properly we should write $\operatorname {mat}_{q}( a \times b, ([a] \times [b])\cap S, r)$.

3.2.1 Reduction when S has at most two entries in some row

We wish to show that if S has two or fewer entries in some row, then $\operatorname {mat}_{q}(m \times n, S, r)$ can be reduced to a sum of polynomial multiples of similar but simpler expressions. We begin with a useful proposition that we use in the proof of Theorem 3.6 to do case analysis.

Proposition 3.5

Suppose that $M = [{\bf v}_{1} \ {\bf v}_{2} \ \cdots \ {\bf v}_{n}]$ is an m×n matrix over F _q of rank r and that the submatrix $[{\bf v}_{n - k + 1} \ \cdots \ {\bf v}_{n}]$ has rank r′. The number of tuples ${\bf w} = (w_{1}, w_{2}, \ldots, w_{n - k}) \in \mathbf {F}_{q}^{n - k}$ such that

is q ^r−r′. For the other q ^n−k−q ^r−r′ tuples, this matrix has rank r+1.

Proof

This is just the rank-nullity theorem: vectors w that do not increase the rank are those that (when augmented by k 0s) lie in the row space of M. The portion of the row space with last k coordinates equal to 0 is exactly the kernel of the projection onto these last k coordinates. The image of this projection has dimension r′, so the kernel has dimension r−r′, as desired. □

Now we use this to prove the main result of this section.

Theorem 3.6

Suppose that S⊆[m]×[n] contains at most two entries in the mth row. Then $\operatorname {mat}_{q}(m \times n, S, r)$ is equal to a linear combination of similar expressions for smaller matrices with coefficients in Z[q].

Corollary 3.7

If m, n and S are chosen minimal (in the sense of row- or column-removal) so that $\operatorname {mat}_{q}(m \times n, S, r)$ is not polynomial in q, then S contains at least three entries in each row and column.

The rest of this subsection is devoted to the proof of Theorem 3.6.

Proof

Consider the mth row of the matrix-to-be: it contains zero, one, or two forced 0s (i.e., elements of S) and either lies in a dimension-r space spanned by the first m−1 rows or lies outside their (r−1)-dimensional span. We separately compute the number of matrices contributed in each of these three cases.

Case 1::

The set S contains no elements in the mth row. We have two possibilities: first, it might be that the first m−1 rows of the matrix span a space of dimension r and the last row lies in this space. Then we would have $\operatorname {mat}_{q}((m - 1) \times n, S, r)$ choices for the first m−1 rows and q ^r choices for the last row, for a total contribution of $\operatorname {mat}_{q}((m - 1) \times n, S, r) \cdot q^{r}$. Second, it might be that the first m−1 rows span a space of dimension r−1 and the last row lies outside this space. Then we would have $\operatorname {mat}_{q}((m - 1) \times n, S, r - 1)$ choices for the first m−1 rows and q ⁿ−q ^r−1 choices for the last row, for a total contribution of $\operatorname {mat}_{q}((m - 1) \times n, S, r - 1) \cdot (q^{n} - q^{r-1})$. Thus, the total contribution in this case is

$$\operatorname {mat}_q\bigl((m - 1) \times n, S, r\bigr) \cdot q^r + \operatorname {mat}_q\bigl((m - 1) \times n, S, r - 1\bigr) \cdot \bigl(q^n - q^{r-1}\bigr). $$

Observe that the two instances of the function $\operatorname {mat}_{q}$ that appear in this expression involve matrices strictly smaller than m×n.

Case 2::

The set S contains one element in the mth row, without loss of generality the entry (m,n). We have two cases depending on the dimension of the space spanned by the nth column, i.e., whether this column is zero or not.

• (a) We count matrices in which the nth column is the zero vector. In this case, we may choose the first m−1 rows to span a space of dimension r in $\operatorname {mat}_{q}((m - 1)\times(n - 1), S, r)$ ways and choose the last row in q ^r ways, or we may choose the first m−1 rows to span a space of dimension r−1 in $\operatorname {mat}_{q}((m - 1)\times(n - 1), S, r - 1)$ ways and choose the last row in q ⁿ⁻¹−q ^r−1 ways.

• (b) We count matrices in which the nth column is not the zero vector. In this case, we may choose the first m−1 rows to span a space of dimension r in $\operatorname {mat}_{q}((m- 1)\times n, S, r) - \operatorname {mat}_{q}((m - 1)\times(n - 1), S, r)$ ways and (by Proposition 3.5) choose the last row in q ^r−1 ways, or we may choose the first m−1 rows to span a space of dimension r−1 in $\operatorname {mat}_{q}((m - 1) \times n, S, r - 1) - \operatorname {mat}_{q}((m - 1)\times(n - 1), S, r - 1)$ ways and choose the last row in q ⁿ⁻¹−q ^r−2 ways.

The total contribution from these subcases is

Case 3::

The set S contains two elements in the mth row, without loss of generality the elements (m,n−1) and (m,n). We have three cases depending on the dimension of the space spanned by the (n−1)th and nth columns.

• (a) We count matrices in which the last two columns span a space of dimension 0, i.e., both columns are all zero. In this case, we may choose the first m−1 rows of the matrix to span a space of dimension r in $\operatorname {mat}_{q}((m - 1) \times (n - 2), S, r)$ ways and choose the last row to lie in this space in q ^r ways, or we may choose the first m−1 rows to span a space of dimension r−1 in $\operatorname {mat}_{q}((m - 1) \times (n - 2), S, r - 1)$ ways and choose the last row in q ⁿ⁻²−q ^r−1 ways.

• (b) We count matrices in which the last two columns span a space of dimension 1. There are three possible ways this can come about: the last column may be all zero and the next-to-last column nonzero, the next-to-last column may be all zero and the last column nonzero, or both columns may be nonzero but parallel. Let S ₁ be the set that results if we remove the last column from S, let S ₁ be the set that results if we remove the next-to-last column from S and replace each element (i,n) in S with (i,n−1), and let S ₂ be the set that results if we remove the last two columns from S and add a new entry (i,n−1) whenever either (i,n−1) or (i,n) appeared in S. With this notation, the matrices we desire to count fall into the following six classes:

  • (i) We may choose the first m−1 rows so that they span a space of dimension r, the last column is zero and the next-to-last column is nonzero in $\operatorname {mat}_{q}((m - 1) \times (n - 1), S, r) - \operatorname {mat}_{q}((m - 1) \times (n - 2), S, r)$ ways, and by Proposition 3.5 we may extend each of these to an m×n matrix of rank r in q ^r−1 ways.

  • (ii) We may choose the first m−1 rows so that they span a space of dimension r−1, the last column is zero and the next-to-last column is nonzero in $\operatorname {mat}_{q}((m - 1) \times (n - 1), S, r - 1) - \operatorname {mat}_{q}((m - 1) \times (n - 2), S, r - 1)$ ways, and by Proposition 3.5 we may extend each of these to a matrix of rank r in q ⁿ⁻²−q ^r−2 ways.

  • (iii) We may choose the first m−1 rows so that they span a space of dimension r, the next-to-last column is zero and the last column is nonzero in $\operatorname {mat}_{q}((m - 1) \times (n - 1), S_{1}, r) - \operatorname {mat}_{q}((m - 1) \times (n - 2), S_{1}, r)$ ways, and by Proposition 3.5 we may extend each of these to a matrix of rank r in q ^r−1 ways.

  • (iv) We may choose the first m−1 rows so that they span a space of dimension r−1, the next-to-last column is zero and the last column is nonzero in $\operatorname {mat}_{q}((m - 1) \times (n - 1), S_{1}, r - 1) - \operatorname {mat}_{q}((m - 1) \times (n - 2), S_{1}, r - 1)$ ways, and by Proposition 3.5 we may extend each of these to a matrix of rank r in q ⁿ⁻²−q ^r−2 ways.

  • (v) We may choose the first m−1 rows so that they span a space of dimension r and the last two columns are nonzero and parallel in $(q - 1)(\operatorname {mat}_{q}((m - 1) \times (n - 1), S_{2}, r) - \operatorname {mat}_{q}((m - 1) \times (n - 2), S_{2}, r))$ ways, and by Proposition 3.5 we may extend each of these to a matrix of rank r in q ^r−1 ways.

  • (vi) We may choose the first m−1 rows so that they span a space of dimension r and the last two columns are nonzero and parallel in $(q - 1)(\operatorname {mat}_{q}((m - 1) \times (n - 1), S_{2}, r - 1) - \operatorname {mat}_{q}((m - 1) \times (n - 2), S_{2}, r - 1))$ ways, and by Proposition 3.5 we may extend each of these to a matrix of rank r in q ⁿ⁻²−q ^r−2 ways.

• (c) We count matrices in which the last two columns span a space of dimension 2. To compute the number of these in which the first m−1 rows span a space of dimension r, subtract from $\operatorname {mat}_{q}((m - 1) \times n, S, r)$ the number of matrices in which the last two columns span a space of dimension less than 2; this number is computed in cases (a), (b)i, (b)iii, and (b)v above. By Proposition 3.5, we may extend each of these to a matrix of rank r in q ^r−2 ways. Alternatively, the first m−1 rows may span a space of dimension r−1, and the number of ways in which this happens is the result of subtracting the appropriate values computed in cases (a), (b)ii, (b)iv, and (b)vi above. By Proposition 3.5, we may extend each of these to a matrix of rank r in q ⁿ⁻²−q ^r−3 ways.

As before, every instance of the function $\operatorname {mat}_{q}$ in each subcase is applied on matrices of size strictly smaller than m×n. (We omit the large, uninformative expression that is the total contribution from the cases above.)

Finally, it is easy to check that these cases are exhaustive and that each yields an application of $\operatorname {mat}_{q}$ on matrices of size smaller than m×n, and that they are combined with coefficients that are polynomials in q, as desired. □

3.2.2 Reductions of this sort cannot work if S has three entries per row

On first glance, it appears that the method of proof of Theorem 3.6 can be extended to show that the function $\operatorname {mat}_{q}(m \times n, S, r)$ is a polynomial in q for any choice of m, n, S, and r. However, as the example of Stembridge [18] (see Fig. 1) shows, this is not the case. In this section, we briefly explain why this recursive approach breaks down for matrices with three or more zeroes per row.

Suppose that we attempt to recurse as in the proof of Theorem 3.6 while removing a row with three forced zero entries, e.g., for the set S such that

is a representative matrix counted in $\operatorname {mat}_{q}(6,S,r)$. For instance, let us remove the bottom row. Then we wish to count the number of matrices of the form

of each rank, refined by the dimension of the space spanned by the first three columns. As in Case 3(b) of the proof of Theorem 3.6, we further refine by the geometric relationship between these columns; it is straightforward to check that if these columns span a space of dimension 0 or 1, or if two of them are linearly dependent, then the induction goes through. The only remaining case is that they span a space of dimension 2 and are pairwise linearly independent.¹ In this case, we may take an appropriate linear combination of the second and third columns to eliminate the first column, in which case our problem appears at first glance to reduce to the problem of counting matrices of the form

in which the first two columns are linearly independent; we have already shown (in Case 3(c) of the proof of Theorem 3.6) that under an appropriate inductive hypothesis this yields a polynomial answer in q. The “catch” is that the apparently valid reduction is actually wrong: in our particular example, the zeroes in positions (1,1) and (2,1) of M′, together with the elimination step, impose the condition that the upper-left 2×2 minor of M″ is 0. This condition cannot be reduced to the requirement that certain entries be equal to 0, so any inductive approach of this nature would have to take a substantially stronger inductive hypothesis that allows us to consider restrictions on larger minors. While this more general question seems potentially interesting, it is quite broad, and it seems likely that polynomiality will only hold under very restrictive conditions on the set of selected minors.

3.2.3 Reduction when $\overline{S}$ has at most two entries in a row

In this section, we show the complementary result of Theorem 3.6 and give a reduction for $\operatorname {mat}_{q}(m \times n, S, r)$ when $\overline{S}$ contains at most two entries in some row.

Theorem 3.8

Suppose that S⊆[m]×[n] and that $\overline{S}$ contains at most two entries in the mth row. Then $\operatorname {mat}_{q}(m \times n, S, r)$ is equal to a linear combination of similar expressions for smaller matrices with coefficients in Z[q].

Proof

As before, we split into cases depending on the number of entries of $\overline{S}$ in the mth row.

Case 1.:

The set $\overline{S}$ contains no entries in the mth row. In this case the mth row is all zero, and we can simply remove it, so $\operatorname {mat}_{q}(m \times n, S, r) = \operatorname {mat}_{q}((m - 1) \times n, S, r)$.

Case 2.:

The set $\overline{S}$ contains one entry in the mth row, without loss of generality the entry (m,n). In this case, the matrices of rank r with zeroes at the positions marked by S fall into two categories: those for which the (m,n) entry is also 0, of which there are $\operatorname {mat}_{q}((m - 1) \times n, S, r)$, and those for which the (m,n) entry is nonzero. In the latter case we may row-reduce, using the mth row to eliminate the nth column, and we find that there are $(q - 1)q^{a} \operatorname {mat}_{q}((m - 1) \times (n- 1), S, r - 1)$ such matrices, where a is the number of entries of $\overline{S}$ among {(1,n),…,(m−1,n)}. Thus, in total we have

$$\operatorname {mat}_q\bigl((m - 1) \times n, S, r\bigr) + (q - 1)q^a \operatorname {mat}_q\bigl((m - 1) \times (n- 1), S, r - 1\bigr) $$

matrices in this case.

Case 3.:

Suppose that the last row of $\overline{S}$ contains two entries in the mth row, without loss of generality the entries (m,n−1) and (m,n). We count matrices M of rank r whose entries at positions marked by S are equal to 0, refining by whether the entries (m,n−1) and (m,n) are also equal to 0.

• (a) If both entries are zero, the number of matrices is just $\operatorname {mat}_{q}((m - 1) \times n, S, r)$.

• (b) If the (m,n) entry is nonzero and the (m,n−1) entry is zero, then we are essentially in Case 2 above: the entry (m,n) may be chosen in (q−1) ways, and we may use the mth row to eliminate the nth column, which gives a factor of q ^b, where b is the number of entries of $\overline{S}$ among {(1,n),…,(m−1,n)}. The rest of the matrix may be filled in $\operatorname {mat}_{q}((m - 1) \times (n - 1), S, r - 1)$ ways. So in total we have a contribution of $(q - 1)q^{b} \operatorname {mat}_{q}((m - 1) \times (n - 1), S, r - 1)$ in this case.

• (c) Similarly, if the (m,n−1) entry is nonzero and the (m,n) entry is zero, we may apply the same technique to eliminate the (n−1)th column, etc. If we set c to be the number of entries of $\overline{S}$ in {(1,n−1),…,(m−1,n−1)} and let S′ be the result of removing the (n−1)th column from S and shifting every entry in the nth column left, then we have in this case a contribution of $(q - 1)q^{c} \operatorname {mat}_{q}((m - 1) \times (n - 1), S', r - 1)$ matrices.

• (d) Finally, if both the (m,n−1) and (m,n) entries are nonzero (which may happen in (q−1)² ways), we use (m,n) entry to kill nonzero entries (i.e., entries of $\overline{S}$) in the nth column. In this case, nonzero entries get “transferred” from the nth column to the (n−1)th; we pick up an extra factor of q every time there is already an entry of $\overline{S}$ in same row in the (n−1)th column. Let S″ be the set that we get by removing the (n−1)th and nth columns from S and adding a new column that has an entry in row i whenever {(i,n−1),(i,n)}⊆S, and let d be the number of i∈[m−1] such that neither (i,n−1) nor (i,n) is in S. In this case, we have a total contribution of $(q - 1)^{2} q^{d} \operatorname {mat}_{q}((m - 1) \times(n - 1), S'', r - 1)$ matrices.

The total number of matrices in this case is the sum of the expressions in the four subcases.

Each of the expressions appearing above yields a linear combination of applications of $\operatorname {mat}_{q}$ on matrices of size smaller than m×n, and the coefficients are polynomials in q, as desired. □

This result cannot be extended to the case of three nonzero entries in some row, again by the example of [18] (see Fig. 1); attempts to follow the same method of proof as in Theorem 3.8 meet an obstruction similar to that of Sect. 3.2.2.

Corollary 3.9

If m, n, and S are chosen minimal (in the sense of row- or column-removal) so that $\operatorname {mat}_{q}(m \times n, S, r)$ is not polynomial in q, then $\overline{S}$ contains at least three entries in each row and column.

Theorems 3.6 and 3.8 allow for the efficient recursive computation of $\operatorname {mat}_{q}(m \times n, S, r)$ for a large class of sets S when m and n are small (though characterizing precisely which sets S seems hard). The code implemented in [9] includes these recursions. Unfortunately, for moderately large m and n, most sets S⊆[m]×[n] do not admit reductions by our theorems.

4 Formula for $\mathrm{mat}_{q}(n,\overline{B},r)$ when B has NE Property

In [7], Haglund proved the following result.

Theorem 4.1

[7, Theorem 1]

For every straight shape S _λ , we have

$$\operatorname {mat}_q(n,\overline{S_{\lambda}},r) = (q-1)^r q^{|\lambda|-r}R_r^\textsf{(SE)}\bigl(S_{\lambda},q^{-1} \bigr). $$

We now extend this result (using the same proof technique) to all shapes with the NE Property, that is, with the property that for any i′<i, j<j′, if (i,j), (i′,j), and (i,j′) belong to B, then (i′,j′) does as well.

Theorem 4.2

Fix any n and r and any set B⊆[n]×[n] with the NE Property. The number of n×n matrices over F _q of rank r whose support is contained in B is

$$ \operatorname {mat}_q(n,\overline{B},r) = (q-1)^r q^{\#B-r}R_r^\textsf{(NE)}\bigl(B,q^{-1}\bigr). $$

(4.3)

Proof

Choose a matrix A counted in $\operatorname {mat}_{q}(n, \overline{B},r)$, that is, whose support is in B, and perform Gaussian elimination in the following (north–east) order: traverse each column from bottom to top, starting with the leftmost (i.e., first) column. When you come to a nonzero entry (i.e., a pivot), use it to eliminate the entries to its north in the same column and to its east in the same row. See Fig. 3 for an example of this stage of the elimination process. Then move on to the next column and repeat until there is at most one nonzero entry in every row and column.

Fig. 3

NE elimination on a representative matrix counted in $\operatorname {mat}_{q}(n, \overline{B},r)$ with a pivot on (i,j) where B has the NE Property

By the NE Property, at each stage of the elimination process just described, we obtain another matrix counted in $\operatorname {mat}_{q}(n, \overline{B},r)$. After elimination, the positions of the pivots are a placement of r nonattacking rooks on B.

Given a fixed placement of r nonattacking rooks on B, let a be the number of cells in B that are directly north or directly east of a rook. There are (q−1)^r q ^a matrices of rank r whose support is in B that give this placement after the elimination procedure described above. It is not hard to see that $a=\#B-r-\operatorname {inv}_{\textsf{NE}}(C,B)$. Thus, summing over all placements or r nonattacking rooks, we obtain

$$\operatorname {mat}_q(n,\overline{B},r)=(q-1)^rq^{\#B-r}\sum _C\bigl(q^{-1}\bigr)^{\operatorname {inv}_\textsf{NE}(C,B)} =(q-1)^rq^{\#B-r}R_r^{\sf(NE)} \bigl(B,q^{-1}\bigr), $$

as desired. □

Note that a priori it is not clear that the expression on the right-hand side of Eq. (4.3) is a polynomial. However, this expression is a polynomial for the following reason: for any rook placement, there cannot be any more inversions than there are empty cells without rooks in them. There are #B cells unoccupied by zeros, and, of these, r have rooks in them. So the maximum value of $\operatorname {inv}_{\textsf{NE}}(C,B)$ is #B−r. Since this is the power of q at the beginning of the formula, there will not be any q ⁻¹ terms, and $\operatorname {mat}_{q}(n,\overline{B},r)$ is a polynomial.

Example 4.4

For n=4 and r=4, the set B=([4]×[4])∖{(1,1),(3,4),(4,1),(4,3),(4,4)} has the NE Property (as in Fig. 4(a)), and there are three placements of four rooks on B (as in Fig. 4(b)). The number of NE-inversions of these placements are 0, 1, and 1, respectively. Thus,

Fig. 4

(a) Set B with the NE Property. (b) Example of computing $\operatorname {mat}_{q}(n,\overline{B},r)$ when B has the NE Property. There are three placements of four rooks in B with 0, 1, and 1 NE-inversions, respectively. By Theorem 4.2, $\operatorname {mat}_{q}(4,\overline{B},4)=(q-1)^{4}q^{11-4}(1+2q^{-1})$

We give two corollaries of Theorem 4.2. First, since a straight shape S _λ has the NE Property, by comparing Haglund’s result and Theorem 4.2 we see that the (NE) and (SE) q-rook numbers of S _λ agree.

Corollary 4.5

For any straight shape S _λ , we have

$$R_r^\textsf{(NE)}(S_{\lambda},q)= R_r^\textsf{(SE)}(S_{\lambda},q). $$

(Recall that in general the (NE) and (SE) q-rook numbers of a general board do not agree; see for example Remark 2.3.)

Second, since any skew shape S _λ/μ has the NE Property, we have the following corollary:

Corollary 4.6

For any skew shape S _λ/μ ,

$$\operatorname {mat}_q(n,\overline{S_{\lambda/\mu}},r)=(q-1)^r f(q), $$

where f(q) is a polynomial with nonnegative integer coefficients.

Example 4.7

For λ/μ=4432/31, we have

In general, for skew shapes S _λ/μ , there is no product formula for $\operatorname {mat}_{q}(n,\overline{S_{\lambda/\mu}},r)$ analogous to (2.2), even when r=n.

5 Studying mat _q (n,S,r) when S is a Rothe diagram

Given a permutation $w\in \mathfrak{S}_{n}$ written as a word w=w ₁ w ₂⋯w _n where w _i is the image of w at i, the Rothe diagram R _w is the set

$$R_w = \bigl\{(i,j) \mid 1\leq i, j\leq n,\,\, w(i)>j,\,\, w^{-1}(j)>i\bigr\}. $$

Equivalently, R _w is the set of elements in [n]×[n] that do not lie directly south or directly east of entries (i,w _i ) of the permutation matrix of w. See Fig. 5 for some examples of Rothe diagrams. Note that #R _w is the number of inversions of w, that is, the number of pairs (i,j) such that i<j but w _i >w _j . Also, R _w has the following property: if (i,j) and (k,ℓ) are in R _w and i>k, j<ℓ, then the entry (k,j) is also in R _w . We call this the Le property of Rothe diagrams.²

Fig. 5

Representative matrices counted by $\operatorname {mat}_{q}(5,R_{w},r)$ where R _w is a Rothe diagram and w is (i) 41523 (vexillary), (ii) 21534 (skew-vexillary), (iii) w=31524 (not skew-vexillary). The entries $a_{i\,w_{i}}$ are in red

The main conjecture for Rothe diagrams, which has been verified for n≤7 using the results in Sect. 3 [9], is the following:

Conjecture 5.1

If R _w is the Rothe diagram of a permutation w in $\mathfrak{S}_{n}$ and 0≤r≤n then $\operatorname {mat}_{q}(n,R_{w},r)/(q-1)^{r}$ is a polynomial in q with nonnegative integer coefficients.

In Sect. 5.1 we give properties of Rothe diagrams that help in calculating $\operatorname {mat}_{q}(n,R_{w},r)$. In Sect. 5.2 we study Conjecture 5.1 for the family of permutations w such that R _w is the complement of a skew shape (after permuting rows and columns). The conjecture holds for such permutations by Corollary 4.6. In Theorem 5.4, we characterize these permutations.

5.1 Properties of mat _q (n,S,r) when S is a Rothe diagram

In this section we give some simple properties of $\operatorname {mat}_{q}(n,S,r)$ when S=R _w is the Rothe diagram of a permutation w. These properties are useful to simplify the size of computations involved in empirically confirming conjectures about $\operatorname {mat}_{q}(n,R_{w},r)$ like Conjectures 5.1 and 6.6.

If the permutation w is the word w ₁ w ₂⋯w _n , the reverse of w is the permutation re(w)=w _n w _n−1⋯w ₁. The complement of w is the permutation c(w)=u ₁ u ₂⋯u _n where u _i =n+1−i−w _i . In addition, the reverse complement of w is the permutation rc(w)=v ₁ v ₂⋯v _n where v _i =n+1−w _n+1−i . Lastly, the left-to-right maxima of w are the values w _i such that w _i >w _j for all j such that 1≤j<i.

Proposition 5.2

Given a permutation w in $\mathfrak{S}_{n}$ and its Rothe diagram R _w , we have

• (i) $\operatorname {mat}_{q}(n,R_{w},r)=\operatorname {mat}_{q}(n,R_{w^{-1}},r)$ and

• (ii) $\operatorname {mat}_{q}(n, R_{w},r) = \operatorname {mat}_{q}(n, R_{rc(w)},r)$.

Proof

It is easy to see that for any permutation w, the diagram $R_{w^{-1}}$ is the transpose of R _w , and the first statement follows immediately. We now consider the second statement.

Fix a permutation w with Rothe diagram R _w . Each element (i,j) of R _w corresponds to the inversion of w formed by the entries with matrix coordinates (i,w _i ) and $(w^{-1}_{j},j)$. In rc(w), these elements of w are transformed to (n+1−i,n+1−w _i ) and $(n + 1 - w^{-1}_{j}, n + 1 - j)$ and still form an inversion; in R _rc(w), this inversion corresponds to the element with coordinates $(n+1-w^{-1}_{j},n+1-w_{i})$. It follows immediately that the diagram R _rc(w) is the result of taking the transpose of R _w , rearranging rows and columns by multiplying on both sides by the permutation matrix of w, and rotating the result by 180^∘. □

Next we characterize the indices of the rows and columns of [n]×[n] entirely contained in $\overline{R_{w}}$. This is useful for computation because it is easy to express $\operatorname {mat}_{q}(n,S,r)$ in terms of values of $\operatorname {mat}_{q}$ for sets obtained by removing rows or columns that contain no elements of S.

Proposition 5.3

The kth column (row) of [n]×[n] is contained in $\overline{R_{w}}$ if and only if k is a left-to-right maximum of w (of w ⁻¹).

Proof

This follows from the definitions of R _w and of the left-to-right maxima. □

5.2 Skew-vexillary permutations

A permutation w is vexillary if its Rothe diagram, up to a permutation of its rows and columns, is the diagram of a partition. Call this partition λ(w). Then by Haglund’s Theorem 4.1, for vexillary permutations w, we have that

$$\operatorname {mat}_q(n, R_w,r) = \operatorname {mat}_q(n,S_{\lambda(w)},r)=(q-1)^rq^{n^2-\operatorname {inv}(w)-r}R_r^\textsf{(NE)} \bigl(\,\overline{S_{\lambda(w)}},q^{-1}\bigr). $$

It is well known that w is vexillary if and only if w avoids 2143 [10], i.e., there is no sequence 1≤i<j<k<l≤n in w such that w _j <w _i <w _l <w _k .

Next we give a characterization of permutations whose Rothe diagram, up to a permutation of rows and columns, is the complement of a skew shape. For such a permutation w, we have by Corollary 4.6 that $\operatorname {mat}_{q}(n,R_{w},r)/(q-1)^{r}$ is a polynomial with nonnegative integer coefficients. So Conjecture 5.1 holds for these permutations.

For the proof, we need the following definition: we say that a skew shape S _λ/μ in [n]×[n] is nonoverlapping if there is no row nor column that contains entries from both S _μ and $\overline{S_{\lambda}}$.

Theorem 5.4

The Rothe diagram of w=w ₁ w ₂⋯w _n is, up to permuting its rows and columns, the complement of a skew shape if and only if w can be decomposed as a ₁ a ₂⋯a _k b ₁ b ₂⋯b _n−k where a _i <b _j and each of a ₁ a ₂⋯a _k and b ₁ b ₂⋯b _n−k is 2143-avoiding.

Proof

First we prove the “if” direction. This argument is illustrated in Fig. 6. Suppose that w can be decomposed into a=a ₁ a ₂⋯a _k and b=b ₁ b ₂⋯b _n−k as in the theorem statement. Then the Rothe diagram R _w is block-diagonal, i.e., it consists of some entries in the upper-left k×k block and some in the lower-right (n−k)×(n−k) block, with no entries in the upper-right k×(n−k) block or lower-left (n−k)×k block. Furthermore, note that the upper-left and lower-right subdiagrams are identical to the Rothe diagrams of the permutations order-isomorphic to a ₁ a ₂⋯a _k and b ₁ b ₂⋯b _n−k , respectively.

Fig. 6

If w can be decomposed as a ₁ a ₂⋯a _k b ₁ b ₂⋯b _n−k where a _i <b _j and both a=a ₁ a ₂⋯a _k and b=b ₁ b ₂⋯b _n−k are 2143 avoiding, then $\overline{R_{w}}$ can be rearranged into a skew shape

Since both of these permutations are 2143-avoiding, and their Rothe diagrams in the upper-left and lower-right corners do not share any rows or columns in common, they can be rearranged independently to form two separate straight shapes. We may then rotate the straight shape corresponding to R _b by 180^∘ via permuting rows and columns (without changing the rearranged upper-left corner) to get a straight shape in the upper-left corner and an upside-down straight shape in the lower-right. This is the outside of a skew shape, as desired.

Second, we prove the “only if” direction of the theorem. Suppose that the diagram R _w , when rearranged, forms the complement of a skew shape S _λ/μ . This skew shape contains the column that was previously (i.e., before rearrangement) given by {(j,w ₁)∣j≥1}. Likewise, it contains the row that was previously given by $\{(w^{-1}_{1},j) \mid j \geq 1\}$. It follows that the skew shape S _λ/μ is nonoverlapping. After rearrangement, every entry of R _w either belongs to S _μ or $\overline{S_{\lambda}}$. We use this partition of the elements of R _w to identify the appropriate decomposition of w.

We color an entry of R _w blue if it belongs to S _μ after rearrangement, otherwise we color it red. We show the following claim: for every entry w _i of w, the elements of R _w in the same row or column as (i,w _i ) are either all blue or all red.

Since S _λ/μ is nonoverlapping, the entries of R _w in each row have the same color, and likewise for columns. If there is an entry (i,w _i ) with elements (i,j) and (k,w _i ) of R _w , then by the Le property of Rothe diagrams, (k,j) is also in R _w . Therefore all three entries have the same color, and the claim follows.

By the argument of the preceding paragraph, we may color the elements of w as follows: if (i,w _i ) is in the same row or column as a red entry of R _w , then we color w _i red, whereas if (i,w _i ) is in the same row or column as a blue entry of R _w , then we color w _i blue, and otherwise we leave w _i uncolored. We observe a few properties of the colored and uncolored elements of the permutation. First, inversions of w can only happen between elements of the same color. Second, w _i is uncolored if and only if w _i is not involved in any inversions. And third, the subword of the blue (respectively, red) elements of w is 2143-avoiding. This is because by definition, the entries of R _w in the same row or column as (i,w _i ) for blue (respectively, red) w _i are exactly the entries in S _μ (respectively, $\overline{S_{\lambda}}$) after rearrangement. This is equivalent to saying that the subword of the blue (respectively, red) elements of w is vexillary and thus 2143-avoiding.

From the three observations above it follows that the permutation w decomposes as u ₁ c ₁ u ₂ c ₂ u ₃ where (i) the u _i are (possibly empty) blocks of uncolored elements, c ₁ is the block of elements of one color of w, and c ₂ is the block of elements of the other color of w; (ii) the entries of each block are smaller than the entries of the following blocks, and (iii) the blocks c ₁ and c ₂ are 2143-avoiding. Finally, if we set a ₁ a ₂⋯a _k =u ₁ c ₁ and b ₁ b ₂⋯b _n−k =u ₂ c ₂ u ₃, we get a desired decomposition of w where a _i <b _j and a ₁ a ₂⋯a _k and b ₁ b ₂⋯b _n−k are 2143-avoiding. □

We call the above permutations skew-vexillary,³ and we denote by λ/μ(w) the skew shape whose complement is the rearrangement of R _w .⁴

Corollary 5.5

By Theorem 4.2, if w is skew-vexillary, then $\operatorname {mat}_{q}(n, R_{w}, r)/(q-1)^{r}$ is equal to $q^{n^{2}-\operatorname {inv}(w)-r}R_{r}^{\textsf{(NE)}}(S_{\lambda/\mu(w)},q^{-1})$, a polynomial with nonnegative integer coefficients. In particular, Conjecture 5.1 holds for skew-vexillary permutations.

If w is a skew-vexillary permutation, then every subpermutation of w is as well. This implies that skew-vexillarity may be rephrased as a pattern-avoidance condition. We do this now.

Proposition 5.6

The permutation $w \in \mathfrak{S}_{n}$ can be decomposed as w=a ₁⋯a _k b ₁⋯b _n−k such that a _i <b _j for all i and j and the permutations a ₁⋯a _k and b ₁⋯b _n−k avoid 2143 if and only if w avoids the nine patterns 24153, 25143, 31524, 31542, 32514, 32541, 42153, 52143, and 214365.

Proof

Call the decomposition in question an “SV-decomposition” (for Skew-Vexillary). First, we show that if w contains any of the nine patterns listed in the statement of the theorem, it does not have an SV-decomposition.

Let p be any of the eight patterns of length 5; it is easy to check that p is indecomposable, i.e., we cannot write p=uv with u, v nonempty and u _i <v _j for all i, j. Thus, if we write w=ab with a _i <b _j , we must have either p contained in a or p contained in b. Since p contains 2143, it follows that either a or b contains 2143, so this decomposition is not SV, as desired.

Now consider the case of the pattern 214365. Any decomposition of w decomposes 214365, and it is easy to see that in any of the four decompositions of 214365, one piece or the other contains a copy of 2143. This completes the proof that any permutation containing the given patterns has no SV-decomposition.

Now consider the converse. Suppose that w is not SV-decomposable. There are two cases.

If w is indecomposable and contains 2143, then w contains a minimal indecomposable permutation that contains 2143. The minimal 2143-containing indecomposable permutations are precisely the eight permutations of length 5 that we consider.

Finally, we show by induction that every decomposable but not SV-decomposable permutation contains one of the nine patterns. Choose such w and write w=ab with a _i <b _j . Without loss of generality, in this decomposition we have that a contains 2143. If b has a descent, it follows immediately that w contains 214365. Otherwise, w=a ₁⋯a _k (k+1)(k+2)⋯n. Observe that a permutation of this form has an SV-decomposition if and only if the shorter permutation a=a ₁⋯a _k has an SV-decomposition; thus, a has no SV-decomposition. If a is indecomposable, we have by the preceding paragraph that a contains one of the nine patterns; if a is decomposable, we have the same result by induction.

Putting the two cases together, every permutation that is not SV-decomposable contains at least one of the nine patterns, as desired. □

Remark 5.7

Vexillary permutations have many other interesting properties (see, for example, [12, Sects. 2.6.5 and 2.8.1]). For example, the Stanley symmetric function [16, Sect. 2] F _w of a vexillary permutation w equals the Schur function s _λ(w). Since a skew-vexillary permutation w with skew shape λ/μ(w) is SV-decomposable, F _w is the product s _{[n]×[n]∖λ} s _μ of Schur functions. Do other properties of vexillary permutations carry over to skew-vexillary permutations?

5.2.1 An enumerative aside

Any structurally interesting class of permutations calls out to be enumerated. Vexillary permutations were enumerated by West [20] (who showed that they are in bijection with 1234-avoiding permutations, which had been enumerated earlier by Gessel [6]). We now enumerate skew-vexillary permutations in terms of the generating function for vexillary permutations. For convenience, we denote by $\mathfrak{S}_{n}(2143)$ the set of vexillary permutations of length n. Also, given two permutations $u \in \mathfrak{S}_{i}$ and $v \in \mathfrak{S}_{j}$, we denote by w=u⊕v the permutation in $\mathfrak{S}_{i + j}$ defined by w _t =u _t for t∈[i] and w _t+i =i+v _t for t∈[j].

Theorem 5.8

Let V(x) be the ordinary generating function for 2143-avoiding permutations, and let SV(x) be the ordinary generating function for skew-vexillary permutations. We have

$$SV(x) = (1 - x)V(x)^2 - V(x) + \frac{1}{1 - x}. $$

Proof

Let V(x)=1+x+2x ²+6x ³+23x ⁴+⋯ be the ordinary generating function for 2143-avoiding permutations, and let I(x)=x ²+3x ³+13x ⁴+⋯ be the generating function for indecomposable 2143-avoiding permutations of length 2 or more. Suppose that $w \in \mathfrak{S}_{n}(2143)$ can be written w=u ₁⊕u ₂ where u ₁ and u ₂ are nonempty permutations. If both u ₁ and u ₂ contain an inversion, the four entries from the two inversions form an instance of 2143 in w, a contradiction. Thus, one of u ₁ and u ₂ must be the identity permutation. It follows immediately that every $w \in \mathfrak{S}_{n}(2143)$ other than the identity can be written as w=id _i ⊕u⊕id _k for identity permutations id _i and id _k (possibly of length 0) and an indecomposable 2143-avoiding permutation u. Thus,

$$V(x) = \frac{1}{1 - x} + \frac{I(x)}{(1 - x)^2}, $$

and so

$$I(x) = (1 - x)^2 V(x) + x - 1. $$

Now suppose that $w \in \mathfrak{S}_{n}$ has the property that w=u⊕v where u and v are 2143-avoiding permutations (possibly of length 0). It follows from the preceding analysis that either w is 2143-avoiding itself or we can write w=id _i ⊕u′⊕id _j ⊕v′⊕id _k where the ids are identity permutations (possibly of length 0) and u′ and v′ are indecomposable 2143-avoiding permutations of length 2 or more. Thus, the generating function SV(x) for such permutations is given by

as desired. □

If w is skew-vexillary, then $\operatorname {mat}_{q}(n,R_{w},r)/(q-1)^{r}$ is of the form $q^{n^{2}-\operatorname {inv}(w)-r}\*\sum_{\text{some } u \in \mathfrak{S}_{n}} q^{-\operatorname {inv}(u)}$ by Corollary 5.5. Another polynomial with this form is the Poincaré polynomial of the strong Bruhat order (see, e.g., [12, Sect. 2.1.2]) in $\mathfrak{S}_{n}$. In the next subsection we study the connections between these and $\operatorname {mat}_{q}(n,R_{w},n)$.

6 Poincaré polynomials, mat _q (n,R _w ,n), and q-rook numbers

A natural question when faced with a family of polynomials with positive integer coefficients is whether they count some nice combinatorial object. In this section, we investigate connections between our polynomials $\operatorname {mat}_{q}(n, R_{w}, n)$ (note in particular that we focus on the case of full rank) and certain well-known polynomials we define now.

As before, let $\operatorname {inv}(w)$ denote the number of inversions #{(i,j)∣i<j,w _i >w _j } of w. Recall the notion of the strong Bruhat order ≺ on the symmetric group [3, Chap. 2]: if t _ij is the transposition that switches i and j, we have as our basic relations that u≺u⋅t _ij in the strong Bruhat order when $\operatorname {inv}(u) + 1 = \operatorname {inv}(u \cdot t_{ij})$, and we extend by transitivity. Let $P_{w}(q)=\sum_{u \succeq w} q^{\operatorname {inv}(u)}$ be the (upper) Poincaré polynomial of w, where we sum over all permutations u that succeed w in the strong Bruhat order. Equivalently, P _w (q) is the rank-generating function of the interval [w,w ₀] in the strong Bruhat order where w ₀ is the largest element n  n−1⋯21 of this order.

Example 6.1

If w=3412, then the permutations in $\mathfrak{S}_{4}$ that succeed w in the Bruhat order are 3412,3421,4312, and 4321. The generating polynomial for this set by number of inversions is P ₂₁₄₃(q)=q ⁶+2q ⁵+q ⁴.

In [15], Sjöstrand gave necessary and sufficient conditions for P _w (q) to be equal to a q-rook number of a skew shape associated to w. Namely, the left hull H _L (w) of w is the smallest skew shape that covers w. Equivalently, H _L (w) is the union over noninversions (i,j) of w of the rectangles {(k,ℓ)∣w _i ≤k≤w _j ,i≤ℓ≤j}. See Fig. 7 for an example of the left hull of a permutation.

Fig. 7

Matrices indicating the (i) Rothe diagram and (ii) left hull of w=35142. The matrix entries $a_{i\,w_{i}}$ are in red

The following special case of a result by Sjöstrand characterizes when P _w (q) is equal to the rook polynomial of the left hull of the permutation w.

Theorem 6.2

[15, Corollary 3.3]

The Bruhat interval [w,w ₀] in $\mathfrak{S}_{n}$ equals the set of rook placements in the left hull H _L (w) of w (and in particular $q^{|\mu|}R_{n}^{\textsf{(SE)}}(H_{L}(w),q)=P_{w}(q)$ where μ is the shape such that H _L (w)=S _λ/μ for some λ) if and only if w avoids the patterns 1324,24153,31524, and 426153.

If w is a skew-vexillary, then by Corollary 5.5 we know that $\operatorname {mat}_{q}(n,R_{w},n)/ (q-1)^{n}$ is (up to a power of q) a q-rook number. Next we show that this q-rook number is essentially a q-rook number of the left hull of a permutation v that avoids the four patterns above. Therefore, by Theorem 6.2 $\operatorname {mat}(n,R_{w},n)/(q-1)^{n}$ is (up to a power of q) a Poincaré polynomial P _v (q).

6.1 mat _q (n,R _w ,n) for skew-vexillary permutations is a Poincaré polynomial

In this section we use Sjöstrand’s result (Theorem 6.2) to show that for skew-vexillary permutations w, the function $\operatorname {mat}_{q}(n,R_{w},n)/(q-1)^{n}$ is not only a polynomial with nonnegative coefficients but, up to a power of q, is a Poincaré polynomial.

Proposition 6.3

If w is skew-vexillary, then

$$\operatorname {mat}_q(n, R_w, n) = q^{\binom{n}{2}-\operatorname {inv}(w)}(q-1)^n \cdot P_v(q) $$

for some permutation $v \in \mathfrak{S}_{n}$.

Proof

If w is skew-vexillary, then by Corollary 5.5 we know that

$$\operatorname {mat}_q(n, R_w,n)/(q-1)^n = q^{n^2 - \operatorname {inv}(w) - n}R_n^\textsf{(NE)}\bigl(S_{\lambda/\mu(w)},q^{-1} \bigr), $$

where R ^(NE)(S _λ/μ(w),q) is the rook polynomial of S _λ/μ(w), the nonoverlapping skew shape whose complement is the rearrangement of R _w . We will show that this polynomial is the Poincaré polynomial P _v (q) of a permutation v.

Define the permutation matrix of v as follows: let w=a ₁ a ₂⋯a _k b ₁ b ₂⋯b _n−k be the decomposition promised by Theorem 5.4. Let λ=(λ ₁,…,λ _n ) and μ=(μ ₁,…). For i=1,…,k, let v _i =min(([n]∖[μ _i ])∖{v ₁,…,v _i−1}), and for j=1,…,n−k, let v _n+1−j =max([λ _n−j ]∖{v _n−j+1,…,v _n }). This defines a 0–1 matrix with exactly one 1 in every row; it follows from the proof of Theorem 5.4 that this matrix is in fact a permutation matrix (with {v ₁,…,v _k }=[k] and {v _k+1,…,v _n }={n−k+1,…,n}). See Fig. 8 for an example of this construction. It is clear that S _λ/μ(w)=H _L (v). Also by Proposition 2.4 we have that $q^{\binom{n}{2}-|\mu|}R_{n}^{\textsf{(NE)}}(S_{\lambda/\mu}(w),q^{-1})=R_{n}^{\textsf{(SE)}}(H_{L}(v),q)$.

Fig. 8

Example of Proposition 6.3. For the permutations w and v shown, we have that $\operatorname {mat}_{q}(5,R_{w},5)/(q-1)^{5} = q^{\binom{5}{2}-\operatorname {inv}(w)}\cdot P_{v}(q)$

By construction, the prefix v ₁⋯v _k avoids 132, the suffix v _k+1⋯v _n avoids 213, and v _j >v _i for j≥k+1 and i≤k. It is easy to see that the set of permutations with such a decomposition is closed under containment of patterns and does not contain any of the permutations 1324, 24153, 31524, and 426153. Therefore, every permutation in this set, and in particular v, avoids these four patterns. By Theorem 6.2 it follows that $q^{|\mu|}R_{n}^{\textsf{(SE)}}(H_{L}(v),q)=P_{v}(q)$. Thus,

as desired. □

Example 6.4

By Theorem 5.4, the permutation w=21534 is skew-vexillary. After rearranging rows and columns (see Fig. 8), the skew shape S _λ/μ(w) is S _55553/1. The associated v is 21453, and we have

Remark 6.5

Note that the result above does not hold for all Rothe diagrams. There exist permutations w for which there does not exist any permutation v such that $\operatorname {mat}_{q}(n, R_{w},r) = q^{\binom{n}{2}-\operatorname {inv}(w)}(q-1)^{r}P_{v}(q)$. For example, take w=31524 (see Fig. 5(iii) and Table 1). In this case,

$$\operatorname {mat}_q(5, R_{31524},5)=q^{6}(q-1)^5 \bigl(q^{10}+4q^9+9q^8+12q^7+10q^6+5q^5+q^4 \bigr). $$

One can show (either by computer search or by a direct argument about the possible structure of the inversions) that there is no permutation v in $\mathfrak{S}_{5}$ such that P _v (q)=q ¹⁰+4q ⁹+9q ⁸+12q ⁷+10q ⁶+5q ⁵+q ⁴.

Table 1 For the four special patterns w of Conjecture 6.6, we give $\operatorname {mat}_{q}(n,R_{w},n)/((q-1)^{n}q^{k})$ where $k=\binom{n}{2}-\operatorname {inv}(w)$, the Poincaré polynomials P _w (q), and $q^{a}R_{n}^{\textsf{(SE)}}(H_{L}(w),q)$ where a is the size of the subtracted partition of the skew shape H _L (w)

We have shown that for skew-vexillary permutations w, $\operatorname {mat}_{q}(n,R_{w},n)/ (q-1)^{n}$ is equal (up to a power of q) to the Poincaré polynomial of some permutation v. Next, we consider the problem of classifying permutations w such that $\operatorname {mat}_{q}(n,R_{w},n)/(q-1)^{n}$ is equal (up to a power of q) to the Poincaré polynomial of the same permutation.

6.2 Further relationships between mat _q (n,R _w ,n) and Poincaré polynomials

Computational evidence for n≤7 [9] suggests the following conjecture.

Conjecture 6.6

Fix a permutation w in $\mathfrak{S}_{n}$ and let R _w be its Rothe diagram. We have that $\operatorname {mat}_{q}(n,R_{w},n)/(q-1)^{n}$ is coefficient-wise less than or equal to $q^{\binom{n}{2}-\operatorname {inv}(w)}P_{w}(q)$. We have equality if and only if w avoids the patterns 1324, 24153, 31524, and 426153.

Remark 6.7

The patterns that appear in Conjecture 6.6 and in Theorem 6.2 are the same. Also, the reverses 4231,35142,42513, and 351624 of these patterns have appeared in related contexts, in work of Gasharov and Reiner [5] in a conjecture of Postnikov [14] proved by Hultman, Linusson, Shareshian, and Sjöstrand [8] and by Gasharov and Reiner [5]. This suggests further interesting connections. These permutations were recently enumerated by Albert and Brignall [1].

The values of the three polynomials $\operatorname {mat}_{q}(n,R_{w},n)/(q-1)^{n}$, P _w (q), and $R^{\textsf{(SE)}}_{n}(H_{L}(w),q)$ when w is equal to the four patterns of Conjecture 6.6 are shown in Table 1. In these cases the three polynomials are all different. By Theorem 6.2 and Theorem 4.2, Conjecture 6.6 is equivalent to the following:

Conjecture 6.8

Fix a permutation w in $\mathfrak{S}_{n}$, let R _w be its Rothe diagram, and let $a_{w} = n^{2} - \#H_{L}(w) - \operatorname {inv}(w)$. We have that $\operatorname {mat}_{q}(n,R_{w},n)/(q-1)^{n}$ is coefficient-wise less than or equal to $q^{a_{w}}\operatorname {mat}_{q}(n,\overline{H_{L}(w)},n)/(q-1)^{n}$. We have equality if and only if w avoids the patterns 1324, 24153, 31524, and 426153.

This conjecture is not true for matrices of lower rank. For example, for $w=21\in \mathfrak{S}_{2}$, we have $\operatorname {mat}_{q}(2,R_{21},1)/(q-1)=2q+1$ and $\operatorname {mat}_{q}(2,\overline{H_{L}(21)},1)/(q-1)=2$.

Remark 6.9

If Conjecture 6.8 holds, then by Theorem 4.2 and [11, Proposition 5.1] it follows that whenever w avoids the four patterns, the shapes $\overline{R_{w}}$ and H _L (w) have the same number of placements of n nonattacking rooks. This is not obvious since for such permutations, the shapes are distinct even after permuting rows and columns. Moreover, computer experiments for n≤7 [9] suggest that the converse is also true, i.e., if w contains any of the four patterns, the shapes have different numbers of rook placements. This apparent equivalence of necessary and sufficient conditions between the “q case” and the “q=1 case” does not necessarily hold in similar settings (see [13, Theorem 7] and [8, Theorem 3.4] for an example).

We end by giving a very preliminary step in proving these conjectures.

Proposition 6.10

If w is a 1324-avoiding permutation, then the complement of H _L (w) has at least as many entries as the Rothe diagram R _w of w.

Proof

Let w be a 1324-avoiding permutation. We give a one-to-one map φ between the entries of the Rothe diagram R _w and the complement of the left hull H _L (w).

Given an entry (i,j) in R _w , we have two possibilities: either there is or there is not an entry (k,w _k ) of w such that k<i and w _k <j (i.e., an entry of w NW of (i,j)). Let A _w be the set of entries of R _w of the first type, and let B _w be the set of entries of the second type. If (i,j)∈A _w then define φ(i,j)=(i,j). If instead (i,j)∈B _w , then define $\varphi(i,j)=(w^{-1}_{j},w_{i})$. See Fig. 9 for an illustration of φ. We show that φ is well defined and injective.

Fig. 9

Example of Proposition 6.10. For the permutation w=4132, the Rothe diagram is R _w ={(1,1),(1,2),(1,3),(3,2)} and the left-hull $\overline{H_{L}(w)}=\{(1,1),(1,2),(1,3), (2,4),(3,4),(4,3),(4,4)\}$. The map φ is given by (1,j)↦(1,j) for j=1,2,3 and (3,2)↦(4,3)

Choose (i,j) in R _w . There are no entries of w above (i,j) in the same column or to its left in the same row. If in addition (i,j) is in A _w , then by the definition of the left hull the entry (i,j) is not in H _L (w). In this case, $\varphi(i,j)=(i,j)\in \overline{H_{L}(w)}$, as desired.

On the other hand, if (i,j) is in B _w , then there is some entry (k,w _k ) of w with k<i and w _k <j. Since w is 1324-avoiding, there can be no entry (ℓ,w _ℓ ) of w such that $\ell\geq w^{-1}_{j}$ and w _ℓ ≥w _i . Thus, $\varphi(i,j)=(w^{-1}_{j},w_{i}) \in \overline{H_{L}(w)}$. This completes the proof that the map φ is well defined.

Finally, we show that φ is one-to-one. Since φ is defined piecewise, it is enough to show that φ is one-to-one on A _w and B _w and that φ(A _w ) and φ(B _w ) are disjoint. The injectivity on A _w is trivial. The injectivity on B _w follows since w is a permutation and so $(w^{-1}_{j},w_{i})$ uniquely defines (i,j). Moreover, $\overline{H_{L}(w)}$ has two components; φ(A _w ) is the NW component, while φ(B _w ) is contained in the SE component, and so the images are disjoint. This completes the proof that φ is one-to-one. □