Journal of Algebraic Combinatorics

, Volume 39, Issue 2, pp 429–456

# Counting matrices over finite fields with support on skew Young diagrams and complements of Rothe diagrams

• Aaron J. Klein
• Joel Brewster Lewis
• Alejandro H. Morales
Article

## Abstract

We consider the problem of finding the number of matrices over a finite field with a certain rank and with support that avoids a subset of the entries. These matrices are a q-analogue of permutations with restricted positions (i.e., rook placements). For general sets of entries, these numbers of matrices are not polynomials in q (Stembridge in Ann. Comb. 2(4):365, 1998); however, when the set of entries is a Young diagram, the numbers, up to a power of q−1, are polynomials with nonnegative coefficients (Haglund in Adv. Appl. Math. 20(4):450, 1998).

In this paper, we give a number of conditions under which these numbers are polynomials in q, or even polynomials with nonnegative integer coefficients. We extend Haglund’s result to complements of skew Young diagrams, and we apply this result to the case where the set of entries is the Rothe diagram of a permutation. In particular, we give a necessary and sufficient condition on the permutation for its Rothe diagram to be the complement of a skew Young diagram up to rearrangement of rows and columns. We end by giving conjectures connecting invertible matrices whose support avoids a Rothe diagram and Poincaré polynomials of the strong Bruhat order.

### Keywords

Rook placements Finite fields Bruhat order Rothe diagrams Pattern avoidance q-analogues

## 1 Introduction

We study certain q-analogues of permutations with restricted positions or, equivalently, of placements of nonattacking rooks. The q-analogue of permutations we work with is invertible n×n matrices over the finite field Fq with q elements, as in [17, Chap. 1]. Then the analogue of permutations with restricted positions is invertible matrices over Fq with some entries required to be zero.

Specifically, given a subset S of {1,2,…,n}×{1,2,…,n}, let $$\operatorname {mat}_{q}(n,S,r)$$ be the number of n×n matrices over Fq with rank r, none of whose nonzero entries lie in S. This is clearly an analogue (in the plain English meaning) of the problem of counting permutations whose permutation matrix has no 1 in the position of any entry of S, but actually much more can be said. In [11, Proposition 5.1] it was shown that $$\operatorname {mat}_{q}(n,S,r)/(q-1)^{r}$$ is in fact an enumerativeq-analogue of permutations with restricted positions; that is, its value, modulo (q−1), counts the placements of r nonattacking rooks on the complement of S.

The function $$\operatorname {mat}_{q}(n,S,r)$$ can exhibit a variety of different behaviors, as seen in the following three examples.

### Examples 1.1

1. 1.
When S=∅, $$\operatorname {mat}_{q}(n,\varnothing,n)$$ is the number of n×n invertible matrices over Fq, which is
$$\bigl(q^{n}-1\bigr) \bigl(q^{n}-q\bigr)\cdots \bigl(q^n-q^{n-1}\bigr)=q^{\binom{n}{2}}(q-1)^n \prod_{i=1}^{n} \bigl(1+q+ \cdots+q^{i-1}\bigr).$$
The term $$\prod_{i=1}^{n}(1+q+\cdots+q^{i-1})$$ in the product is a polynomial with positive coefficients and in fact is the generating series for permutations in $$\mathfrak{S}_{n}$$ by number of inversions.

2. 2.
When n=3 and S is the diagonal {(1,1),(2,2),(3,3)}, we have
$$\operatorname {mat}_q\bigl(3,\bigl\{(1,1),(2,2),(3,3)\bigr\},3\bigr)=(q-1)^3 \bigl(q^3+2q^2-q\bigr).$$
The number of invertible n×n matrices for general n over Fq with zero diagonal was computed in [11, Proposition 2.2]; as in this example, it is of the form (q−1)nf(q) for a polynomial f with both positive and negative coefficients.

3. 3.
When n=7, Stembridge [18] found a set F with 28 elements (shown in Fig. 1) such that $$\operatorname {mat}_{q}(7,F,7)$$ is given by a quasi-polynomial in q, that is, by two distinct polynomials depending on whether q is even or odd. The set F is the complement of the incidence matrix of the Fano plane.

From the examples above we see that $$\operatorname {mat}_{q}(n,S,r)$$ is not necessarily a polynomial in q, and if it is a polynomial in q, it might or might not be of the form (q−1)rf(q) where f(q) is a polynomial with nonnegative integer coefficients. Then a natural question to ask is the following:

### Question 1.2

What families of setsSare there such that$$\operatorname {mat}_{q}(n,S,r)/(q-1)^{r}$$is (i) not a polynomial inq, (ii) a polynomial inq, or (iii) a polynomial inqwith nonnegative integer coefficients?

In the remainder of this introduction, we give a summary of our progress toward answering this question.

### 1.1 Outline and summary of results

In Sect. 2, we give the definitions and notation that will be used throughout the paper including the definition and some properties of q-rook numbers.

In Sect. 3, we address general conditions on r and S under which the function $$\operatorname {mat}_{q}(n,S,r)$$ is always a polynomial in q. We show that if r=1, then $$\operatorname {mat}_{q}(n, S, 1)$$ is a polynomial in q for any set S, though not necessarily with nonnegative coefficients. (It is an open question whether there is a set S such that $$\operatorname {mat}_{q}(n, S, 2)$$ is nonpolynomial in q.) Our main result of this section is to extend work of Stembridge [18] to give reductions to compute $$\operatorname {mat}_{q}(n, S, r)$$ in terms of smaller instances of similar problems when S has a row or column with very few or very many entries.

In the rest of the paper, we discuss special families of sets S such that $$\operatorname {mat}_{q}(n,S,r)/(q-1)^{r}$$ is a polynomial in q with nonnegative integer coefficients. Haglund [7] showed that if the set S is a straight shape, then $$\operatorname {mat}_{q}(n,\overline{S},r)/(q-1)^{r}$$ is a polynomial with nonnegative integer coefficients. Our second main result, proved in Sect. 4, is to extend this to complements of skew shapes.

### Corollary 4.6

For any skew shapeSλ/μ,
$$\operatorname {mat}_q(n,\overline{S_{\lambda/\mu}},r) = (q-1)^r f(q),$$
wheref(q) is a polynomial with nonnegative integer coefficients.

In fact, we show that this is true for an even larger class of shapes than skew shapes, namely those that have what we call the North–East Property. Also, because $$\operatorname {mat}_{q}(n,S,r)$$ is invariant under permuting rows and columns, we have that $$\operatorname {mat}_{q}(n,\overline{S},r)/(q-1)^{r}$$ is a polynomial with nonnegative integer coefficients for any set S that is a straight or skew shape after permuting rows and columns.

Another natural family of diagrams is the collection of Rothe diagrams of permutations, which appear in the study of Schubert calculus. The Rothe diagram Rw of a permutation w is a subset of {1,2,…,n}×{1,2,…,n} whose cardinality is equal to the number of inversions of w; it is given by
$$R_w = \bigl\{(i,j) \mid 1\leq i, j\leq n,\,\, w(i)>j,\,\, w^{-1}(j)>i\bigr\}.$$
See Fig. 5 for some examples of Rothe diagrams. Lascoux and Schützenberger [10] showed that the Rothe diagram Rw of a permutation w is a straight shape up to permutation of rows and columns if and only if w, written as a word w1w2wn, avoids the permutation pattern 2143 (i.e., there is no sequence i<j<k<l such that wj<wi<wl<wk). Our third main result gives an analogous criterion for the case of complements of skew shapes.

### Theorem 5.4

The Rothe diagramRwof a permutationwis, up to permuting its rows and columns, the complement of a skew shape if and only ifwcan be decomposed asw=a1a2akb1b2bnkwhereai<bjfor alliandj, and botha1a2akandb1b2bnkare 2143-avoiding.

We also show that this condition is equivalent to the statement that w avoids the nine patterns 24153, 25143, 31524, 31542, 32514, 32541, 42153, 52143, and 214365, and we express the generating series for these permutations in terms of the generating series for 2143-avoiding permutations.

By Corollary 4.6, if w satisfies the condition above, then $$\operatorname {mat}_{q}(n,R_{w},r)/(q-1)^{r}$$ is a polynomial with nonnegative integer coefficients. Surprisingly, computer calculations for n≤7 and 0≤rn [9] suggest that $$\operatorname {mat}_{q}(n,R_{w},r)/(q-1)^{r}$$ is a polynomial with nonnegative integer coefficients for all permutations w (see Conjecture 5.1). Moreover, computer calculations also suggest that when w avoids the permutation patterns 1324,24153,31524, and 426153, we have that $$\operatorname {mat}_{q}(n,R_{w},n)/(q-1)^{n}$$ is (up to a power of q) the Poincaré polynomial$$P_{w}(q) = \sum_{u\geq w} q^{\operatorname {inv}(u)}$$, where the sum is over all permutations u of n above w in the strong Bruhat order (see Conjecture 6.6). Interestingly, these four patterns have appeared in related contexts [5, 8, 14, 15].

Supplementary code for calculating $$\operatorname {mat}_{q}(n,S,r)$$ and other related objects and data generated by this code to test the conjectures in Sect. 6 are available at the following website: http://sites.google.com/site/matrixfinitefields/.

## 2 Definitions

We denote [n]={1,2,…,n}. The support of a matrix A is the set of indices (i,j) of the nonzero entries aij≠0. Fix integers n and r such that n≥1 and nr≥0, and let S be a subset of [n]×[n]. We define $$\operatorname {mat}_{q}(n,S,r)$$ to be the number of n×n matrices over Fq with rank r and support contained in $$\overline{S}$$, the complement of S. That is, $$\operatorname {mat}_{q}(n, S, r)$$ counts matrices A of rank r such that if (i,j)∈S, then aij=0. We consider the problem of computing $$\operatorname {mat}_{q}(n,S,r)$$.

We now define several special types of diagrams that will be important to us in what follows. Examples of these diagrams are given in Fig. 2. We say that S⊆[n]×[n] is a straight shape if its elements form a Young diagram of a partition. (Throughout this paper we use English notation and matrix coordinates for partitions.) Thus, to every integer partition λ with at most n parts and with largest part at most n (i.e., to each sequence of integers (λ1,λ2,…,λn) such that nλ1λ2≥⋯≥λn≥0), there is an associated set S=Sλ={(i,j)∣1≤jλi}. We denote by |λ| the sizeλ1+λ2+⋯ of the shape λ. This is also the number of entries in Sλ. Similarly, if λ and μ are partitions such that SμSλ, then we say that the set SλSμ is a skew shape, and we denote it by Sλ/μ. Lastly, we say that a set S⊆[n]×[n] has the North–East (NE) Property if for all i,i′,j,j′∈[n] such that i′<i and j<j′, we have that if (i,j),(i′,j), and (i,j′) are in S, then so is (i′,j′). Note that for any partitions λ and μ, $$S_{\lambda}, \overline{S_{\lambda}}$$, and Sλ/μ have the NE Property. But $$\overline{S_{\lambda/\mu}}$$ in general does not have this property.

We denote by $$\mathfrak{S}_{n}$$ the group of permutations on [n]. We write permutations as words w=w1w2wn where wi is the image of w at i. Let $$\operatorname {inv}(w)$$ denote the number of inversions #{(i,j)∣i<j,wi>wj} of w. We also identify each permutation w with its permutation matrix, the n×n 0–1 matrix with 1s in positions (i,wi).

We think of the 1s in a permutation matrix as n nonattacking rooks on [n]×[n]. In this case, the number of inversions of the permutation is exactly the number of elements in [n]×[n] that are not directly below/south (in the same column) or to the right/east (in the same row) of any placed rook. We generalize this as follows. Given a subset B of [n]×[n] (sometimes called a board) and a rook placement C of r nonattacking rooks on B, the SE-inversion number$$\operatorname {inv}_{\textsf{SE}}(C,B)$$ is the number of elements in Bnot directly south (in the same column) or to the east (in the same row) of any placed rook. Then the rth (SE) q-rook number of Garsia and Remmel [4] is
$$R^\textsf{(SE)}_r(B,q) = \sum_{C} q^{\operatorname {inv}_{\textsf{SE}}(C,B)},$$
where the sum is over all rook placements C of r nonattacking rooks on B. We define the north–east inversion number $$\operatorname {inv}_{\textsf{NE}}(C,B)$$ and rook polynomial $$R^{\textsf{(NE)}}_{r}(B,q)$$ analogously.

### Proposition 2.1

[4] Given an integer partitionλ=(λ1,λ2,…,λn) such thatnλ1λ2≥⋯≥λn≥0, setSλ={(i,j)∣1≤in,1≤iλj}. The Garsia–Remmelq-rook number$$R_{n}^{\textsf{(SE)}}(S_{\lambda},q)$$is
$$R_n^\textsf{(SE)}(S_{\lambda},q) = \prod _{i=1}^n [\lambda_{n-i+1}-i+1]_q,$$
(2.2)
where [m]q=1+q+q2+⋯+qm−1.

### Remark 2.3

We will see as a corollary of Theorems 4.1 and 4.2 that for a straight shape λ, $$R_{r}^{\textsf{(SE)}}(S_{\lambda},q)= R_{r}^{\textsf{(NE)}}(S_{\lambda},q)$$. However, this is not true for all skew shapes. For example, if λ/μ=4432/31, we have $$R_{3}^{\textsf{(SE)}}(S_{4432/31},q)=1+6q^{2}+5q^{3}+3q^{4}+2q^{5}+q^{6}$$ and $$R_{3}^{\textsf{(NE)}}(S_{4432/31},q)=2q+8q^{2} + 7q^{3}+q^{4}$$. But for skew shapes in the case of n rooks we do have an analogous relation, as the following result shows.

### Proposition 2.4

For a skew shapeSλ/μ⊆[n]×[n] we have
$$R_n^\textsf{(SE)}(S_{\lambda/\mu},q) =q^{\binom{n}{2}-|\mu|}\cdot R_n^{\sf(NE)}\bigl(S_{\lambda/\mu},q^{-1}\bigr).$$

### Proof

For each rook placement of n rooks on Sλ/μ, the number of SE-inversions is equal to the number of inversions of the associated permutation w minus the size of μ. On the other hand, the number of NE-inversions of this rook placement on Sλ/μ is $$\binom{n}{2}$$ minus the number of inversions of w. The result follows. □

## 3 General polynomiality results

In this section, we give some general conditions under which $$\operatorname {mat}_{q}(n, S, r)$$ is a polynomial. In Sect. 3.1, we show that for any n and S, the function $$\operatorname {mat}_{q}(n, S, 1)$$ is polynomial in q. In Sect. 3.2, we give reduction formulas for computing $$\operatorname {mat}_{q}(n, S, r)$$ in terms of smaller instances when S has a row or column with either very few or very many entries.

Throughout this section we work with rectangular matrices of any dimensions rather than just square matrices. Thus, in this section, for integers m, n and r and a subset S of [m]×[n], we denote by $$\operatorname {mat}_{q}(m \times n, S, r)$$ the number of m×n matrices of rank r over Fq whose support avoids S.

### 3.1 Polynomial formula for the rank-one case matq(n,S,1)

In Fig. 1 we showed an example by Stembridge [18] of a set S⊆[7]×[7] such that $$\operatorname {mat}_{q}(7,S,7)$$ is not a polynomial in q. In this paper, we mainly focus on studying certain families of sets S where $$\operatorname {mat}_{q}(n,S,r)$$ is a polynomial in q. But before looking at particular sets S, we consider the rank r=1 case for an arbitrary set S.

### Proposition 3.1

For anymandnand any setS⊆[m]×[n], $$\operatorname {mat}_{q}(m \times n,S,1)$$is a polynomial inq.

### Proof

Fix m, n, and S. We count matrices with a given collection of nonzero rows. Given a nonempty subset T⊆[m] of rows, let aS(T) be the number of columns with no entries which are both in one of the rows of T and in S. Then there are exactly $$(q^{a_{S}(T)} - 1)(q - 1)^{\# T - 1}$$ matrices of rank 1 over Fq whose support avoids S and whose nonzero rows are exactly those in T. It follows immediately that
$$\operatorname {mat}_q(m \times n, S, 1) = \sum _{\substack{T \subseteq [m] \\ \text{nonempty}}} \bigl(q^{a_S(T)} - 1\bigr) (q - 1)^{\# T - 1}$$
(3.2)
is a polynomial in q. □

### Example 3.3

Take the 4×4 shape S={(i,i)∣1≤i≤4}. Then (In fact, one can show that if S is the diagonal {(i,i)∣1≤in}, then $$\operatorname {mat}_{q}(n\times n,S,1) = \frac{1}{q-1}((2q-1)^{n} - 2q^{n}+1)$$.)

### Remark 3.4

In later sections of this paper, we show that for certain diagrams S (straight shapes, skew shapes, and conjecturally Rothe diagrams of permutations), the function $$\operatorname {mat}_{q}(n, S, r)/(q - 1)^{r}$$ is not only a polynomial in q but also has nonnegative coefficients. However, this is not the case for matrices of rank 1: although each summand is a power of q−1 times a polynomial with positive coefficients, the powers of q−1 differ. So, as in Example 3.3, negative coefficients can turn up for certain choices of S. Interestingly, if we substitute t=q−1 into (3.2), we obtain a polynomial in t with nonnegative coefficients. Is this true more generally?

### 3.2 Reduction formulas when S has rows with few or many entries

In [18, Theorem 8.2], Stembridge gave some structural restrictions on a minimal set S such that $$\operatorname {mat}_{q}(n, S, n)$$ is nonpolynomial. In particular, he showed that in such a minimal example, every row contains at least three entries of $$\overline{S}$$ and at least two entries of S. In this section, we push his results slightly further: we show that for any rank r, if either S or $$\overline{S}$$ has a row with at most two entries, then we can express $$\operatorname {mat}_{q}(m \times n, S, r)$$ as a linear combination with polynomial coefficients of similar expressions for matrices of strictly smaller size. (Of course, the same arguments apply to columns as well as to rows.) Furthermore, we explain why this approach cannot be used in the case where S contains three entries in some row. This does not settle the question of whether there are some m and n and a set S⊆[m]×[n] with only three entries per row such that $$\operatorname {mat}_{q}(m \times n, S, r)$$ is nonpolynomial in q, though we conjecture that such examples exist; in Stembridge’s example [18] (see Fig. 1), the set S has four entries per row, and $$\overline{S}$$ has three.

For convenience, throughout this section we write expressions like $$\operatorname {mat}_{q}(a \times b, S, r)$$ without worrying whether S⊆[a]×[b], where properly we should write $$\operatorname {mat}_{q}( a \times b, ([a] \times [b])\cap S, r)$$.

#### 3.2.1 Reduction when S has at most two entries in some row

We wish to show that if S has two or fewer entries in some row, then $$\operatorname {mat}_{q}(m \times n, S, r)$$ can be reduced to a sum of polynomial multiples of similar but simpler expressions. We begin with a useful proposition that we use in the proof of Theorem 3.6 to do case analysis.

### Proposition 3.5

Suppose that$$M = [{\bf v}_{1} \ {\bf v}_{2} \ \cdots \ {\bf v}_{n}]$$is anm×nmatrix overFqof rankrand that the submatrix$$[{\bf v}_{n - k + 1} \ \cdots \ {\bf v}_{n}]$$has rankr′. The number of tuples$${\bf w} = (w_{1}, w_{2}, \ldots, w_{n - k}) \in \mathbf {F}_{q}^{n - k}$$such thatisqrr. For the otherqnkqrrtuples, this matrix has rankr+1.

### Proof

This is just the rank-nullity theorem: vectors w that do not increase the rank are those that (when augmented by k 0s) lie in the row space of M. The portion of the row space with last k coordinates equal to 0 is exactly the kernel of the projection onto these last k coordinates. The image of this projection has dimension r′, so the kernel has dimension rr′, as desired. □

Now we use this to prove the main result of this section.

### Theorem 3.6

Suppose thatS⊆[m]×[n] contains at most two entries in themth row. Then$$\operatorname {mat}_{q}(m \times n, S, r)$$is equal to a linear combination of similar expressions for smaller matrices with coefficients inZ[q].

### Corollary 3.7

Ifm, nandSare chosen minimal (in the sense of row- or column-removal) so that$$\operatorname {mat}_{q}(m \times n, S, r)$$is not polynomial inq, thenScontains at least three entries in each row and column.

The rest of this subsection is devoted to the proof of Theorem 3.6.

### Proof

Consider the mth row of the matrix-to-be: it contains zero, one, or two forced 0s (i.e., elements of S) and either lies in a dimension-r space spanned by the first m−1 rows or lies outside their (r−1)-dimensional span. We separately compute the number of matrices contributed in each of these three cases.
Case 1:
The set S contains no elements in the mth row. We have two possibilities: first, it might be that the first m−1 rows of the matrix span a space of dimension r and the last row lies in this space. Then we would have $$\operatorname {mat}_{q}((m - 1) \times n, S, r)$$ choices for the first m−1 rows and qr choices for the last row, for a total contribution of $$\operatorname {mat}_{q}((m - 1) \times n, S, r) \cdot q^{r}$$. Second, it might be that the first m−1 rows span a space of dimension r−1 and the last row lies outside this space. Then we would have $$\operatorname {mat}_{q}((m - 1) \times n, S, r - 1)$$ choices for the first m−1 rows and qnqr−1 choices for the last row, for a total contribution of $$\operatorname {mat}_{q}((m - 1) \times n, S, r - 1) \cdot (q^{n} - q^{r-1})$$. Thus, the total contribution in this case is
$$\operatorname {mat}_q\bigl((m - 1) \times n, S, r\bigr) \cdot q^r + \operatorname {mat}_q\bigl((m - 1) \times n, S, r - 1\bigr) \cdot \bigl(q^n - q^{r-1}\bigr).$$
Observe that the two instances of the function $$\operatorname {mat}_{q}$$ that appear in this expression involve matrices strictly smaller than m×n.
Case 2:
The set S contains one element in the mth row, without loss of generality the entry (m,n). We have two cases depending on the dimension of the space spanned by the nth column, i.e., whether this column is zero or not.
1. (a)

We count matrices in which the nth column is the zero vector. In this case, we may choose the first m−1 rows to span a space of dimension r in $$\operatorname {mat}_{q}((m - 1)\times(n - 1), S, r)$$ ways and choose the last row in qr ways, or we may choose the first m−1 rows to span a space of dimension r−1 in $$\operatorname {mat}_{q}((m - 1)\times(n - 1), S, r - 1)$$ ways and choose the last row in qn−1qr−1 ways.

2. (b)

We count matrices in which the nth column is not the zero vector. In this case, we may choose the first m−1 rows to span a space of dimension r in $$\operatorname {mat}_{q}((m- 1)\times n, S, r) - \operatorname {mat}_{q}((m - 1)\times(n - 1), S, r)$$ ways and (by Proposition 3.5) choose the last row in qr−1 ways, or we may choose the first m−1 rows to span a space of dimension r−1 in $$\operatorname {mat}_{q}((m - 1) \times n, S, r - 1) - \operatorname {mat}_{q}((m - 1)\times(n - 1), S, r - 1)$$ ways and choose the last row in qn−1qr−2 ways.

The total contribution from these subcases is
Case 3:
The set S contains two elements in the mth row, without loss of generality the elements (m,n−1) and (m,n). We have three cases depending on the dimension of the space spanned by the (n−1)th and nth columns.
1. (a)

We count matrices in which the last two columns span a space of dimension 0, i.e., both columns are all zero. In this case, we may choose the first m−1 rows of the matrix to span a space of dimension r in $$\operatorname {mat}_{q}((m - 1) \times (n - 2), S, r)$$ ways and choose the last row to lie in this space in qr ways, or we may choose the first m−1 rows to span a space of dimension r−1 in $$\operatorname {mat}_{q}((m - 1) \times (n - 2), S, r - 1)$$ ways and choose the last row in qn−2qr−1 ways.

2. (b)
We count matrices in which the last two columns span a space of dimension 1. There are three possible ways this can come about: the last column may be all zero and the next-to-last column nonzero, the next-to-last column may be all zero and the last column nonzero, or both columns may be nonzero but parallel. Let S1 be the set that results if we remove the last column from S, let S1 be the set that results if we remove the next-to-last column from S and replace each element (i,n) in S with (i,n−1), and let S2 be the set that results if we remove the last two columns from S and add a new entry (i,n−1) whenever either (i,n−1) or (i,n) appeared in S. With this notation, the matrices we desire to count fall into the following six classes:
1. (i)

We may choose the first m−1 rows so that they span a space of dimension r, the last column is zero and the next-to-last column is nonzero in $$\operatorname {mat}_{q}((m - 1) \times (n - 1), S, r) - \operatorname {mat}_{q}((m - 1) \times (n - 2), S, r)$$ ways, and by Proposition 3.5 we may extend each of these to an m×n matrix of rank r in qr−1 ways.

2. (ii)

We may choose the first m−1 rows so that they span a space of dimension r−1, the last column is zero and the next-to-last column is nonzero in $$\operatorname {mat}_{q}((m - 1) \times (n - 1), S, r - 1) - \operatorname {mat}_{q}((m - 1) \times (n - 2), S, r - 1)$$ ways, and by Proposition 3.5 we may extend each of these to a matrix of rank r in qn−2qr−2 ways.

3. (iii)

We may choose the first m−1 rows so that they span a space of dimension r, the next-to-last column is zero and the last column is nonzero in $$\operatorname {mat}_{q}((m - 1) \times (n - 1), S_{1}, r) - \operatorname {mat}_{q}((m - 1) \times (n - 2), S_{1}, r)$$ ways, and by Proposition 3.5 we may extend each of these to a matrix of rank r in qr−1 ways.

4. (iv)

We may choose the first m−1 rows so that they span a space of dimension r−1, the next-to-last column is zero and the last column is nonzero in $$\operatorname {mat}_{q}((m - 1) \times (n - 1), S_{1}, r - 1) - \operatorname {mat}_{q}((m - 1) \times (n - 2), S_{1}, r - 1)$$ ways, and by Proposition 3.5 we may extend each of these to a matrix of rank r in qn−2qr−2 ways.

5. (v)

We may choose the first m−1 rows so that they span a space of dimension r and the last two columns are nonzero and parallel in $$(q - 1)(\operatorname {mat}_{q}((m - 1) \times (n - 1), S_{2}, r) - \operatorname {mat}_{q}((m - 1) \times (n - 2), S_{2}, r))$$ ways, and by Proposition 3.5 we may extend each of these to a matrix of rank r in qr−1 ways.

6. (vi)

We may choose the first m−1 rows so that they span a space of dimension r and the last two columns are nonzero and parallel in $$(q - 1)(\operatorname {mat}_{q}((m - 1) \times (n - 1), S_{2}, r - 1) - \operatorname {mat}_{q}((m - 1) \times (n - 2), S_{2}, r - 1))$$ ways, and by Proposition 3.5 we may extend each of these to a matrix of rank r in qn−2qr−2 ways.

3. (c)

We count matrices in which the last two columns span a space of dimension 2. To compute the number of these in which the first m−1 rows span a space of dimension r, subtract from $$\operatorname {mat}_{q}((m - 1) \times n, S, r)$$ the number of matrices in which the last two columns span a space of dimension less than 2; this number is computed in cases (a), (b)i, (b)iii, and (b)v above. By Proposition 3.5, we may extend each of these to a matrix of rank r in qr−2 ways. Alternatively, the first m−1 rows may span a space of dimension r−1, and the number of ways in which this happens is the result of subtracting the appropriate values computed in cases (a), (b)ii, (b)iv, and (b)vi above. By Proposition 3.5, we may extend each of these to a matrix of rank r in qn−2qr−3 ways.

As before, every instance of the function $$\operatorname {mat}_{q}$$ in each subcase is applied on matrices of size strictly smaller than m×n. (We omit the large, uninformative expression that is the total contribution from the cases above.)

Finally, it is easy to check that these cases are exhaustive and that each yields an application of $$\operatorname {mat}_{q}$$ on matrices of size smaller than m×n, and that they are combined with coefficients that are polynomials in q, as desired. □

#### 3.2.2 Reductions of this sort cannot work if S has three entries per row

On first glance, it appears that the method of proof of Theorem 3.6 can be extended to show that the function $$\operatorname {mat}_{q}(m \times n, S, r)$$ is a polynomial in q for any choice of m, n, S, and r. However, as the example of Stembridge [18] (see Fig. 1) shows, this is not the case. In this section, we briefly explain why this recursive approach breaks down for matrices with three or more zeroes per row.

Suppose that we attempt to recurse as in the proof of Theorem 3.6 while removing a row with three forced zero entries, e.g., for the set S such that is a representative matrix counted in $$\operatorname {mat}_{q}(6,S,r)$$. For instance, let us remove the bottom row. Then we wish to count the number of matrices of the form of each rank, refined by the dimension of the space spanned by the first three columns. As in Case 3(b) of the proof of Theorem 3.6, we further refine by the geometric relationship between these columns; it is straightforward to check that if these columns span a space of dimension 0 or 1, or if two of them are linearly dependent, then the induction goes through. The only remaining case is that they span a space of dimension 2 and are pairwise linearly independent.1 In this case, we may take an appropriate linear combination of the second and third columns to eliminate the first column, in which case our problem appears at first glance to reduce to the problem of counting matrices of the form in which the first two columns are linearly independent; we have already shown (in Case 3(c) of the proof of Theorem 3.6) that under an appropriate inductive hypothesis this yields a polynomial answer in q. The “catch” is that the apparently valid reduction is actually wrong: in our particular example, the zeroes in positions (1,1) and (2,1) of M′, together with the elimination step, impose the condition that the upper-left 2×2 minor of M″ is 0. This condition cannot be reduced to the requirement that certain entries be equal to 0, so any inductive approach of this nature would have to take a substantially stronger inductive hypothesis that allows us to consider restrictions on larger minors. While this more general question seems potentially interesting, it is quite broad, and it seems likely that polynomiality will only hold under very restrictive conditions on the set of selected minors.

#### 3.2.3 Reduction when $$\overline{S}$$ has at most two entries in a row

In this section, we show the complementary result of Theorem 3.6 and give a reduction for $$\operatorname {mat}_{q}(m \times n, S, r)$$ when $$\overline{S}$$ contains at most two entries in some row.

### Theorem 3.8

Suppose thatS⊆[m]×[n] and that$$\overline{S}$$contains at most two entries in themth row. Then$$\operatorname {mat}_{q}(m \times n, S, r)$$is equal to a linear combination of similar expressions for smaller matrices with coefficients inZ[q].

### Proof

As before, we split into cases depending on the number of entries of $$\overline{S}$$ in the mth row.
Case 1.

The set $$\overline{S}$$ contains no entries in the mth row. In this case the mth row is all zero, and we can simply remove it, so $$\operatorname {mat}_{q}(m \times n, S, r) = \operatorname {mat}_{q}((m - 1) \times n, S, r)$$.

Case 2.
The set $$\overline{S}$$ contains one entry in the mth row, without loss of generality the entry (m,n). In this case, the matrices of rank r with zeroes at the positions marked by S fall into two categories: those for which the (m,n) entry is also 0, of which there are $$\operatorname {mat}_{q}((m - 1) \times n, S, r)$$, and those for which the (m,n) entry is nonzero. In the latter case we may row-reduce, using the mth row to eliminate the nth column, and we find that there are $$(q - 1)q^{a} \operatorname {mat}_{q}((m - 1) \times (n- 1), S, r - 1)$$ such matrices, where a is the number of entries of $$\overline{S}$$ among {(1,n),…,(m−1,n)}. Thus, in total we have
$$\operatorname {mat}_q\bigl((m - 1) \times n, S, r\bigr) + (q - 1)q^a \operatorname {mat}_q\bigl((m - 1) \times (n- 1), S, r - 1\bigr)$$
matrices in this case.
Case 3.
Suppose that the last row of $$\overline{S}$$ contains two entries in the mth row, without loss of generality the entries (m,n−1) and (m,n). We count matrices M of rank r whose entries at positions marked by S are equal to 0, refining by whether the entries (m,n−1) and (m,n) are also equal to 0.
1. (a)

If both entries are zero, the number of matrices is just $$\operatorname {mat}_{q}((m - 1) \times n, S, r)$$.

2. (b)

If the (m,n) entry is nonzero and the (m,n−1) entry is zero, then we are essentially in Case 2 above: the entry (m,n) may be chosen in (q−1) ways, and we may use the mth row to eliminate the nth column, which gives a factor of qb, where b is the number of entries of $$\overline{S}$$ among {(1,n),…,(m−1,n)}. The rest of the matrix may be filled in $$\operatorname {mat}_{q}((m - 1) \times (n - 1), S, r - 1)$$ ways. So in total we have a contribution of $$(q - 1)q^{b} \operatorname {mat}_{q}((m - 1) \times (n - 1), S, r - 1)$$ in this case.

3. (c)

Similarly, if the (m,n−1) entry is nonzero and the (m,n) entry is zero, we may apply the same technique to eliminate the (n−1)th column, etc. If we set c to be the number of entries of $$\overline{S}$$ in {(1,n−1),…,(m−1,n−1)} and let S′ be the result of removing the (n−1)th column from S and shifting every entry in the nth column left, then we have in this case a contribution of $$(q - 1)q^{c} \operatorname {mat}_{q}((m - 1) \times (n - 1), S', r - 1)$$ matrices.

4. (d)

Finally, if both the (m,n−1) and (m,n) entries are nonzero (which may happen in (q−1)2 ways), we use (m,n) entry to kill nonzero entries (i.e., entries of $$\overline{S}$$) in the nth column. In this case, nonzero entries get “transferred” from the nth column to the (n−1)th; we pick up an extra factor of q every time there is already an entry of $$\overline{S}$$ in same row in the (n−1)th column. Let S″ be the set that we get by removing the (n−1)th and nth columns from S and adding a new column that has an entry in row i whenever {(i,n−1),(i,n)}⊆S, and let d be the number of i∈[m−1] such that neither (i,n−1) nor (i,n) is in S. In this case, we have a total contribution of $$(q - 1)^{2} q^{d} \operatorname {mat}_{q}((m - 1) \times(n - 1), S'', r - 1)$$ matrices.

The total number of matrices in this case is the sum of the expressions in the four subcases.
Each of the expressions appearing above yields a linear combination of applications of $$\operatorname {mat}_{q}$$ on matrices of size smaller than m×n, and the coefficients are polynomials in q, as desired. □

This result cannot be extended to the case of three nonzero entries in some row, again by the example of [18] (see Fig. 1); attempts to follow the same method of proof as in Theorem 3.8 meet an obstruction similar to that of Sect. 3.2.2.

### Corollary 3.9

Ifm, n, andSare chosen minimal (in the sense of row- or column-removal) so that$$\operatorname {mat}_{q}(m \times n, S, r)$$is not polynomial inq, then$$\overline{S}$$contains at least three entries in each row and column.

Theorems 3.6 and 3.8 allow for the efficient recursive computation of $$\operatorname {mat}_{q}(m \times n, S, r)$$ for a large class of sets S when m and n are small (though characterizing precisely which sets S seems hard). The code implemented in [9] includes these recursions. Unfortunately, for moderately large m and n, most sets S⊆[m]×[n] do not admit reductions by our theorems.

## 4 Formula for $$\mathrm{mat}_{q}(n,\overline{B},r)$$ when B has NE Property

In [7], Haglund proved the following result.

### Theorem 4.1

[7, Theorem 1]

For every straight shapeSλ, we have
$$\operatorname {mat}_q(n,\overline{S_{\lambda}},r) = (q-1)^r q^{|\lambda|-r}R_r^\textsf{(SE)}\bigl(S_{\lambda},q^{-1} \bigr).$$

We now extend this result (using the same proof technique) to all shapes with the NE Property, that is, with the property that for any i′<i, j<j′, if (i,j), (i′,j), and (i,j′) belong to B, then (i′,j′) does as well.

### Theorem 4.2

Fix anynandrand any setB⊆[n]×[n] with theNEProperty. The number ofn×nmatrices overFqof rankrwhose support is contained inBis
$$\operatorname {mat}_q(n,\overline{B},r) = (q-1)^r q^{\#B-r}R_r^\textsf{(NE)}\bigl(B,q^{-1}\bigr).$$
(4.3)

### Proof

Choose a matrix A counted in $$\operatorname {mat}_{q}(n, \overline{B},r)$$, that is, whose support is in B, and perform Gaussian elimination in the following (north–east) order: traverse each column from bottom to top, starting with the leftmost (i.e., first) column. When you come to a nonzero entry (i.e., a pivot), use it to eliminate the entries to its north in the same column and to its east in the same row. See Fig. 3 for an example of this stage of the elimination process. Then move on to the next column and repeat until there is at most one nonzero entry in every row and column.

By the NE Property, at each stage of the elimination process just described, we obtain another matrix counted in $$\operatorname {mat}_{q}(n, \overline{B},r)$$. After elimination, the positions of the pivots are a placement of r nonattacking rooks on B.

Given a fixed placement of r nonattacking rooks on B, let a be the number of cells in B that are directly north or directly east of a rook. There are (q−1)rqa matrices of rank r whose support is in B that give this placement after the elimination procedure described above. It is not hard to see that $$a=\#B-r-\operatorname {inv}_{\textsf{NE}}(C,B)$$. Thus, summing over all placements or r nonattacking rooks, we obtain
$$\operatorname {mat}_q(n,\overline{B},r)=(q-1)^rq^{\#B-r}\sum _C\bigl(q^{-1}\bigr)^{\operatorname {inv}_\textsf{NE}(C,B)} =(q-1)^rq^{\#B-r}R_r^{\sf(NE)} \bigl(B,q^{-1}\bigr),$$
as desired. □

Note that a priori it is not clear that the expression on the right-hand side of Eq. (4.3) is a polynomial. However, this expression is a polynomial for the following reason: for any rook placement, there cannot be any more inversions than there are empty cells without rooks in them. There are #B cells unoccupied by zeros, and, of these, r have rooks in them. So the maximum value of $$\operatorname {inv}_{\textsf{NE}}(C,B)$$ is #Br. Since this is the power of q at the beginning of the formula, there will not be any q−1 terms, and $$\operatorname {mat}_{q}(n,\overline{B},r)$$ is a polynomial.

### Example 4.4

For n=4 and r=4, the set B=([4]×[4])∖{(1,1),(3,4),(4,1),(4,3),(4,4)} has the NE Property (as in Fig. 4(a)), and there are three placements of four rooks on B (as in Fig. 4(b)). The number of NE-inversions of these placements are 0, 1, and 1, respectively. Thus,

We give two corollaries of Theorem 4.2. First, since a straight shape Sλ has the NE Property, by comparing Haglund’s result and Theorem 4.2 we see that the (NE) and (SE)q-rook numbers of Sλ agree.

### Corollary 4.5

For any straight shapeSλ, we have
$$R_r^\textsf{(NE)}(S_{\lambda},q)= R_r^\textsf{(SE)}(S_{\lambda},q).$$

(Recall that in general the (NE) and (SE)q-rook numbers of a general board do not agree; see for example Remark 2.3.)

Second, since any skew shape Sλ/μ has the NE Property, we have the following corollary:

### Corollary 4.6

For any skew shapeSλ/μ,
$$\operatorname {mat}_q(n,\overline{S_{\lambda/\mu}},r)=(q-1)^r f(q),$$
wheref(q) is a polynomial with nonnegative integer coefficients.

### Example 4.7

For λ/μ=4432/31, we have In general, for skew shapes Sλ/μ, there is no product formula for $$\operatorname {mat}_{q}(n,\overline{S_{\lambda/\mu}},r)$$ analogous to (2.2), even when r=n.

## 5 Studying matq(n,S,r) when S is a Rothe diagram

Given a permutation $$w\in \mathfrak{S}_{n}$$ written as a word w=w1w2wn where wi is the image of w at i, the Rothe diagramRw is the set
$$R_w = \bigl\{(i,j) \mid 1\leq i, j\leq n,\,\, w(i)>j,\,\, w^{-1}(j)>i\bigr\}.$$
Equivalently, Rw is the set of elements in [n]×[n] that do not lie directly south or directly east of entries (i,wi) of the permutation matrix of w. See Fig. 5 for some examples of Rothe diagrams. Note that #Rw is the number of inversions of w, that is, the number of pairs (i,j) such that i<j but wi>wj. Also, Rw has the following property: if (i,j) and (k,) are in Rw and i>k, j<, then the entry (k,j) is also in Rw. We call this the Le property of Rothe diagrams.2

The main conjecture for Rothe diagrams, which has been verified for n≤7 using the results in Sect. 3 [9], is the following:

### Conjecture 5.1

IfRwis the Rothe diagram of a permutationwin$$\mathfrak{S}_{n}$$and 0≤rnthen$$\operatorname {mat}_{q}(n,R_{w},r)/(q-1)^{r}$$is a polynomial inqwith nonnegative integer coefficients.

In Sect. 5.1 we give properties of Rothe diagrams that help in calculating $$\operatorname {mat}_{q}(n,R_{w},r)$$. In Sect. 5.2 we study Conjecture 5.1 for the family of permutations w such that Rw is the complement of a skew shape (after permuting rows and columns). The conjecture holds for such permutations by Corollary 4.6. In Theorem 5.4, we characterize these permutations.

### 5.1 Properties of matq(n,S,r) when S is a Rothe diagram

In this section we give some simple properties of $$\operatorname {mat}_{q}(n,S,r)$$ when S=Rw is the Rothe diagram of a permutation w. These properties are useful to simplify the size of computations involved in empirically confirming conjectures about $$\operatorname {mat}_{q}(n,R_{w},r)$$ like Conjectures 5.1 and 6.6.

If the permutation w is the word w1w2wn, the reverse of w is the permutation re(w)=wnwn−1w1. The complement of w is the permutation c(w)=u1u2un where ui=n+1−iwi. In addition, the reverse complement of w is the permutation rc(w)=v1v2vn where vi=n+1−wn+1−i. Lastly, the left-to-right maxima of w are the values wi such that wi>wj for all j such that 1≤j<i.

### Proposition 5.2

Given a permutationwin$$\mathfrak{S}_{n}$$and its Rothe diagramRw, we have
1. (i)

$$\operatorname {mat}_{q}(n,R_{w},r)=\operatorname {mat}_{q}(n,R_{w^{-1}},r)$$and

2. (ii)

$$\operatorname {mat}_{q}(n, R_{w},r) = \operatorname {mat}_{q}(n, R_{rc(w)},r)$$.

### Proof

It is easy to see that for any permutation w, the diagram $$R_{w^{-1}}$$ is the transpose of Rw, and the first statement follows immediately. We now consider the second statement.

Fix a permutation w with Rothe diagram Rw. Each element (i,j) of Rw corresponds to the inversion of w formed by the entries with matrix coordinates (i,wi) and $$(w^{-1}_{j},j)$$. In rc(w), these elements of w are transformed to (n+1−i,n+1−wi) and $$(n + 1 - w^{-1}_{j}, n + 1 - j)$$ and still form an inversion; in Rrc(w), this inversion corresponds to the element with coordinates $$(n+1-w^{-1}_{j},n+1-w_{i})$$. It follows immediately that the diagram Rrc(w) is the result of taking the transpose of Rw, rearranging rows and columns by multiplying on both sides by the permutation matrix of w, and rotating the result by 180. □

Next we characterize the indices of the rows and columns of [n]×[n] entirely contained in $$\overline{R_{w}}$$. This is useful for computation because it is easy to express $$\operatorname {mat}_{q}(n,S,r)$$ in terms of values of $$\operatorname {mat}_{q}$$ for sets obtained by removing rows or columns that contain no elements of S.

### Proposition 5.3

Thekth column (row) of [n]×[n] is contained in$$\overline{R_{w}}$$if and only ifkis a left-to-right maximum ofw (ofw−1).

### Proof

This follows from the definitions of Rw and of the left-to-right maxima. □

### 5.2 Skew-vexillary permutations

A permutation w is vexillary if its Rothe diagram, up to a permutation of its rows and columns, is the diagram of a partition. Call this partition λ(w). Then by Haglund’s Theorem 4.1, for vexillary permutations w, we have that
$$\operatorname {mat}_q(n, R_w,r) = \operatorname {mat}_q(n,S_{\lambda(w)},r)=(q-1)^rq^{n^2-\operatorname {inv}(w)-r}R_r^\textsf{(NE)} \bigl(\,\overline{S_{\lambda(w)}},q^{-1}\bigr).$$
It is well known that w is vexillary if and only if w avoids 2143 [10], i.e., there is no sequence 1≤i<j<k<ln in w such that wj<wi<wl<wk.

Next we give a characterization of permutations whose Rothe diagram, up to a permutation of rows and columns, is the complement of a skew shape. For such a permutation w, we have by Corollary 4.6 that $$\operatorname {mat}_{q}(n,R_{w},r)/(q-1)^{r}$$ is a polynomial with nonnegative integer coefficients. So Conjecture 5.1 holds for these permutations.

For the proof, we need the following definition: we say that a skew shape Sλ/μ in [n]×[n] is nonoverlapping if there is no row nor column that contains entries from both Sμ and $$\overline{S_{\lambda}}$$.

### Theorem 5.4

The Rothe diagram ofw=w1w2wnis, up to permuting its rows and columns, the complement of a skew shape if and only ifwcan be decomposed asa1a2akb1b2bnkwhereai<bjand each ofa1a2akandb1b2bnkis 2143-avoiding.

### Proof

First we prove the “if” direction. This argument is illustrated in Fig. 6. Suppose that w can be decomposed into a=a1a2ak and b=b1b2bnk as in the theorem statement. Then the Rothe diagram Rw is block-diagonal, i.e., it consists of some entries in the upper-left k×k block and some in the lower-right (nk)×(nk) block, with no entries in the upper-right k×(nk) block or lower-left (nkk block. Furthermore, note that the upper-left and lower-right subdiagrams are identical to the Rothe diagrams of the permutations order-isomorphic to a1a2ak and b1b2bnk, respectively.

Since both of these permutations are 2143-avoiding, and their Rothe diagrams in the upper-left and lower-right corners do not share any rows or columns in common, they can be rearranged independently to form two separate straight shapes. We may then rotate the straight shape corresponding to Rb by 180 via permuting rows and columns (without changing the rearranged upper-left corner) to get a straight shape in the upper-left corner and an upside-down straight shape in the lower-right. This is the outside of a skew shape, as desired.

Second, we prove the “only if” direction of the theorem. Suppose that the diagram Rw, when rearranged, forms the complement of a skew shape Sλ/μ. This skew shape contains the column that was previously (i.e., before rearrangement) given by {(j,w1)∣j≥1}. Likewise, it contains the row that was previously given by $$\{(w^{-1}_{1},j) \mid j \geq 1\}$$. It follows that the skew shape Sλ/μ is nonoverlapping. After rearrangement, every entry of Rw either belongs to Sμ or $$\overline{S_{\lambda}}$$. We use this partition of the elements of Rw to identify the appropriate decomposition of w.

We color an entry of Rw blue if it belongs to Sμ after rearrangement, otherwise we color it red. We show the following claim: for every entry wi of w, the elements of Rw in the same row or column as (i,wi) are either all blue or all red.

Since Sλ/μ is nonoverlapping, the entries of Rw in each row have the same color, and likewise for columns. If there is an entry (i,wi) with elements (i,j) and (k,wi) of Rw, then by the Le property of Rothe diagrams, (k,j) is also in Rw. Therefore all three entries have the same color, and the claim follows.

By the argument of the preceding paragraph, we may color the elements of w as follows: if (i,wi) is in the same row or column as a red entry of Rw, then we color wi red, whereas if (i,wi) is in the same row or column as a blue entry of Rw, then we color wi blue, and otherwise we leave wi uncolored. We observe a few properties of the colored and uncolored elements of the permutation. First, inversions of w can only happen between elements of the same color. Second, wi is uncolored if and only if wi is not involved in any inversions. And third, the subword of the blue (respectively, red) elements of w is 2143-avoiding. This is because by definition, the entries of Rw in the same row or column as (i,wi) for blue (respectively, red) wi are exactly the entries in Sμ (respectively, $$\overline{S_{\lambda}}$$) after rearrangement. This is equivalent to saying that the subword of the blue (respectively, red) elements of w is vexillary and thus 2143-avoiding.

From the three observations above it follows that the permutation w decomposes as u1c1u2c2u3 where (i) the ui are (possibly empty) blocks of uncolored elements, c1 is the block of elements of one color of w, and c2 is the block of elements of the other color of w; (ii) the entries of each block are smaller than the entries of the following blocks, and (iii) the blocks c1 and c2 are 2143-avoiding. Finally, if we set a1a2ak=u1c1 and b1b2bnk=u2c2u3, we get a desired decomposition of w where ai<bj and a1a2ak and b1b2bnk are 2143-avoiding. □

We call the above permutations skew-vexillary,3 and we denote by λ/μ(w) the skew shape whose complement is the rearrangement of Rw.4

### Corollary 5.5

By Theorem 4.2, ifwis skew-vexillary, then$$\operatorname {mat}_{q}(n, R_{w}, r)/(q-1)^{r}$$is equal to$$q^{n^{2}-\operatorname {inv}(w)-r}R_{r}^{\textsf{(NE)}}(S_{\lambda/\mu(w)},q^{-1})$$, a polynomial with nonnegative integer coefficients. In particular, Conjecture 5.1 holds for skew-vexillary permutations.

If w is a skew-vexillary permutation, then every subpermutation of w is as well. This implies that skew-vexillarity may be rephrased as a pattern-avoidance condition. We do this now.

### Proposition 5.6

The permutation$$w \in \mathfrak{S}_{n}$$can be decomposed asw=a1akb1bnksuch thatai<bjfor all iand jand the permutationsa1akandb1bnkavoid 2143 if and only ifwavoids the nine patterns 24153, 25143, 31524, 31542, 32514, 32541, 42153, 52143, and 214365.

### Proof

Call the decomposition in question an “SV-decomposition” (for Skew-Vexillary). First, we show that if w contains any of the nine patterns listed in the statement of the theorem, it does not have an SV-decomposition.

Let p be any of the eight patterns of length 5; it is easy to check that p is indecomposable, i.e., we cannot write p=uv with u, v nonempty and ui<vj for all ij. Thus, if we write w=ab with ai<bj, we must have either p contained in a or p contained in b. Since p contains 2143, it follows that either a or b contains 2143, so this decomposition is not SV, as desired.

Now consider the case of the pattern 214365. Any decomposition of w decomposes 214365, and it is easy to see that in any of the four decompositions of 214365, one piece or the other contains a copy of 2143. This completes the proof that any permutation containing the given patterns has no SV-decomposition.

Now consider the converse. Suppose that w is not SV-decomposable. There are two cases.

If w is indecomposable and contains 2143, then w contains a minimal indecomposable permutation that contains 2143. The minimal 2143-containing indecomposable permutations are precisely the eight permutations of length 5 that we consider.

Finally, we show by induction that every decomposable but not SV-decomposable permutation contains one of the nine patterns. Choose such w and write w=ab with ai<bj. Without loss of generality, in this decomposition we have that a contains 2143. If b has a descent, it follows immediately that w contains 214365. Otherwise, w=a1ak(k+1)(k+2)⋯n. Observe that a permutation of this form has an SV-decomposition if and only if the shorter permutation a=a1ak has an SV-decomposition; thus, a has no SV-decomposition. If a is indecomposable, we have by the preceding paragraph that a contains one of the nine patterns; if a is decomposable, we have the same result by induction.

Putting the two cases together, every permutation that is not SV-decomposable contains at least one of the nine patterns, as desired. □

### Remark 5.7

Vexillary permutations have many other interesting properties (see, for example, [12, Sects. 2.6.5 and 2.8.1]). For example, the Stanley symmetric function [16, Sect. 2] Fw of a vexillary permutation w equals the Schur function sλ(w). Since a skew-vexillary permutation w with skew shape λ/μ(w) is SV-decomposable, Fw is the product s[n]×[n]∖λsμ of Schur functions. Do other properties of vexillary permutations carry over to skew-vexillary permutations?

#### 5.2.1 An enumerative aside

Any structurally interesting class of permutations calls out to be enumerated. Vexillary permutations were enumerated by West [20] (who showed that they are in bijection with 1234-avoiding permutations, which had been enumerated earlier by Gessel [6]). We now enumerate skew-vexillary permutations in terms of the generating function for vexillary permutations. For convenience, we denote by $$\mathfrak{S}_{n}(2143)$$ the set of vexillary permutations of length n. Also, given two permutations $$u \in \mathfrak{S}_{i}$$ and $$v \in \mathfrak{S}_{j}$$, we denote by w=uv the permutation in $$\mathfrak{S}_{i + j}$$ defined by wt=ut for t∈[i] and wt+i=i+vt for t∈[j].

### Theorem 5.8

LetV(x) be the ordinary generating function for 2143-avoiding permutations, and letSV(x) be the ordinary generating function for skew-vexillary permutations. We have
$$SV(x) = (1 - x)V(x)^2 - V(x) + \frac{1}{1 - x}.$$

### Proof

Let V(x)=1+x+2x2+6x3+23x4+⋯ be the ordinary generating function for 2143-avoiding permutations, and let I(x)=x2+3x3+13x4+⋯ be the generating function for indecomposable 2143-avoiding permutations of length 2 or more. Suppose that $$w \in \mathfrak{S}_{n}(2143)$$ can be written w=u1u2 where u1 and u2 are nonempty permutations. If both u1 and u2 contain an inversion, the four entries from the two inversions form an instance of 2143 in w, a contradiction. Thus, one of u1 and u2 must be the identity permutation. It follows immediately that every $$w \in \mathfrak{S}_{n}(2143)$$ other than the identity can be written as w=idiu⊕idk for identity permutations idi and idk (possibly of length 0) and an indecomposable 2143-avoiding permutation u. Thus,
$$V(x) = \frac{1}{1 - x} + \frac{I(x)}{(1 - x)^2},$$
and so
$$I(x) = (1 - x)^2 V(x) + x - 1.$$
Now suppose that $$w \in \mathfrak{S}_{n}$$ has the property that w=uv where u and v are 2143-avoiding permutations (possibly of length 0). It follows from the preceding analysis that either w is 2143-avoiding itself or we can write w=idiu′⊕idjv′⊕idk where the ids are identity permutations (possibly of length 0) and u′ and v′ are indecomposable 2143-avoiding permutations of length 2 or more. Thus, the generating function SV(x) for such permutations is given by as desired. □

If w is skew-vexillary, then $$\operatorname {mat}_{q}(n,R_{w},r)/(q-1)^{r}$$ is of the form $$q^{n^{2}-\operatorname {inv}(w)-r}\*\sum_{\text{some } u \in \mathfrak{S}_{n}} q^{-\operatorname {inv}(u)}$$ by Corollary 5.5. Another polynomial with this form is the Poincaré polynomial of the strong Bruhat order (see, e.g., [12, Sect. 2.1.2]) in $$\mathfrak{S}_{n}$$. In the next subsection we study the connections between these and $$\operatorname {mat}_{q}(n,R_{w},n)$$.

## 6 Poincaré polynomials, matq(n,Rw,n), and q-rook numbers

A natural question when faced with a family of polynomials with positive integer coefficients is whether they count some nice combinatorial object. In this section, we investigate connections between our polynomials $$\operatorname {mat}_{q}(n, R_{w}, n)$$ (note in particular that we focus on the case of full rank) and certain well-known polynomials we define now.

As before, let $$\operatorname {inv}(w)$$ denote the number of inversions #{(i,j)∣i<j,wi>wj} of w. Recall the notion of the strong Bruhat order ≺ on the symmetric group [3, Chap. 2]: if tij is the transposition that switches i and j, we have as our basic relations that uutij in the strong Bruhat order when $$\operatorname {inv}(u) + 1 = \operatorname {inv}(u \cdot t_{ij})$$, and we extend by transitivity. Let $$P_{w}(q)=\sum_{u \succeq w} q^{\operatorname {inv}(u)}$$ be the (upper) Poincaré polynomial of w, where we sum over all permutations u that succeed w in the strong Bruhat order. Equivalently, Pw(q) is the rank-generating function of the interval [w,w0] in the strong Bruhat order where w0 is the largest element nn−1⋯21 of this order.

### Example 6.1

If w=3412, then the permutations in $$\mathfrak{S}_{4}$$ that succeed w in the Bruhat order are 3412,3421,4312, and 4321. The generating polynomial for this set by number of inversions is P2143(q)=q6+2q5+q4.

In [15], Sjöstrand gave necessary and sufficient conditions for Pw(q) to be equal to a q-rook number of a skew shape associated to w. Namely, the left hullHL(w) of w is the smallest skew shape that covers w. Equivalently, HL(w) is the union over noninversions (i,j) of w of the rectangles {(k,)∣wikwj,ij}. See Fig. 7 for an example of the left hull of a permutation.

The following special case of a result by Sjöstrand characterizes when Pw(q) is equal to the rook polynomial of the left hull of the permutation w.

### Theorem 6.2

[15, Corollary 3.3]

The Bruhat interval [w,w0] in$$\mathfrak{S}_{n}$$equals the set of rook placements in the left hullHL(w) ofw (and in particular$$q^{|\mu|}R_{n}^{\textsf{(SE)}}(H_{L}(w),q)=P_{w}(q)$$whereμis the shape such thatHL(w)=Sλ/μfor someλ) if and only ifwavoids the patterns 1324,24153,31524, and 426153.

If w is a skew-vexillary, then by Corollary 5.5 we know that $$\operatorname {mat}_{q}(n,R_{w},n)/ (q-1)^{n}$$ is (up to a power of q) a q-rook number. Next we show that this q-rook number is essentially a q-rook number of the left hull of a permutation v that avoids the four patterns above. Therefore, by Theorem 6.2 $$\operatorname {mat}(n,R_{w},n)/(q-1)^{n}$$ is (up to a power of q) a Poincaré polynomial Pv(q).

### 6.1 matq(n,Rw,n) for skew-vexillary permutations is a Poincaré polynomial

In this section we use Sjöstrand’s result (Theorem 6.2) to show that for skew-vexillary permutations w, the function $$\operatorname {mat}_{q}(n,R_{w},n)/(q-1)^{n}$$ is not only a polynomial with nonnegative coefficients but, up to a power of q, is a Poincaré polynomial.

### Proposition 6.3

Ifwis skew-vexillary, then
$$\operatorname {mat}_q(n, R_w, n) = q^{\binom{n}{2}-\operatorname {inv}(w)}(q-1)^n \cdot P_v(q)$$
for some permutation$$v \in \mathfrak{S}_{n}$$.

### Proof

If w is skew-vexillary, then by Corollary 5.5 we know that
$$\operatorname {mat}_q(n, R_w,n)/(q-1)^n = q^{n^2 - \operatorname {inv}(w) - n}R_n^\textsf{(NE)}\bigl(S_{\lambda/\mu(w)},q^{-1} \bigr),$$
where R(NE)(Sλ/μ(w),q) is the rook polynomial of Sλ/μ(w), the nonoverlapping skew shape whose complement is the rearrangement of Rw. We will show that this polynomial is the Poincaré polynomial Pv(q) of a permutation v.
Define the permutation matrix of v as follows: let w=a1a2akb1b2bnk be the decomposition promised by Theorem 5.4. Let λ=(λ1,…,λn) and μ=(μ1,…). For i=1,…,k, let vi=min(([n]∖[μi])∖{v1,…,vi−1}), and for j=1,…,nk, let vn+1−j=max([λnj]∖{vnj+1,…,vn}). This defines a 0–1 matrix with exactly one 1 in every row; it follows from the proof of Theorem 5.4 that this matrix is in fact a permutation matrix (with {v1,…,vk}=[k] and {vk+1,…,vn}={nk+1,…,n}). See Fig. 8 for an example of this construction. It is clear that Sλ/μ(w)=HL(v). Also by Proposition 2.4 we have that $$q^{\binom{n}{2}-|\mu|}R_{n}^{\textsf{(NE)}}(S_{\lambda/\mu}(w),q^{-1})=R_{n}^{\textsf{(SE)}}(H_{L}(v),q)$$.
By construction, the prefix v1vk avoids 132, the suffix vk+1vn avoids 213, and vj>vi for jk+1 and ik. It is easy to see that the set of permutations with such a decomposition is closed under containment of patterns and does not contain any of the permutations 1324, 24153, 31524, and 426153. Therefore, every permutation in this set, and in particular v, avoids these four patterns. By Theorem 6.2 it follows that $$q^{|\mu|}R_{n}^{\textsf{(SE)}}(H_{L}(v),q)=P_{v}(q)$$. Thus, as desired. □

### Example 6.4

By Theorem 5.4, the permutation w=21534 is skew-vexillary. After rearranging rows and columns (see Fig. 8), the skew shape Sλ/μ(w) is S55553/1. The associated v is 21453, and we have

### Remark 6.5

Note that the result above does not hold for all Rothe diagrams. There exist permutations w for which there does not exist any permutation v such that $$\operatorname {mat}_{q}(n, R_{w},r) = q^{\binom{n}{2}-\operatorname {inv}(w)}(q-1)^{r}P_{v}(q)$$. For example, take w=31524 (see Fig. 5(iii) and Table 1). In this case,
$$\operatorname {mat}_q(5, R_{31524},5)=q^{6}(q-1)^5 \bigl(q^{10}+4q^9+9q^8+12q^7+10q^6+5q^5+q^4 \bigr).$$
One can show (either by computer search or by a direct argument about the possible structure of the inversions) that there is no permutation v in $$\mathfrak{S}_{5}$$ such that Pv(q)=q10+4q9+9q8+12q7+10q6+5q5+q4.
Table 1

For the four special patterns w of Conjecture 6.6, we give $$\operatorname {mat}_{q}(n,R_{w},n)/((q-1)^{n}q^{k})$$ where $$k=\binom{n}{2}-\operatorname {inv}(w)$$, the Poincaré polynomials Pw(q), and $$q^{a}R_{n}^{\textsf{(SE)}}(H_{L}(w),q)$$ where a is the size of the subtracted partition of the skew shape HL(w)

w

1324

24153 or 31524

426153

$$\frac{\operatorname {mat}_{q}(n,R_{w},n)}{(q-1)^{n}q^{k}}$$

q6+3 q5+5 q4+5q3+3 q2+q

q10+4q9+9q8+12q7+10q6+5q5+q4

q15+5q14+14q13+24q12+27q11+19q10+7q9+q8

Pw(q)

q6+3q5+5q4+6q3+4q2+q

q10+4q9+9q8+13q7+11q6+5q5+q4

q15+5q14+14q13+25q12+28q11+19q10+7q9+q8

$$q^{a}R_{n}^{\textsf{(SE)}}(H_{L}(w))$$

q6+3q5+5q4+6q3+5q2+3q+1

q10+4q9+9q8+13q7+12q6+7q5+2q4

q15+5q14+14q13+25q12+29q11+21q10+8q9+q8

We have shown that for skew-vexillary permutations w, $$\operatorname {mat}_{q}(n,R_{w},n)/ (q-1)^{n}$$ is equal (up to a power of q) to the Poincaré polynomial of some permutation v. Next, we consider the problem of classifying permutations w such that $$\operatorname {mat}_{q}(n,R_{w},n)/(q-1)^{n}$$ is equal (up to a power of q) to the Poincaré polynomial of the same permutation.

### 6.2 Further relationships between matq(n,Rw,n) and Poincaré polynomials

Computational evidence for n≤7 [9] suggests the following conjecture.

### Conjecture 6.6

Fix a permutationwin$$\mathfrak{S}_{n}$$and letRwbe its Rothe diagram. We have that$$\operatorname {mat}_{q}(n,R_{w},n)/(q-1)^{n}$$is coefficient-wise less than or equal to$$q^{\binom{n}{2}-\operatorname {inv}(w)}P_{w}(q)$$. We have equality if and only ifwavoids the patterns 1324, 24153, 31524, and 426153.

### Remark 6.7

The patterns that appear in Conjecture 6.6 and in Theorem 6.2 are the same. Also, the reverses 4231,35142,42513, and 351624 of these patterns have appeared in related contexts, in work of Gasharov and Reiner [5] in a conjecture of Postnikov [14] proved by Hultman, Linusson, Shareshian, and Sjöstrand [8] and by Gasharov and Reiner [5]. This suggests further interesting connections. These permutations were recently enumerated by Albert and Brignall [1].

The values of the three polynomials $$\operatorname {mat}_{q}(n,R_{w},n)/(q-1)^{n}$$, Pw(q), and $$R^{\textsf{(SE)}}_{n}(H_{L}(w),q)$$ when w is equal to the four patterns of Conjecture 6.6 are shown in Table 1. In these cases the three polynomials are all different. By Theorem 6.2 and Theorem 4.2, Conjecture 6.6 is equivalent to the following:

### Conjecture 6.8

Fix a permutationwin$$\mathfrak{S}_{n}$$, letRwbe its Rothe diagram, and let$$a_{w} = n^{2} - \#H_{L}(w) - \operatorname {inv}(w)$$. We have that$$\operatorname {mat}_{q}(n,R_{w},n)/(q-1)^{n}$$is coefficient-wise less than or equal to$$q^{a_{w}}\operatorname {mat}_{q}(n,\overline{H_{L}(w)},n)/(q-1)^{n}$$. We have equality if and only ifwavoids the patterns 1324, 24153, 31524, and 426153.

This conjecture is not true for matrices of lower rank. For example, for $$w=21\in \mathfrak{S}_{2}$$, we have $$\operatorname {mat}_{q}(2,R_{21},1)/(q-1)=2q+1$$ and $$\operatorname {mat}_{q}(2,\overline{H_{L}(21)},1)/(q-1)=2$$.

### Remark 6.9

If Conjecture 6.8 holds, then by Theorem 4.2 and [11, Proposition 5.1] it follows that whenever w avoids the four patterns, the shapes $$\overline{R_{w}}$$ and HL(w) have the same number of placements of n nonattacking rooks. This is not obvious since for such permutations, the shapes are distinct even after permuting rows and columns. Moreover, computer experiments for n≤7 [9] suggest that the converse is also true, i.e., if w contains any of the four patterns, the shapes have different numbers of rook placements. This apparent equivalence of necessary and sufficient conditions between the “q case” and the “q=1 case” does not necessarily hold in similar settings (see [13, Theorem 7] and [8, Theorem 3.4] for an example).

We end by giving a very preliminary step in proving these conjectures.

### Proposition 6.10

Ifwis a 1324-avoiding permutation, then the complement ofHL(w) has at least as many entries as the Rothe diagramRwofw.

### Proof

Let w be a 1324-avoiding permutation. We give a one-to-one map φ between the entries of the Rothe diagram Rw and the complement of the left hull HL(w).

Given an entry (i,j) in Rw, we have two possibilities: either there is or there is not an entry (k,wk) of w such that k<i and wk<j (i.e., an entry of w NW of (i,j)). Let Aw be the set of entries of Rw of the first type, and let Bw be the set of entries of the second type. If (i,j)∈Aw then define φ(i,j)=(i,j). If instead (i,j)∈Bw, then define $$\varphi(i,j)=(w^{-1}_{j},w_{i})$$. See Fig. 9 for an illustration of φ. We show that φ is well defined and injective.

Choose (i,j) in Rw. There are no entries of w above (i,j) in the same column or to its left in the same row. If in addition (i,j) is in Aw, then by the definition of the left hull the entry (i,j) is not in HL(w). In this case, $$\varphi(i,j)=(i,j)\in \overline{H_{L}(w)}$$, as desired.

On the other hand, if (i,j) is in Bw, then there is some entry (k,wk) of w with k<i and wk<j. Since w is 1324-avoiding, there can be no entry (,w) of w such that $$\ell\geq w^{-1}_{j}$$ and wwi. Thus, $$\varphi(i,j)=(w^{-1}_{j},w_{i}) \in \overline{H_{L}(w)}$$. This completes the proof that the map φ is well defined.

Finally, we show that φ is one-to-one. Since φ is defined piecewise, it is enough to show that φ is one-to-one on Aw and Bw and that φ(Aw) and φ(Bw) are disjoint. The injectivity on Aw is trivial. The injectivity on Bw follows since w is a permutation and so $$(w^{-1}_{j},w_{i})$$ uniquely defines (i,j). Moreover, $$\overline{H_{L}(w)}$$ has two components; φ(Aw) is the NW component, while φ(Bw) is contained in the SE component, and so the images are disjoint. This completes the proof that φ is one-to-one. □

## Footnotes

1. 1.

The case that they are independent will simply be the complement of all other cases, so it is nonpolynomial in a minimal example if and only if our selected case is nonpolynomial.

2. 2.

The name “Le” was invented in [14, Sect. 6] in a context where the three entries in question formed a backwards letter “L”; here we keep this terminology even though for Rw the three entries form instead the letter “Γ”.

3. 3.

Note that in the literature [2, Proposition 2.3] there is another meaning of the term “skew-vexillary permutation,” which does not seem to be related to our definition.

4. 4.

The “function” λ/μ(w) is not actually well defined most of the time since you can switch the upper-left and lower-right corners by permuting rows and columns. Luckily, nothing we use it for depends on this choice.

## Notes

### Acknowledgements

We would like to express thanks to the following individuals for their contributions: Alexander Postnikov for pointing us in the direction of Rothe diagrams; Richard Stanley and Steven V Sam for helpful suggestions and discussions; John Stembridge for making his Maple reduce package [19] publicly available; Sara Billey and Brendan Pawlowski for mentioning the example in Remark 5.7; and the anonymous referee for several helpful comments including the observation at the end of Remark 3.4. AJK is grateful to the PRIMES program at the MIT Mathematics Department, where this research was done. JBL and AHM would like to dedicate this paper to the memories of Richard Geller and Nicolás del Castillo.

Joel Brewster Lewis supported by NSF RTG grant NSF/DMS-1148634. Alejandro H. Morales supported by a CRM-ISM Postdoctoral Fellowship.

### References

1. 1.
Albert, M.H., Brignall, R.: Enumerating indices of Schubert varieties defined by inclusions (2013). arXiv:1301.3188
2. 2.
Billey, S.C., Jockusch, W., Stanley, R.P.: Some combinatorial properties of Schubert polynomials. J. Algebr. Comb. 2(4), 345–374 (1993)
3. 3.
Björner, A., Brenti, F.: Combinatorics of Coxeter Groups. Graduate Texts in Mathematics. Springer, Berlin (2005)
4. 4.
Garsia, A.M., Remmel, J.B.: q-counting rook configurations and a formula of Frobenius. J. Comb. Theory, Ser. A 41(2), 246–275 (1986)
5. 5.
Gasharov, V., Reiner, V.: Cohomology of smooth Schubert varieties in partial flag manifolds. J. Lond. Math. Soc. 66(3), 550–562 (2002)
6. 6.
Gessel, I.: Symmetric functions and P-recursiveness. J. Comb. Theory, Ser. A 53, 257–285 (1990)
7. 7.
Haglund, J.: q-rook polynomials and matrices over finite fields. Adv. Appl. Math. 20(4), 450–487 (1998)
8. 8.
Hultman, A., Linusson, S., Shareshian, J., Sjöstrand, J.: From Bruhat intervals to intersection lattices and a conjecture of Postnikov. J. Comb. Theory, Ser. A 119(3), 564–580 (2009)
9. 9.
Klein, A.J., Lewis, J.B., Morales, A.H.: Supplementary code and data (2012). http://sites.google.com/site/matrixfinitefields/
10. 10.
Lascoux, A., Schützenberger, M.-P.: Polynômes de Schubert. C. R. Math. Acad. Sci. 294(13), 447–450 (1982)
11. 11.
Lewis, J.B., Liu, R., Panova, G., Morales, A.H., Sam, S.V., Zhang, Y.X.: Matrices with restricted entries and q-analogues of permutations. J. Comb. 2(3), 355–396 (2012)
12. 12.
Manivel, L.: Symmetric Functions, Schubert Polynomials and Degeneracy Loci. SMF/AMS Texts and Monographs. Barnes & Noble, New York (2001)
13. 13.
Oh, S., Postnikov, A., Yoo, H.: Bruhat order, smooth Schubert varieties, and hyperplane arrangements. J. Comb. Theory, Ser. A 115(7), 1156–1166 (2008)
14. 14.
Postnikov, A.: Total positivity, Grassmannians, and networks (2006). arXiv:math/0609764 [math.CO]
15. 15.
Sjöstrand, J.: Bruhat intervals are rooks on skew Ferrers boards. J. Comb. Theory, Ser. A 114(7), 1182–1198 (2007)
16. 16.
Stanley, R.P.: On the number of reduced decompositions of elements of Coxeter groups. Eur. J. Comb. 5(4), 359–372 (1984)
17. 17.
Stanley, R.P.: Enumerative Combinatorics, vol. 1, 2nd edn. Cambridge University Press, Cambridge (2012). http://www-math.mit.edu/~rstan/ec/ec1.pdf
18. 18.
Stembridge, J.R.: Counting points on varieties over finite fields related to a conjecture of Kontsevich. Ann. Comb. 2(4), 365–385 (1998)
19. 19.
Stembridge, J.R.: Reduce package (2009). http://www.math.lsa.umich.edu/~jrs/data/reduce/
20. 20.
West, J.: Generating trees and the Catalan and Schröder numbers. Discrete Math. 146, 247–262 (1995)

## Authors and Affiliations

• Aaron J. Klein
• 1
• Joel Brewster Lewis
• 2
• Alejandro H. Morales
• 3
1. 1.Brookline High SchoolBrooklineUSA
2. 2.School of MathematicsUniversity of MinnesotaMinneapolisUSA
3. 3.Laboratoire de Combinatoire et d‘Informatique Mathématique (LaCIM)Université du Québec à MontréalMontréalCanada