Determinantal identities for flagged Schur and Schubert polynomials

Abstract

We prove new determinantal identities for a family of flagged Schur polynomials. As a corollary of these identities we obtain determinantal expressions of Schubert polynomials for certain vexillary permutations.

1 Introduction

Schur polynomials were defined by Issai Schur [12] as characters of irreducible representations of ; they are symmetric polynomials in n variables, indexed by Young diagrams $\lambda$ with at most n rows. Schur polynomials form a basis in the ring of all symmetric polynomials. This basis plays a prominent role in various algebraic problems.

Schur polynomials also have a combinatorial definition: they are obtained as sums of monomials indexed by semistandard Young tableaux of shape $\lambda$. These are fillings of boxes of $\lambda$ by nonnegative integers satisfying certain conditions. This construction is due to W. Specht [14], who showed that such tableaux index a basis in the representation of corresponding to $\lambda$.

The classical Jacobi–Trudi formula provides a determinantal expression for Schur polynomials via complete symmetric functions: for a partition $\lambda =(\lambda _1,\lambda _2,\dots ,\lambda _m)$, the corresponding Schur polynomial $s_\lambda (x_1,\dots ,x_n)$ is equal to

$$\begin{aligned} s_\lambda (x_1,\dots ,x_n)=\det {( h_{\lambda _i-i+j})_{i,j=1}^m}, \end{aligned}$$

where

is the k-th complete symmetric function, i.e., the sum of all monomials of degree k in variables $x_1,\dots ,x_n$.

In 1982, A. Lascoux and M.-P. Schützenberger generalized the notion of Schur polynomials, defining flagged Schur polynomials. Their definition is similar to the combinatorial definition of Schur polynomials, with the only difference: they put extra constraints on the entries occuring in semistandard Young tableaux, encoded by a sequence of integers referred to as a flag. These polynomials are not symmetric anymore. However, they also satisfy an analogue of the Jacobi–Trudi identity, due to I. Gessel and M. Wachs.

Flagged Schur polynomials were defined because of their relation to Schubert polynomials. This is another remarkable family of polynomials. It is also due to A. Lascoux and M.-P. Schützenberger. These polynomials are indexed by permutations; they represent cohomology classes of Schubert cycles in flag varieties. Just as Schur polynomials, they admit an algebraic definition (we recall it in Sect. 4.1), as well as a combinatorial one, due to S. Fomin and An. Kirillov [3]. The latter definition uses certain combinatorial objects called pipe dreams, or rc-graphs. It turns out that for a “nice” family of permutations, known as vexillary permutations, Schubert polynomials are equal to some flagged Schur polynomials. This gives determinantal expressions for Schubert polynomials of vexillary permutations.

In this paper we consider flagged Schur polynomials with flags of some special type (the so-called h-flagged ones). We show that for these polynomials there is a formula expressing them as determinants whose entries are 1-flagged Schur polynomials, divided by certain monomial denominators. This determinantal formula is different from the generalized Jacobi–Trudi formula discussed above.

The proof of our formula is based on the well-known Lindström–Gessel–Viennot lemma. We present flagged Schur polynomials as weighted sums over tuples of nonintersecting lattice paths and express these sums as determinants using this lemma.

Then we reinterpret our formula in terms of Schubert polynomials. It expresses the Schubert polynomial of an h-shifted dominant permutation as a determinant whose entries are Schubert polynomials of 1-shifted dominant permutations. In particular, this gives determinantal formulas for the Schubert polynomials corresponding to the conjecturally “most singular” Schubert varieties.

This paper is organized as follows. In Sect. 2 we give the definition of flagged Schur varieties and recall the generalized Jacobi–Trudi identity. In Sect. 3 we present a path interpretation of h-flagged Schur polynomials and prove our main result, Theorem 3.7. To make our exposition complete, in Sect. 3.3 we give a proof of our main combinatorial tool, the Lindström–Gessel–Viennot lemma. Section 4 is devoted to Schubert polynomials. We recall their definition, discuss their relation with flagged Schur polynomials and reformulate our main result in terms of Schubert polynomials (Theorem 4.6). Finally, in Sect. 5 we speak about its applications to some special classes of permutations and discuss its (mostly conjectural) relations to the geometry of Schubert varieties.

2 Flagged Schur polynomials and the Jacobi–Trudi identity

2.1 Definition of flagged Schur polynomials

We start with recalling the definition of flagged Schur polynomials, introduced by A. Lascoux and M.-P. Schützenberger [9].

Fix two positive integers $m\leqslant n$. Let $\lambda =(\lambda _1\geqslant \dots \geqslant \lambda _m>0)$ be a partition and let $b=(b_1\leqslant b_2\leqslant \dots \leqslant b_m=n)$ be a sequence of nonstrictly increasing positive integers. A flagged semistandard Young tableau T of shape $\lambda$ and flags b is an array $t_{ij}$ of positive integers $t_{ij}$, $1\leqslant j\leqslant \lambda _i$, $1\leqslant i\leqslant m$, such that $1\leqslant t_{ij}\leqslant t_{i,j+1}\leqslant b_i$ and $t_{ij}<t_{i+1,j}$. In other words, the boxes of a Young diagram of shape $\lambda$ are filled by numbers $t_{ij}$ non-strictly increasing along the rows and strictly increasing along the columns, in such a way that the numbers in the i-th row do not exceed $b_i$. Denote the set of all such tableaux by ${\mathscr {T}}(\lambda ,b)$. To each tableau T we assign a monomial , where $p_k$ is the number of entries in T equal to k.

Definition 2.1

A flagged Schur polynomial of shape $\lambda$ and flags b is defined as

Example 2.2

If $b_1=\dots =b_m=n$, then ${\mathscr {T}}(\lambda ,b)$ is the set of semistandard Young tableaux of shape $\lambda$ whose entries do not exceed n. In this case $s_\lambda (b)$ is just the usual Schur polynomial $s_\lambda (x_1,\dots ,x_n)$ in n variables.

2.2 h-flagged Schur polynomials

We will be mostly interested in flagged Schur polynomials with flags of a special form.

Definition 2.3

Let h be a positive integer, $b=(h+1,h+2,\dots , h+m)$. Then $s_{\lambda }^{(h)}=s_\lambda (b)$ is called an h-flagged Schur polynomial of shape $\lambda$. We will also speak about h-flagged semistandard Young tableaux of shape $\lambda$.

Example 2.4

1-flagged semistandard Young tableaux of shape $\lambda$ correspond to Young subdiagrams $\mu \subset \lambda$. Indeed, let T be such a tableau. The entries $t_{ij}$ in each (i-th) row of T are equal either to i or to $i+1$. The union of all boxes $(i,j)\in T$ such that $t_{ij}=i$ forms a subdiagram $\mu$, and $\mu$ uniquely determines the tableau T.

Example 2.5

h-flagged semistandard Young tableaux of shape $\lambda$ correspond to h-tuples

$$\begin{aligned} \mu _1\subseteq \mu _2\subseteq \cdots \subseteq \mu _h\subseteq \lambda \end{aligned}$$

of embedded Young diagrams, where $\mu _\ell =\{(i,j): t_{ij}\leqslant i-1+\ell \}$. Alternatively, replacing each entry $t_{ij}$ by $t_{ij}'=h+i-t_{ij}$ transforms a semistandard Young tableau T into a plane partition $T'$, i.e. a tableau of the same shape whose nonnegative entries non-strictly decrease along both rows and columns. Such tableau can be viewed as a three-dimensional Young diagram of height at most h whose base fits in $\lambda$.

Example 2.6

$s_{(2,1)}^{(1)}(x_1,x_2,x_3)=x_1^2x_2+x_1^2x_3+x_1x_2x_3+x_1x^2_2+x_2^2x_3$. The set of 1-flagged Young tableaux for the diagram (2, 1) is as follows:

2.3 Generalized Jacobi–Trudi identity

The following determinantal expression for flagged Schubert polynomials is well known. It appeared first in the paper [5] by I. Gessel. Its proof is also given in M. Wachs’s paper [17, Theorem 1.3]. In their paper [2], Y. Chen, B. Li and J. Louck generalize this identity for the case of double flagged Schur polynomials and give yet another its proof, using the lattice path interpretation of these polynomials and the Lindström–Gessel–Viennot lemma.

Theorem 2.7

([2, 5, 17]) Let $s_\lambda (b)$ be a flagged Schur function of shape $\lambda$ and flags b. Then

$$\begin{aligned} s_\lambda (b)=\det {(h_{\lambda _i-i+j}(b_i))_{1\leqslant i,j\leqslant m}}, \end{aligned}$$

where

is the complete symmetric function of degree d in k variables, $h_0=1$, and $h_\ell =0$ for $\ell <0$.

Corollary 2.8

For $b=(n,\dots ,n)$, this is the classical Jacobi–Trudi formula for Schur polynomials in n variables:

$$\begin{aligned} s_\lambda (x_1,\dots ,x_n)=\det {(h_{\lambda _i-i+j}(x_1,\dots ,x_n))_{1\leqslant i,j\leqslant m}}. \end{aligned}$$

Remark 2.9

The “path interpretation” of the Jacobi–Trudi formula and its proof using the Lindström–Gessel–Viennot lemma can be found, for instance, in [15, 7.16].

Fig. 1

Correspondence between subdiagrams and paths

3 Another determinantal formula for flagged Schur polynomials

3.1 Path interpretation of 1-flagged Schur polynomials

Let $\lambda =(\lambda _1,\dots ,\lambda _{n})$ be a fixed partition. Consider an oriented graph $G_\lambda$ whose vertices are lattice points inside $\lambda$ or at its boundary, and edges are given by vertical and horizontal line segments joining the neighboring lattice points. The orientation on the edges is as follows: the horizontal and the vertical edges are directed towards east and north, respectively. Let us assign weights to the edges of $G_\lambda$ in the following way: to each horizontal edge $(r,-s)-(r+1,-s)$ we assign the weight $x^{-1}_{s+1}$, as shown on Fig. 1, while all vertical edges have weight 1. For a path P, its weight $\mathop {\mathrm {wt}}P$ is defined as the product of the weights of all its edges.

Definition 3.1

Let A and B be two vertices of $G_\lambda$. The partition function $Z_\lambda (A,B)$ is defined as the weighted sum over all paths from A to B,

The following easy proposition expresses 1-flagged Schur polynomials via partition functions.

Proposition 3.2

Proof

As it was discussed in Example 2.4, the Young subdiagrams $\mu \subset \lambda$ correspond to 1-flagged semistandard Young tableaux of the shape $\lambda$. Each of these tableaux defines a monomial in $x_1,\dots ,x_{n+1}$, which we denote by $x^\mu$. On the other hand, let $P_\mu$ be the path leading from A to B and bounding the diagram $\mu$ from below. Let us prove that

This can be proven by induction on the number of boxes in $\mu$: for $\mu =\varnothing$, the weight of the corresponding tableau is equal to $x_2^{\lambda _1}x_3^{\lambda _2}\!\cdots x_{n+1}^{\lambda _n}=x_1^{\lambda _1}x_2^{\lambda _1}\!\cdots x_{n+1}^{\lambda _n}/x_1^{\lambda _1}$. Adding a box in the i-th row of $\mu$ results in multiplying this weight by $x_i/x_{i+1}$. $\square$

Let us introduce the following notation. Let $\mu =(\mu _1,\dots ,\mu _m)$ be a Young diagram, and $(y_1,\dots ,y_m)$ an (ordered) sequence of variables, with the number of variables equal to the number of rows in $\mu$. We set

$$\begin{aligned} \underline{(y_1,\dots ,y_m)}^\mu =y_1^{\mu _1}\!\cdots y_m^{\mu _m}. \end{aligned}$$

With this notation, Proposition 3.2 can be rewritten as follows:

Example 3.3

The diagram $\lambda =(5,3,2,2)$ with the weights on its edges is shown on Fig. 1. Consider the path $P_\mu$ drawn in red; its weight equals $\mathop {\mathrm {wt}}P_\mu =x_1^{-2}x_2^{-1}x_4^{-2}$. The corresponding summand in the flagged Schur polynomial is then equal to

The Young tableau corresponding to $\mu$ is as follows:

3.2 Expressing h-flagged Schur polynomials via 1-flagged Schur polynomials

In this subsection we give a lattice path expression for h-flagged Schur polynomials, different from the one given in Theorem 2.7. Using the Lindström–Gessel–Viennot lemma, we express them as determinants involving 1-flagged polynomials from the previous subsection.

As before, let $\lambda =(\lambda _1,\dots ,\lambda _m)$ be a Young tableau, and let $h\geqslant 1$. We consider h-flagged Young tableaux: the tableaux with flags $(h+1,\dots ,h+m)$. To each such tableau T we can assign an h-tuple of lattice paths drawn in the fourth quarter of the plane that start at the point , end at $B=(\ell ,0)$, where $\ell =\lambda _1$, and pass inside the diagram $\lambda$. These paths are constructed in the following way. For $k\leqslant h$, denote by $\mu _k$ the set of boxes $t_{ij}$ of T such that $t_{ij}\leqslant i+k-1$. We thus obtain a set of embedded Young tableaux $\mu _1\subset \mu _2\subset \cdots \subset \mu _h\subset \lambda$. Consider the set of lattice paths $P_1,\dots ,P_h$ bounding these diagrams; they pass non-strictly above one another, and all of them are bounded by $\lambda$ from below.

Example 3.4

Fig. 2 shows a 2-flagged Young tableau for the diagram $\lambda =(5,4,4,2)$. It corresponds to the pair of paths, shown at the figure on the right.

Fig. 2

Paths corresponding to a 2-flagged tableau

Now let us shift each (k-th) path to the vector . This will give us an h-tuple of non-crossing paths. The k-th path $P_k$ will then start at the point and end at .

Introduce weights at the horizontal segments of the lattice as in the previous example. One can easily see that the monomial corresponding to T is nothing but

Note that the multiple in front of the product of the weights is the inverse to the product of the weights of all edges in this diagram.

Fig. 3

Shifted and extended paths

Let us also extend paths as shown on the picture: the k-th path $P_k$ is extended by $h-k$ vertical segments down and by $h-k$ horizontal segments to the right. Denote the extended path by $P_k'$ and its endpoints by $A_k'$ and $B_k'$. The extensions are shown on Fig. 3 by dashed lines.

It is clear that the weights of $P_k$ and $P_k'$ differ by a “staircase monomial”: $\mathop {\mathrm {wt}}P_k=\mathop {\mathrm {wt}}P_k'\cdot x_k^{h-k}$. This means that

This allows us to express h-flagged Schur polynomials as weighted sums over sets of noncrossing paths. To do this, let us introduce some further notation.

Definition 3.5

Let $\lambda =(\lambda _1,\dots ,\lambda _m)$ be a Young diagram, and $k,\ell$ two nonnegative integers. Let us shift $\lambda$ by the vector . The minimal Young diagram containing this set is denoted by $\lambda (k,\ell )$. In other terms, is obtained from $\lambda$ by adding k rows of length $\lambda _1$ from above and $\ell$ columns of height m from the left.

Consider the diagram . Let $A_i'$ and $B_i'$ be as shown in Fig. 3: namely, and $B'_i=( \lambda _1+h-1,1-i)$. Denote by $Z_{\mathrm{nc}}((A_1',B_1'),\dots ,(A_h',B_h'))$ the weighted sum taken over all h-tuples of non-crossing paths joining $A'_1$ with $B_1'$, $\dots$, $A_h'$ with $B'_h$ and contained in the diagram . (The index “nc” stands for “non-crossing”). The above discussion shows that the h-flagged Schur function $s_\lambda ^{h}(x_1,\dots ,x_n)$ equals

The following statement is a particular case of the well-known Lindström–Gessel–Viennot lemma. To make our exposition complete, we formulate this lemma in full generality and give its proof in Sect. 3.3.

Lemma 3.6

The following identity holds:

This lemma together with the expressions for Schur polynomials via lattice paths gives us the following result which expresses an h-flagged Schur polynomial via 1-flagged Schur polynomials.

Theorem 3.7

The following equality holds:

Remark 3.8

This determinantal formula is different from the Jacobi–Trudi formula (and, to the best of our knowledge, cannot be reduced to it). Note, in particular, that all matrix entries in the Jacobi–Trudi formula are polynomials in $x_1,\dots ,x_n$, while the matrix elements in our formula are polynomials in different sets of variables.

Example 3.9

Let $h=2$, $\lambda =(2,1)$. Then our formula looks as follows:

We will also need a slightly different form of this determinantal formula. It can be obtained from Theorem 3.7 by algebraic manipulations with determinants, but we will give a combinatorial proof instead. Let us modify Definition 3.5 in the following way.

Definition 3.10

Let $\lambda =(\lambda _1,\dots ,\lambda _m)$ be a Young diagram. A $(k,\ell )$-extended diagram (notation: ) is defined as follows:

In other words, is obtained from $\lambda$ by adding “staircases” of size k and $\ell$ on the top and to the left of it, respectively.

Example 3.11

This picture shows $\lambda =(3,3,1)$ and .

Now look at Fig. 3 again. We can extend the lengths of paths not by $1, 2, \dots , h-1$, but by . The result is shown on Fig. 4.

Fig. 4

Another way of extending paths

Note that the k-th path is situated inside the diagram . This allows us to formulate the following corollary of Theorem 3.7. Its proof also follows from the Lindström–Gessel–Viennot lemma; the only difference concerns the weights which are added while extending paths.

Corollary 3.12

The following equality holds:

Remark 3.13

This corollary can also be obtained from Theorem 3.7 by purely algebraic manipulations with the determinant.

Remark 3.14

Note that the right-hand side depends upon $x_k$ for $k>h+m$, while the left-hand side does not. This means that all these $x_k$ cancel out while evaluating the determinant.

We will need this form of our determinantal identity while dealing with Schubert polynomials in the next section.

3.3 The Lindström–Gessel–Viennot lemma

In this subsection we prove the Lindström–Gessel–Viennot lemma on noncrossing paths. This material is by no means new: having appeared first in the preprint [6], now it can be found in many books and articles, for instance, in [16].

Let ${\Gamma }$ be an arbitrary oriented graph without oriented cycles, with weights assigned to its edges (we suppose that the weights are elements of a certain ring R). For a path P joining two vertices A and B, by the weight $\mathop {\mathrm {wt}}P$ of P we mean the product of weights of all of its edges. As before, we define the partition function Z(A, B) to be the sum of weights of all paths joining A with B,

An h-tuple of paths $P_1:A_1\rightarrow B_1$, $\dots$, $P_h:A_h\rightarrow B_h$ is said to be noncrossing if any two paths from it have no common points. An (ordered) set of h starting points and h endpoints $(A_1,\dots ,A_h)$ and $(B_1,\dots ,B_h)$ is said to be compatible if for any permutation $\sigma \in S_h$ the existence of a noncrossing h-tuple of paths joining $A_1$ with $B_{\sigma (1)}$, $\dots$, $A_h$ with $B_{\sigma (h)}$ implies that $\sigma$ is the identity permutation.

For two h-tuples of starting points and endpoints, consider a weighted sum over all h-tuples of noncrossing paths $(P_1,\dots ,P_h)$,

Lemma 3.15

(Lindström–Gessel–Viennot) If $(A_1,\dots ,A_h)$ and $(B_1,\dots ,B_h)$ are compatible, the function $Z_{\mathrm{nc}}((A_1,B_1),\dots ,(A_h,B_h))$ admits the following determinantal expression:

Proof

The determinant in the right-hand side can be expressed as the sum over permutations

(1)

This is an alternating sum over all h-tuples of paths joining $A_1,\dots ,A_h$ with $B_1,\dots ,B_h$ in arbitrary order, taken with the appropriate sign. Such an h-tuple can be either noncrossing or have crossings; in the former case, it corresponds to the identity permutation. So we only need to show that the alternating sum over all h-tuples of paths with crossings is zero.

To do this, we construct a weight-preserving bijection on the set of h-tuples of paths with crossings. It is constructed as follows. We start with an arbitrary h-tuple of paths $(P_1,\dots ,P_h)$ with crossings, corresponding to a permutation $\sigma$. Consider the path $P_i$ with the smallest number of its starting point, such that it meets another path, say, $P_j$. Let C be the first common point of these paths. Now let us “switch the tails” of these two paths: let $\widetilde{P}_i$ be the path which coincides with $P_i$ before C and with $P_j$ after C. Likewise, let $\widetilde{P}_j$ coincide with $P_j$ before C and with $P_i$ after C (see Fig. 5). We have obtained a new h-tuple of paths $(P_1,\dots ,\widetilde{P}_i,\dots ,\widetilde{P}_j,\dots ,P_h)$.

Fig. 5

Involution on the set of h-tuples of crossing paths

This map on the h-tuples of crossing paths is involutive. Obviously, $\mathop {\mathrm {wt}}P_i\mathop {\mathrm {wt}}P_j=\mathop {\mathrm {wt}}\widetilde{P}_i\mathop {\mathrm {wt}}\widetilde{P}_j$, so it is weight-preserving. Finally, the permutations corresponding to $(P_1,\dots ,P_h)$ and $(P_1,\dots ,\widetilde{P}_i,\dots ,\widetilde{P}_j,\dots ,P_h)$ are obtained from each other by multiplication by a transposition, so they are counted in expression (1) with the opposite signs. This means that the total contribution of this pair of h-tuples of paths into (1) is zero, and the determinant is equal to the sum of weights over all h-tuples of noncrossing paths. The lemma is proven. $\square$

4 Schubert polynomials

4.1 Definition of Schubert polynomials

Schubert polynomials were introduced for studying the cohomology ring of a full flag variety. Their definition is due to A. Lascoux and M.-P. Schützeberger [9]; implicitly it appeared in the paper [1] by J. Bernstein, I. Gelfand and S. Gelfand. Let us recall it.

Let $S_n$ denote the symmetric group, i.e., the group of permutations of an n-element set $\{1,\dots , n\}$. We will use the one-line notation for permutations; i.e., (1342) is a permutation taking 1 to 1, 2 to 3, 3 to 4 and 4 to 2. Let $s_i=(1,2,\dots ,i+1,i,i+2,\dots ,n)$ denote the i-th simple transposition, and $w_0=(n,n-1,\dots ,2,1)$ the element of maximal length. We mean by $\ell (w)$ the length of the permutation w, i.e., the minimal number k of simple transpositions $s_{i_1},\dots ,s_{i_k}$ such that $w=s_{i_1}\!\dots s_{i_k}$.

For each $i\leqslant n-1$, define a divided difference operator as follows:

Definition 4.1

Let $w\in S_n$ be a permutation. Consider a reduced decomposition of into a product of simple transpositions:

Then the Schubert polynomial corresponding to w is defined as follows:

(Note that the order of indices is reversed.)

Since the divided difference operators satisfy the braid relations:

the polynomial ${\mathfrak {S}}_w$ depends only on w and not on its decomposition into simple transpositions.

There are standard embeddings $S_n\hookrightarrow S_m$ for $n<m$; under this embedding, $S_n$ permutes only the first n elements of $\{1,\dots , m\}$. Sometimes we will treat permutations as elements of the direct limit $S_\infty =\lim _{\rightarrow } S_n$. It can be shown (cf. [10]) that the Schubert polynomials are stable under these embeddings, so they can be regarded as elements of the polynomial ring in countably many variables.

4.2 Schubert and flagged Schur polynomials

Recall that a permutation $w\in S_n$ is said to be vexillary if it is (2143)-avoiding, i.e., there are no four numbers $i<j<k<\ell$ such that $w(j)<w(i)<w(\ell )<w(k)$. M. Wachs [17] shows that the Schubert polynomials of vexillary permutations can be obtained as flagged Schur polynomials for certain flags. This is done as follows.

We define the inversion sets of a permutation w as ${{\mathrm{I}}}_i(w)=\{j: j>i,\, w(j)<w(i)\}$. It can be shown that a permutation is vexillary iff its inversion sets form a chain (see [11] for details).

Let $c_i=\#\, {{\mathrm{I}}}_i$. The sequence of these cardinalities $(c_1,\dots ,c_n)$ is called the Lehmer code of w. Denote by $\lambda (w)$ the partition obtained by arranging $c_1,\dots , c_n$ in the decreasing order. Likewise, let $b(w)=(b_1(w)\leqslant \cdots \leqslant b_m(w))$ be the sequence of integers $\min {{\mathrm{I}}}_i(w)-1$ for all nonempty ${\mathrm{I}}_i(w)$, arranged in the increasing order.

Theorem 4.2

([17, Theorem 2.3]) If w is vexillary, then ${\mathfrak {S}}_w=s_{\lambda (w)}(b(w))$.

4.3 Dominant permutations

Definition 4.3

A permutation $w\in S_n$ is called dominant if it is 132-avoiding, i.e., there is no triple $i<j<k$ such that $w(i)<w(k)<w(j)$.

Clearly, if a permutation is 132-avoiding, it is also 2143-avoiding. So, all dominant permutations are vexillary.

It is well known (cf., for instance, [11]) that dominant permutations are characterized by the following property: their Schubert polynomial ${\mathfrak {S}}_w$ is a monomial in $x_1,\dots ,x_{n-1}$. Moreover, the Lehmer code $\lambda (w)$ of such a permutation is a partition, and ${\mathfrak {S}}_w=\underline{(x_1,\dots ,x_{n-1})}^{\lambda (w)}$.

For a dominant permutation w with its code $\lambda =(\lambda _1,\dots ,\lambda _n)$, let $(i_1,\dots ,i_k)$ be the numbers of the rows of $\lambda$ which are strictly shorter than the previous rows:

$$\begin{aligned} \lambda _1=\cdots =\lambda _{i_1-1}>\lambda _{i_2}=\cdots =\lambda _{i_3-1}>\cdots \end{aligned}$$

Then the flag of w is as follows:

$$\begin{aligned} b(w)=(i_1,\dots ,i_1,i_2,\dots ,i_2,\dots ), \end{aligned}$$

with $i_j-i_{j-1}$ entries equal to $i_j$.

A flagged tableau of shape $\lambda$ with this flag is unique: all its entries in the i-th row are equal to i. So we can replace this flag by $(1,2,3,\dots )$. Hence, for a dominant permutation w its Schubert polynomial is a “0-flagged Schur polynomial of shape $\lambda$”.

4.4 Shifts of permutations

Let $w\in S_n$ be a permutation, and let h be a positive integer. The h-shifted permutation is defined as

In different terms, we add h elements “in front of the set $\{1,\dots ,n\}$” acted on by permutations such that the action of on these new elements is trivial, and the action on the remaining elements coincides with the action of w. The following proposition is straightforward.

Proposition 4.4

Let w be a vexillary permutation with the Lehmer code $\lambda (w)$ and flag $b(w)=(b_1,b_2,\dots )$, and h a positive integer. Then

• is vexillary;

• 

• .

This gives us a description of permutations whose Schubert polynomials are equal to h-flagged Schur polynomials.

Remark 4.5

Of course, for a dominant permutation w, the permutations are not dominant anymore.

4.5 Extension of dominant permutations

In this subsection we define an operation on dominant permutations which corresponds to $(k,\ell )$-extension of their Young diagrams.

Let w be a dominant permutation. Define a (1, 0)-extended permutation $\widehat{w}[1,0]$ as follows:

This is shown on Fig. 6: the first entry is “doubled”, represented by a circle, is replaced by two entries. In this picture $w=(42135)$, . Clearly, the code of the latter permutation is obtained from the code of the former permutation exactly by the operation of (1, 0)-extension of Young diagrams in the sense of Definition 3.10. Moreover, is also dominant.

Fig. 6

(1,0)-extension of a permutation

In a similar manner we can define the (0, 1)-extension of a permutation. The only difference is that the element replaced by two elements is not 1, but the preimage of 1:

Figure 7 shows an example of the (0, 1)-extension. In this example the original permutation is (42135), and the extended permutation is (532146).

Fig. 7

(0, 1)-extension of a permutation

A $(k,\ell )$-extension of a dominant permutation is obtained as a result of k successive applications of (1, 0)-extension and $\ell$ applications of (0, 1)-extension (these operations commute). Again, the Young diagram corresponding to its code is obtained from the Young diagram of w by a $(k,\ell )$-extension.

4.6 Determinantal formula for Schubert polynomials of shifted dominant permutations

With the definitions from the two previous subsections, we can reformulate Corollary 3.12 for Schubert polynomials. The formula from that theorem will give the determinantal expression for the Schubert polynomial of an h-shifted dominant permutation w in terms of 1-shifted dominant permutations obtained from w by extensions.

Theorem 4.6

Let w be a dominant permutation, let $\lambda =(\lambda _1,\lambda _2,\dots )$ be its Lehmer code. Then the Schubert polynomial of the h-shifted permutation satisfies the following determinantal expression:

5 Schubert polynomials of Richardson permutations

In this section we discuss some corollaries of Theorem 4.6 and formulate several conjectures concerning Schubert polynomials.

5.1 Catalan numbers

Fix some notation. Let $w_0(n)=(n,n-1,\dots ,2,1)\in S_n$ be the element of maximal length. It is dominant, and the corresponding Young diagram is the staircase shape ${\Lambda }_n=(n-1,n-2,\dots ,2,1)$.

Recall that the n-th Catalan number is equal to the number of subdiagrams of the staircase shape ${\Lambda }_n$. This number has several different q-generalizations. Let $\mathop {\mathrm {Cat}}_q(n)$ denote the n-th Carlitz–Riordan q-Catalan number, obtained as the weighted sum of all such subdiagrams, where a subdiagram $\mu$ has weight $q^{n(n-1)/2-|\mu |}$,

$$\begin{aligned} \mathop {\mathrm {Cat}}\nolimits _q(n)=\!\sum _{\mu \subset {\Lambda }_n}\! q^{n(n-1)/2-|\mu |}. \end{aligned}$$

Here is a first observation, made by Alex Woo [18].

Theorem 5.1

([18]) Let . Then the principal specialization of ${\mathfrak {S}}_w$ equals the n-th Carlitz–Riordan q-Catalan number times a power of q,

Corollary 5.2

The value of ${\mathfrak {S}}_w$ at the identity equals the n-th Catalan number,

In this case ${\mathfrak {S}}_w=s_{{\Lambda }_n}^{(1)}$ is the 1-flagged polynomial of the staircase partition.

5.2 Catalan–Hankel determinants

Consider the same permutation $w_0(n)$ shifted by h instead of 1. Theorem 4.6 expresses ${\mathfrak {S}}_{1^h\times w_0}$ via ${\mathfrak {S}}_{1\times \widehat{w_0(n)}[k,\ell ]}$.

Note that , since the extension of a staircase diagram is again a staircase diagram. This means that all permutations occurring in the determinantal expression are ${\mathfrak {S}}_{1\times w_0(m)}$ for $m=n,\dots ,n+2h-2$.

Specializing this identity at 1 and using Corollary 5.2, we get the following proposition.

Proposition 5.3

The following identity holds:

This determinant is a Hankel determinant of the sequence of Catalan numbers; we will later refer to it as a Catalan–Hankel determinant.

Remark 5.4

Since ${\mathfrak {S}}_{1^h\times w_0(n)}$ is the flagged Schur polynomial $s_{{\Lambda }_n}^{(h)}$, we see from the definition of a flagged Schur polynomial that its value at 1 is equal to the number of plane partitions with parts at most h and shape ${\Lambda }_n$, i.e., three-dimensional Young diagrams inside a prism with base ${\Lambda }_n$ and height h. This observation appeared in [4, Section 1.2].

It is also possible to establish an explicit bijection between the monomials in ${\mathfrak {S}}_{1^h\times w_0(h)}$, indexed by the so-called “pipe dreams” (see [3]), and such plane partitions. See [13] for the details.

5.3 Richardson elements in the symmetric group

Definition 5.5

Let $i_1<i_2<i_3<\dots <i_k=n$. Permutation $(i_1,\,i_1-1,$ $\dots ,\,2,\,1,\,i_2,\,i_2-1,\,\dots ,\,i_1+1,\, i_3,\,\dots ,\,i_2+1,\,\dots )\in S_n$ is called a Richardson permutation.

In different terms, a Richardson permutation is equal to the product of several permutations of type with nonintersecting supports. In particular, $w_0(n)$ and are Richardson permutations.

Richardson elements have a nice description in terms of algebraic groups. For a parabolic subgroup consider its Levi decomposition . Then the longest element in the Weyl group $W_L$ of L is a Richardson element in the Weyl group $W\cong S_n$ of .

Richardson permutations are not vexillary (unless they are shifted staircase permutations). However, they are obtained as products of several shifted staircase permutations with pairwise disjoint supports. So the Schubert polynomial of such a permutation is equal to the product of the Schubert polynomials of its factors.

Hence, Theorem 4.6 shows that the Schubert polynomials of Richardson permutations can be presented as products of several determinants, and their specialization at 1 is equal to the product of several Catalan–Hankel determinants from Proposition 5.3.

The study of Richardson permutations is motivated by the following question.

Question 5.6

For a given n, find the permutations $w\in S_n$ such that ${\mathfrak {S}}_{w}(1,\dots ,1)$ is as large as possible.

This question has a geometric interpretation: A. Knutson and E. Miller [7] showed that the value ${\mathfrak {S}}_w(1,\dots ,1)$ is equal to the degree of the matrix Schubert variety corresponding to w. If w satisfies the condition that, for every (i, j) with $i+j>n$, either $(w_0w)^{-1}(i)\leqslant j$ or $w_0w(j)\leqslant i$, this degree equals the multiplicity of the singular point $X_e\subset X_w$, where is the Schubert variety in a full flag variety corresponding to w. This multiplicity is a measure of “how singular” $X_w$ is; we are then trying to find the Schubert variety with the “worst” singularities (in the aforementioned sense).

Table 1 Permutations producing the largest values of ${\mathfrak {S}}_w(1,\dots ,1)$

These values were computed for all permutations with $n\leqslant 10$. It turned out that the maximum is always attained at a Richardson permutation. This allows us to formulate the following conjecture.

Conjecture 5.7

For a given n, the permutation $w\in S_n$ with the largest value of ${\mathfrak {S}}_w(1,\dots ,1)$ is Richardson.

In Table 1 we give the list of permutations w corresponding to the largest value of ${\mathfrak {S}}_w(1,\dots ,1)$ and the values themselves for $n\leqslant 10$ (for $n=5,6$, there is more than one such permutation). This table was computed by I. Kochulin [8].

However, this conjecture remains a purely experimental observation; so far we cannot see any reason why the permutation producing the largest value of its Schubert polynomial at 1 has to be Richardson.