Abstract
We prove new determinantal identities for a family of flagged Schur polynomials. As a corollary of these identities we obtain determinantal expressions of Schubert polynomials for certain vexillary permutations.
Introduction
Schur polynomials were defined by Issai Schur [12] as characters of irreducible representations of ; they are symmetric polynomials in n variables, indexed by Young diagrams \(\lambda \) with at most n rows. Schur polynomials form a basis in the ring of all symmetric polynomials. This basis plays a prominent role in various algebraic problems.
Schur polynomials also have a combinatorial definition: they are obtained as sums of monomials indexed by semistandard Young tableaux of shape \(\lambda \). These are fillings of boxes of \(\lambda \) by nonnegative integers satisfying certain conditions. This construction is due to W. Specht [14], who showed that such tableaux index a basis in the representation of corresponding to \(\lambda \).
The classical Jacobi–Trudi formula provides a determinantal expression for Schur polynomials via complete symmetric functions: for a partition \(\lambda =(\lambda _1,\lambda _2,\dots ,\lambda _m)\), the corresponding Schur polynomial \(s_\lambda (x_1,\dots ,x_n)\) is equal to
where
is the kth complete symmetric function, i.e., the sum of all monomials of degree k in variables \(x_1,\dots ,x_n\).
In 1982, A. Lascoux and M.P. Schützenberger generalized the notion of Schur polynomials, defining flagged Schur polynomials. Their definition is similar to the combinatorial definition of Schur polynomials, with the only difference: they put extra constraints on the entries occuring in semistandard Young tableaux, encoded by a sequence of integers referred to as a flag. These polynomials are not symmetric anymore. However, they also satisfy an analogue of the Jacobi–Trudi identity, due to I. Gessel and M. Wachs.
Flagged Schur polynomials were defined because of their relation to Schubert polynomials. This is another remarkable family of polynomials. It is also due to A. Lascoux and M.P. Schützenberger. These polynomials are indexed by permutations; they represent cohomology classes of Schubert cycles in flag varieties. Just as Schur polynomials, they admit an algebraic definition (we recall it in Sect. 4.1), as well as a combinatorial one, due to S. Fomin and An. Kirillov [3]. The latter definition uses certain combinatorial objects called pipe dreams, or rcgraphs. It turns out that for a “nice” family of permutations, known as vexillary permutations, Schubert polynomials are equal to some flagged Schur polynomials. This gives determinantal expressions for Schubert polynomials of vexillary permutations.
In this paper we consider flagged Schur polynomials with flags of some special type (the socalled hflagged ones). We show that for these polynomials there is a formula expressing them as determinants whose entries are 1flagged Schur polynomials, divided by certain monomial denominators. This determinantal formula is different from the generalized Jacobi–Trudi formula discussed above.
The proof of our formula is based on the wellknown Lindström–Gessel–Viennot lemma. We present flagged Schur polynomials as weighted sums over tuples of nonintersecting lattice paths and express these sums as determinants using this lemma.
Then we reinterpret our formula in terms of Schubert polynomials. It expresses the Schubert polynomial of an hshifted dominant permutation as a determinant whose entries are Schubert polynomials of 1shifted dominant permutations. In particular, this gives determinantal formulas for the Schubert polynomials corresponding to the conjecturally “most singular” Schubert varieties.
This paper is organized as follows. In Sect. 2 we give the definition of flagged Schur varieties and recall the generalized Jacobi–Trudi identity. In Sect. 3 we present a path interpretation of hflagged Schur polynomials and prove our main result, Theorem 3.7. To make our exposition complete, in Sect. 3.3 we give a proof of our main combinatorial tool, the Lindström–Gessel–Viennot lemma. Section 4 is devoted to Schubert polynomials. We recall their definition, discuss their relation with flagged Schur polynomials and reformulate our main result in terms of Schubert polynomials (Theorem 4.6). Finally, in Sect. 5 we speak about its applications to some special classes of permutations and discuss its (mostly conjectural) relations to the geometry of Schubert varieties.
Flagged Schur polynomials and the Jacobi–Trudi identity
Definition of flagged Schur polynomials
We start with recalling the definition of flagged Schur polynomials, introduced by A. Lascoux and M.P. Schützenberger [9].
Fix two positive integers \(m\leqslant n\). Let \(\lambda =(\lambda _1\geqslant \dots \geqslant \lambda _m>0)\) be a partition and let \(b=(b_1\leqslant b_2\leqslant \dots \leqslant b_m=n)\) be a sequence of nonstrictly increasing positive integers. A flagged semistandard Young tableau T of shape \(\lambda \) and flags b is an array \(t_{ij}\) of positive integers \(t_{ij}\), \(1\leqslant j\leqslant \lambda _i\), \(1\leqslant i\leqslant m\), such that \(1\leqslant t_{ij}\leqslant t_{i,j+1}\leqslant b_i\) and \(t_{ij}<t_{i+1,j}\). In other words, the boxes of a Young diagram of shape \(\lambda \) are filled by numbers \(t_{ij}\) nonstrictly increasing along the rows and strictly increasing along the columns, in such a way that the numbers in the ith row do not exceed \(b_i\). Denote the set of all such tableaux by \({\mathscr {T}}(\lambda ,b)\). To each tableau T we assign a monomial , where \(p_k\) is the number of entries in T equal to k.
Definition 2.1
A flagged Schur polynomial of shape \(\lambda \) and flags b is defined as
Example 2.2
If \(b_1=\dots =b_m=n\), then \({\mathscr {T}}(\lambda ,b)\) is the set of semistandard Young tableaux of shape \(\lambda \) whose entries do not exceed n. In this case \(s_\lambda (b)\) is just the usual Schur polynomial \(s_\lambda (x_1,\dots ,x_n)\) in n variables.
hflagged Schur polynomials
We will be mostly interested in flagged Schur polynomials with flags of a special form.
Definition 2.3
Let h be a positive integer, \(b=(h+1,h+2,\dots , h+m)\). Then \(s_{\lambda }^{(h)}=s_\lambda (b)\) is called an hflagged Schur polynomial of shape \(\lambda \). We will also speak about hflagged semistandard Young tableaux of shape \(\lambda \).
Example 2.4
1flagged semistandard Young tableaux of shape \(\lambda \) correspond to Young subdiagrams \(\mu \subset \lambda \). Indeed, let T be such a tableau. The entries \(t_{ij}\) in each (ith) row of T are equal either to i or to \(i+1\). The union of all boxes \((i,j)\in T\) such that \(t_{ij}=i\) forms a subdiagram \(\mu \), and \(\mu \) uniquely determines the tableau T.
Example 2.5
hflagged semistandard Young tableaux of shape \(\lambda \) correspond to htuples
of embedded Young diagrams, where \(\mu _\ell =\{(i,j): t_{ij}\leqslant i1+\ell \}\). Alternatively, replacing each entry \(t_{ij}\) by \(t_{ij}'=h+it_{ij}\) transforms a semistandard Young tableau T into a plane partition \(T'\), i.e. a tableau of the same shape whose nonnegative entries nonstrictly decrease along both rows and columns. Such tableau can be viewed as a threedimensional Young diagram of height at most h whose base fits in \(\lambda \).
Example 2.6
\(s_{(2,1)}^{(1)}(x_1,x_2,x_3)=x_1^2x_2+x_1^2x_3+x_1x_2x_3+x_1x^2_2+x_2^2x_3\). The set of 1flagged Young tableaux for the diagram (2, 1) is as follows:
Generalized Jacobi–Trudi identity
The following determinantal expression for flagged Schubert polynomials is well known. It appeared first in the paper [5] by I. Gessel. Its proof is also given in M. Wachs’s paper [17, Theorem 1.3]. In their paper [2], Y. Chen, B. Li and J. Louck generalize this identity for the case of double flagged Schur polynomials and give yet another its proof, using the lattice path interpretation of these polynomials and the Lindström–Gessel–Viennot lemma.
Theorem 2.7
([2, 5, 17]) Let \(s_\lambda (b)\) be a flagged Schur function of shape \(\lambda \) and flags b. Then
where
is the complete symmetric function of degree d in k variables, \(h_0=1\), and \(h_\ell =0\) for \(\ell <0\).
Corollary 2.8
For \(b=(n,\dots ,n)\), this is the classical Jacobi–Trudi formula for Schur polynomials in n variables:
Remark 2.9
The “path interpretation” of the Jacobi–Trudi formula and its proof using the Lindström–Gessel–Viennot lemma can be found, for instance, in [15, 7.16].
Another determinantal formula for flagged Schur polynomials
Path interpretation of 1flagged Schur polynomials
Let \(\lambda =(\lambda _1,\dots ,\lambda _{n})\) be a fixed partition. Consider an oriented graph \(G_\lambda \) whose vertices are lattice points inside \(\lambda \) or at its boundary, and edges are given by vertical and horizontal line segments joining the neighboring lattice points. The orientation on the edges is as follows: the horizontal and the vertical edges are directed towards east and north, respectively. Let us assign weights to the edges of \(G_\lambda \) in the following way: to each horizontal edge \((r,s)(r+1,s)\) we assign the weight \(x^{1}_{s+1}\), as shown on Fig. 1, while all vertical edges have weight 1. For a path P, its weight \(\mathop {\mathrm {wt}}P\) is defined as the product of the weights of all its edges.
Definition 3.1
Let A and B be two vertices of \(G_\lambda \). The partition function \(Z_\lambda (A,B)\) is defined as the weighted sum over all paths from A to B,
The following easy proposition expresses 1flagged Schur polynomials via partition functions.
Proposition 3.2
Proof
As it was discussed in Example 2.4, the Young subdiagrams \(\mu \subset \lambda \) correspond to 1flagged semistandard Young tableaux of the shape \(\lambda \). Each of these tableaux defines a monomial in \(x_1,\dots ,x_{n+1}\), which we denote by \(x^\mu \). On the other hand, let \(P_\mu \) be the path leading from A to B and bounding the diagram \(\mu \) from below. Let us prove that
This can be proven by induction on the number of boxes in \(\mu \): for \(\mu =\varnothing \), the weight of the corresponding tableau is equal to \(x_2^{\lambda _1}x_3^{\lambda _2}\!\cdots x_{n+1}^{\lambda _n}=x_1^{\lambda _1}x_2^{\lambda _1}\!\cdots x_{n+1}^{\lambda _n}/x_1^{\lambda _1}\). Adding a box in the ith row of \(\mu \) results in multiplying this weight by \(x_i/x_{i+1}\). \(\square \)
Let us introduce the following notation. Let \(\mu =(\mu _1,\dots ,\mu _m)\) be a Young diagram, and \((y_1,\dots ,y_m)\) an (ordered) sequence of variables, with the number of variables equal to the number of rows in \(\mu \). We set
With this notation, Proposition 3.2 can be rewritten as follows:
Example 3.3
The diagram \(\lambda =(5,3,2,2)\) with the weights on its edges is shown on Fig. 1. Consider the path \(P_\mu \) drawn in red; its weight equals \(\mathop {\mathrm {wt}}P_\mu =x_1^{2}x_2^{1}x_4^{2}\). The corresponding summand in the flagged Schur polynomial is then equal to
The Young tableau corresponding to \(\mu \) is as follows:
Expressing hflagged Schur polynomials via 1flagged Schur polynomials
In this subsection we give a lattice path expression for hflagged Schur polynomials, different from the one given in Theorem 2.7. Using the Lindström–Gessel–Viennot lemma, we express them as determinants involving 1flagged polynomials from the previous subsection.
As before, let \(\lambda =(\lambda _1,\dots ,\lambda _m)\) be a Young tableau, and let \(h\geqslant 1\). We consider hflagged Young tableaux: the tableaux with flags \((h+1,\dots ,h+m)\). To each such tableau T we can assign an htuple of lattice paths drawn in the fourth quarter of the plane that start at the point , end at \(B=(\ell ,0)\), where \(\ell =\lambda _1\), and pass inside the diagram \(\lambda \). These paths are constructed in the following way. For \(k\leqslant h\), denote by \(\mu _k\) the set of boxes \(t_{ij}\) of T such that \(t_{ij}\leqslant i+k1\). We thus obtain a set of embedded Young tableaux \(\mu _1\subset \mu _2\subset \cdots \subset \mu _h\subset \lambda \). Consider the set of lattice paths \(P_1,\dots ,P_h\) bounding these diagrams; they pass nonstrictly above one another, and all of them are bounded by \(\lambda \) from below.
Example 3.4
Fig. 2 shows a 2flagged Young tableau for the diagram \(\lambda =(5,4,4,2)\). It corresponds to the pair of paths, shown at the figure on the right.
Now let us shift each (kth) path to the vector . This will give us an htuple of noncrossing paths. The kth path \(P_k\) will then start at the point and end at .
Introduce weights at the horizontal segments of the lattice as in the previous example. One can easily see that the monomial corresponding to T is nothing but
Note that the multiple in front of the product of the weights is the inverse to the product of the weights of all edges in this diagram.
Let us also extend paths as shown on the picture: the kth path \(P_k\) is extended by \(hk\) vertical segments down and by \(hk\) horizontal segments to the right. Denote the extended path by \(P_k'\) and its endpoints by \(A_k'\) and \(B_k'\). The extensions are shown on Fig. 3 by dashed lines.
It is clear that the weights of \(P_k\) and \(P_k'\) differ by a “staircase monomial”: \(\mathop {\mathrm {wt}}P_k=\mathop {\mathrm {wt}}P_k'\cdot x_k^{hk}\). This means that
This allows us to express hflagged Schur polynomials as weighted sums over sets of noncrossing paths. To do this, let us introduce some further notation.
Definition 3.5
Let \(\lambda =(\lambda _1,\dots ,\lambda _m)\) be a Young diagram, and \(k,\ell \) two nonnegative integers. Let us shift \(\lambda \) by the vector . The minimal Young diagram containing this set is denoted by \(\lambda (k,\ell )\). In other terms, is obtained from \(\lambda \) by adding k rows of length \(\lambda _1\) from above and \(\ell \) columns of height m from the left.
Consider the diagram . Let \(A_i'\) and \(B_i'\) be as shown in Fig. 3: namely, and \(B'_i=( \lambda _1+h1,1i)\). Denote by \(Z_{\mathrm{nc}}((A_1',B_1'),\dots ,(A_h',B_h'))\) the weighted sum taken over all htuples of noncrossing paths joining \(A'_1\) with \(B_1'\), \(\dots \), \(A_h'\) with \(B'_h\) and contained in the diagram . (The index “nc” stands for “noncrossing”). The above discussion shows that the hflagged Schur function \(s_\lambda ^{h}(x_1,\dots ,x_n)\) equals
The following statement is a particular case of the wellknown Lindström–Gessel–Viennot lemma. To make our exposition complete, we formulate this lemma in full generality and give its proof in Sect. 3.3.
Lemma 3.6
The following identity holds:
This lemma together with the expressions for Schur polynomials via lattice paths gives us the following result which expresses an hflagged Schur polynomial via 1flagged Schur polynomials.
Theorem 3.7
The following equality holds:
Remark 3.8
This determinantal formula is different from the Jacobi–Trudi formula (and, to the best of our knowledge, cannot be reduced to it). Note, in particular, that all matrix entries in the Jacobi–Trudi formula are polynomials in \(x_1,\dots ,x_n\), while the matrix elements in our formula are polynomials in different sets of variables.
Example 3.9
Let \(h=2\), \(\lambda =(2,1)\). Then our formula looks as follows:
We will also need a slightly different form of this determinantal formula. It can be obtained from Theorem 3.7 by algebraic manipulations with determinants, but we will give a combinatorial proof instead. Let us modify Definition 3.5 in the following way.
Definition 3.10
Let \(\lambda =(\lambda _1,\dots ,\lambda _m)\) be a Young diagram. A \((k,\ell )\)extended diagram (notation: ) is defined as follows:
In other words, is obtained from \(\lambda \) by adding “staircases” of size k and \(\ell \) on the top and to the left of it, respectively.
Example 3.11
This picture shows \(\lambda =(3,3,1)\) and .
Now look at Fig. 3 again. We can extend the lengths of paths not by \(1, 2, \dots , h1\), but by . The result is shown on Fig. 4.
Note that the kth path is situated inside the diagram . This allows us to formulate the following corollary of Theorem 3.7. Its proof also follows from the Lindström–Gessel–Viennot lemma; the only difference concerns the weights which are added while extending paths.
Corollary 3.12
The following equality holds:
Remark 3.13
This corollary can also be obtained from Theorem 3.7 by purely algebraic manipulations with the determinant.
Remark 3.14
Note that the righthand side depends upon \(x_k\) for \(k>h+m\), while the lefthand side does not. This means that all these \(x_k\) cancel out while evaluating the determinant.
We will need this form of our determinantal identity while dealing with Schubert polynomials in the next section.
The Lindström–Gessel–Viennot lemma
In this subsection we prove the Lindström–Gessel–Viennot lemma on noncrossing paths. This material is by no means new: having appeared first in the preprint [6], now it can be found in many books and articles, for instance, in [16].
Let \({\Gamma }\) be an arbitrary oriented graph without oriented cycles, with weights assigned to its edges (we suppose that the weights are elements of a certain ring R). For a path P joining two vertices A and B, by the weight \(\mathop {\mathrm {wt}}P\) of P we mean the product of weights of all of its edges. As before, we define the partition function Z(A, B) to be the sum of weights of all paths joining A with B,
An htuple of paths \(P_1:A_1\rightarrow B_1\), \(\dots \), \(P_h:A_h\rightarrow B_h\) is said to be noncrossing if any two paths from it have no common points. An (ordered) set of h starting points and h endpoints \((A_1,\dots ,A_h)\) and \((B_1,\dots ,B_h)\) is said to be compatible if for any permutation \(\sigma \in S_h\) the existence of a noncrossing htuple of paths joining \(A_1\) with \(B_{\sigma (1)}\), \(\dots \), \(A_h\) with \(B_{\sigma (h)}\) implies that \(\sigma \) is the identity permutation.
For two htuples of starting points and endpoints, consider a weighted sum over all htuples of noncrossing paths \((P_1,\dots ,P_h)\),
Lemma 3.15
(Lindström–Gessel–Viennot) If \((A_1,\dots ,A_h)\) and \((B_1,\dots ,B_h)\) are compatible, the function \(Z_{\mathrm{nc}}((A_1,B_1),\dots ,(A_h,B_h))\) admits the following determinantal expression:
Proof
The determinant in the righthand side can be expressed as the sum over permutations
This is an alternating sum over all htuples of paths joining \(A_1,\dots ,A_h\) with \(B_1,\dots ,B_h\) in arbitrary order, taken with the appropriate sign. Such an htuple can be either noncrossing or have crossings; in the former case, it corresponds to the identity permutation. So we only need to show that the alternating sum over all htuples of paths with crossings is zero.
To do this, we construct a weightpreserving bijection on the set of htuples of paths with crossings. It is constructed as follows. We start with an arbitrary htuple of paths \((P_1,\dots ,P_h)\) with crossings, corresponding to a permutation \(\sigma \). Consider the path \(P_i\) with the smallest number of its starting point, such that it meets another path, say, \(P_j\). Let C be the first common point of these paths. Now let us “switch the tails” of these two paths: let \(\widetilde{P}_i\) be the path which coincides with \(P_i\) before C and with \(P_j\) after C. Likewise, let \(\widetilde{P}_j\) coincide with \(P_j\) before C and with \(P_i\) after C (see Fig. 5). We have obtained a new htuple of paths \((P_1,\dots ,\widetilde{P}_i,\dots ,\widetilde{P}_j,\dots ,P_h)\).
This map on the htuples of crossing paths is involutive. Obviously, \(\mathop {\mathrm {wt}}P_i\mathop {\mathrm {wt}}P_j=\mathop {\mathrm {wt}}\widetilde{P}_i\mathop {\mathrm {wt}}\widetilde{P}_j\), so it is weightpreserving. Finally, the permutations corresponding to \((P_1,\dots ,P_h)\) and \((P_1,\dots ,\widetilde{P}_i,\dots ,\widetilde{P}_j,\dots ,P_h)\) are obtained from each other by multiplication by a transposition, so they are counted in expression (1) with the opposite signs. This means that the total contribution of this pair of htuples of paths into (1) is zero, and the determinant is equal to the sum of weights over all htuples of noncrossing paths. The lemma is proven. \(\square \)
Schubert polynomials
Definition of Schubert polynomials
Schubert polynomials were introduced for studying the cohomology ring of a full flag variety. Their definition is due to A. Lascoux and M.P. Schützeberger [9]; implicitly it appeared in the paper [1] by J. Bernstein, I. Gelfand and S. Gelfand. Let us recall it.
Let \(S_n\) denote the symmetric group, i.e., the group of permutations of an nelement set \(\{1,\dots , n\}\). We will use the oneline notation for permutations; i.e., (1342) is a permutation taking 1 to 1, 2 to 3, 3 to 4 and 4 to 2. Let \(s_i=(1,2,\dots ,i+1,i,i+2,\dots ,n)\) denote the ith simple transposition, and \(w_0=(n,n1,\dots ,2,1)\) the element of maximal length. We mean by \(\ell (w)\) the length of the permutation w, i.e., the minimal number k of simple transpositions \(s_{i_1},\dots ,s_{i_k}\) such that \(w=s_{i_1}\!\dots s_{i_k}\).
For each \(i\leqslant n1\), define a divided difference operator as follows:
Definition 4.1
Let \(w\in S_n\) be a permutation. Consider a reduced decomposition of into a product of simple transpositions:
Then the Schubert polynomial corresponding to w is defined as follows:
(Note that the order of indices is reversed.)
Since the divided difference operators satisfy the braid relations:
the polynomial \({\mathfrak {S}}_w\) depends only on w and not on its decomposition into simple transpositions.
There are standard embeddings \(S_n\hookrightarrow S_m\) for \(n<m\); under this embedding, \(S_n\) permutes only the first n elements of \(\{1,\dots , m\}\). Sometimes we will treat permutations as elements of the direct limit \(S_\infty =\lim _{\rightarrow } S_n\). It can be shown (cf. [10]) that the Schubert polynomials are stable under these embeddings, so they can be regarded as elements of the polynomial ring in countably many variables.
Schubert and flagged Schur polynomials
Recall that a permutation \(w\in S_n\) is said to be vexillary if it is (2143)avoiding, i.e., there are no four numbers \(i<j<k<\ell \) such that \(w(j)<w(i)<w(\ell )<w(k)\). M. Wachs [17] shows that the Schubert polynomials of vexillary permutations can be obtained as flagged Schur polynomials for certain flags. This is done as follows.
We define the inversion sets of a permutation w as \({{\mathrm{I}}}_i(w)=\{j: j>i,\, w(j)<w(i)\}\). It can be shown that a permutation is vexillary iff its inversion sets form a chain (see [11] for details).
Let \(c_i=\#\, {{\mathrm{I}}}_i\). The sequence of these cardinalities \((c_1,\dots ,c_n)\) is called the Lehmer code of w. Denote by \(\lambda (w)\) the partition obtained by arranging \(c_1,\dots , c_n\) in the decreasing order. Likewise, let \(b(w)=(b_1(w)\leqslant \cdots \leqslant b_m(w))\) be the sequence of integers \(\min {{\mathrm{I}}}_i(w)1\) for all nonempty \({\mathrm{I}}_i(w)\), arranged in the increasing order.
Theorem 4.2
([17, Theorem 2.3]) If w is vexillary, then \({\mathfrak {S}}_w=s_{\lambda (w)}(b(w))\).
Dominant permutations
Definition 4.3
A permutation \(w\in S_n\) is called dominant if it is 132avoiding, i.e., there is no triple \(i<j<k\) such that \(w(i)<w(k)<w(j)\).
Clearly, if a permutation is 132avoiding, it is also 2143avoiding. So, all dominant permutations are vexillary.
It is well known (cf., for instance, [11]) that dominant permutations are characterized by the following property: their Schubert polynomial \({\mathfrak {S}}_w\) is a monomial in \(x_1,\dots ,x_{n1}\). Moreover, the Lehmer code \(\lambda (w)\) of such a permutation is a partition, and \({\mathfrak {S}}_w=\underline{(x_1,\dots ,x_{n1})}^{\lambda (w)}\).
For a dominant permutation w with its code \(\lambda =(\lambda _1,\dots ,\lambda _n)\), let \((i_1,\dots ,i_k)\) be the numbers of the rows of \(\lambda \) which are strictly shorter than the previous rows:
Then the flag of w is as follows:
with \(i_ji_{j1}\) entries equal to \(i_j\).
A flagged tableau of shape \(\lambda \) with this flag is unique: all its entries in the ith row are equal to i. So we can replace this flag by \((1,2,3,\dots )\). Hence, for a dominant permutation w its Schubert polynomial is a “0flagged Schur polynomial of shape \(\lambda \)”.
Shifts of permutations
Let \(w\in S_n\) be a permutation, and let h be a positive integer. The hshifted permutation is defined as
In different terms, we add h elements “in front of the set \(\{1,\dots ,n\}\)” acted on by permutations such that the action of on these new elements is trivial, and the action on the remaining elements coincides with the action of w. The following proposition is straightforward.
Proposition 4.4
Let w be a vexillary permutation with the Lehmer code \(\lambda (w)\) and flag \(b(w)=(b_1,b_2,\dots )\), and h a positive integer. Then

is vexillary;

.
This gives us a description of permutations whose Schubert polynomials are equal to hflagged Schur polynomials.
Remark 4.5
Of course, for a dominant permutation w, the permutations are not dominant anymore.
Extension of dominant permutations
In this subsection we define an operation on dominant permutations which corresponds to \((k,\ell )\)extension of their Young diagrams.
Let w be a dominant permutation. Define a (1, 0)extended permutation \(\widehat{w}[1,0]\) as follows:
This is shown on Fig. 6: the first entry is “doubled”, represented by a circle, is replaced by two entries. In this picture \(w=(42135)\), . Clearly, the code of the latter permutation is obtained from the code of the former permutation exactly by the operation of (1, 0)extension of Young diagrams in the sense of Definition 3.10. Moreover, is also dominant.
In a similar manner we can define the (0, 1)extension of a permutation. The only difference is that the element replaced by two elements is not 1, but the preimage of 1:
Figure 7 shows an example of the (0, 1)extension. In this example the original permutation is (42135), and the extended permutation is (532146).
A \((k,\ell )\)extension of a dominant permutation is obtained as a result of k successive applications of (1, 0)extension and \(\ell \) applications of (0, 1)extension (these operations commute). Again, the Young diagram corresponding to its code is obtained from the Young diagram of w by a \((k,\ell )\)extension.
Determinantal formula for Schubert polynomials of shifted dominant permutations
With the definitions from the two previous subsections, we can reformulate Corollary 3.12 for Schubert polynomials. The formula from that theorem will give the determinantal expression for the Schubert polynomial of an hshifted dominant permutation w in terms of 1shifted dominant permutations obtained from w by extensions.
Theorem 4.6
Let w be a dominant permutation, let \(\lambda =(\lambda _1,\lambda _2,\dots )\) be its Lehmer code. Then the Schubert polynomial of the hshifted permutation satisfies the following determinantal expression:
Schubert polynomials of Richardson permutations
In this section we discuss some corollaries of Theorem 4.6 and formulate several conjectures concerning Schubert polynomials.
Catalan numbers
Fix some notation. Let \(w_0(n)=(n,n1,\dots ,2,1)\in S_n\) be the element of maximal length. It is dominant, and the corresponding Young diagram is the staircase shape \({\Lambda }_n=(n1,n2,\dots ,2,1)\).
Recall that the nth Catalan number is equal to the number of subdiagrams of the staircase shape \({\Lambda }_n\). This number has several different qgeneralizations. Let \(\mathop {\mathrm {Cat}}_q(n)\) denote the nth Carlitz–Riordan qCatalan number, obtained as the weighted sum of all such subdiagrams, where a subdiagram \(\mu \) has weight \(q^{n(n1)/2\mu }\),
Here is a first observation, made by Alex Woo [18].
Theorem 5.1
([18]) Let . Then the principal specialization of \({\mathfrak {S}}_w\) equals the nth Carlitz–Riordan qCatalan number times a power of q,
Corollary 5.2
The value of \({\mathfrak {S}}_w\) at the identity equals the nth Catalan number,
In this case \({\mathfrak {S}}_w=s_{{\Lambda }_n}^{(1)}\) is the 1flagged polynomial of the staircase partition.
Catalan–Hankel determinants
Consider the same permutation \(w_0(n)\) shifted by h instead of 1. Theorem 4.6 expresses \({\mathfrak {S}}_{1^h\times w_0}\) via \({\mathfrak {S}}_{1\times \widehat{w_0(n)}[k,\ell ]}\).
Note that , since the extension of a staircase diagram is again a staircase diagram. This means that all permutations occurring in the determinantal expression are \({\mathfrak {S}}_{1\times w_0(m)}\) for \(m=n,\dots ,n+2h2\).
Specializing this identity at 1 and using Corollary 5.2, we get the following proposition.
Proposition 5.3
The following identity holds:
This determinant is a Hankel determinant of the sequence of Catalan numbers; we will later refer to it as a Catalan–Hankel determinant.
Remark 5.4
Since \({\mathfrak {S}}_{1^h\times w_0(n)}\) is the flagged Schur polynomial \(s_{{\Lambda }_n}^{(h)}\), we see from the definition of a flagged Schur polynomial that its value at 1 is equal to the number of plane partitions with parts at most h and shape \({\Lambda }_n\), i.e., threedimensional Young diagrams inside a prism with base \({\Lambda }_n\) and height h. This observation appeared in [4, Section 1.2].
It is also possible to establish an explicit bijection between the monomials in \({\mathfrak {S}}_{1^h\times w_0(h)}\), indexed by the socalled “pipe dreams” (see [3]), and such plane partitions. See [13] for the details.
Richardson elements in the symmetric group
Definition 5.5
Let \(i_1<i_2<i_3<\dots <i_k=n\). Permutation \((i_1,\,i_11,\) \(\dots ,\,2,\,1,\,i_2,\,i_21,\,\dots ,\,i_1+1,\, i_3,\,\dots ,\,i_2+1,\,\dots )\in S_n\) is called a Richardson permutation.
In different terms, a Richardson permutation is equal to the product of several permutations of type with nonintersecting supports. In particular, \(w_0(n)\) and are Richardson permutations.
Richardson elements have a nice description in terms of algebraic groups. For a parabolic subgroup consider its Levi decomposition . Then the longest element in the Weyl group \(W_L\) of L is a Richardson element in the Weyl group \(W\cong S_n\) of .
Richardson permutations are not vexillary (unless they are shifted staircase permutations). However, they are obtained as products of several shifted staircase permutations with pairwise disjoint supports. So the Schubert polynomial of such a permutation is equal to the product of the Schubert polynomials of its factors.
Hence, Theorem 4.6 shows that the Schubert polynomials of Richardson permutations can be presented as products of several determinants, and their specialization at 1 is equal to the product of several Catalan–Hankel determinants from Proposition 5.3.
The study of Richardson permutations is motivated by the following question.
Question 5.6
For a given n, find the permutations \(w\in S_n\) such that \({\mathfrak {S}}_{w}(1,\dots ,1)\) is as large as possible.
This question has a geometric interpretation: A. Knutson and E. Miller [7] showed that the value \({\mathfrak {S}}_w(1,\dots ,1)\) is equal to the degree of the matrix Schubert variety corresponding to w. If w satisfies the condition that, for every (i, j) with \(i+j>n\), either \((w_0w)^{1}(i)\leqslant j\) or \(w_0w(j)\leqslant i\), this degree equals the multiplicity of the singular point \(X_e\subset X_w\), where is the Schubert variety in a full flag variety corresponding to w. This multiplicity is a measure of “how singular” \(X_w\) is; we are then trying to find the Schubert variety with the “worst” singularities (in the aforementioned sense).
These values were computed for all permutations with \(n\leqslant 10\). It turned out that the maximum is always attained at a Richardson permutation. This allows us to formulate the following conjecture.
Conjecture 5.7
For a given n, the permutation \(w\in S_n\) with the largest value of \({\mathfrak {S}}_w(1,\dots ,1)\) is Richardson.
In Table 1 we give the list of permutations w corresponding to the largest value of \({\mathfrak {S}}_w(1,\dots ,1)\) and the values themselves for \(n\leqslant 10\) (for \(n=5,6\), there is more than one such permutation). This table was computed by I. Kochulin [8].
However, this conjecture remains a purely experimental observation; so far we cannot see any reason why the permutation producing the largest value of its Schubert polynomial at 1 has to be Richardson.
References
Bernšteĭn, I.N., Gel’fand, I.M., Gel’fand, S.I.: Schubert cells, and the cohomology of the spaces \(G/P\). Uspehi Mat. Nauk 28(3)(171), 3–26 (1973) (in Russian)
Chen, W.Y.C., Li, B., Louck, J.D.: The flagged double Schur function. J. Algebraic Combin. 15(1), 7–26 (2002)
Fomin, S., Kirillov, A.N.: The YangBaxter equation, symmetric functions, and Schubert polynomials. Discrete Math. 153(1–3), 123–143 (1996)
Fomin, S., Kirillov, A.N.: Reduced words and plane partitions. J. Algebraic Combin. 6(4), 311–319 (1997)
Gessel, I.M.: Determinants and plane partitions. Unpublished manuscript
Gessel, I.M., Viennot, X.G.: Determinants, paths and plane partitions (1989). http://people.brandeis.edu/~gessel/homepage/papers/pp.pdf
Knutson, A., Miller, E.: Gröbner geometry of Schubert polynomials. Ann. Math. 161(3), 1245–1318 (2005)
Kochulin, I.: RCgraphs of Symmetric Groups. Master’s thesis, National Research University Higher School of Economics, Moscow (2012)
Lascoux, A., Schützenberger, M.P.: Polynômes de Schubert. C. R. Acad. Sci. Paris Sér. I Math. 294(13), 447–450 (1982)
Macdonald, I.G.: Notes on Schubert Polynomials. Publications du Laboratoire de Combinatoire et d’Informatique Mathématique, vol. 6. Université du Québec à Montréal, Montréal (1991)
Manivel, L.: Fonctions Symétriques, Polynômes de Schubert et Lieux de Dégénérescence. Cours Spécialisés, vol. 3. Société Mathématique de France, Paris (1998)
Schur, I.: Gesammelte Abhandlungen. I. Springer, Berlin (1973)
Serrano, L., Stump, C.: Maximal fillings of moon polyominoes, simplicial complexes, and Schubert polynomials. Electron. J. Combin. 19(1), N 16 (2012)
Specht, W.: Die irreduziblen Darstellungen der symmetrischen Gruppe. Math. Z. 39, 696–711 (1935)
Stanley, R.P.: Enumerative Combinatorics. II. Cambridge Studies in Advanced Mathematics, vol. 62. Cambridge University Press, Cambridge (1999)
Stanley, R.P.: Enumerative Combinatorics I. Cambridge Studies in Advanced Mathematics, vol. 49, 2nd edn. Cambridge University Press, Cambridge (2012)
Wachs, M.L.: Flagged Schur functions, Schubert polynomials, and symmetrizing operators. J. Combin. Theory Ser. A 40(2), 276–289 (1985)
Woo, A.: Catalan numbers and Schubert polynomials for \(w=1(n+1)\cdots 2\) (2004). arXiv:math/0407160
Acknowledgments
This paper was completed during the second author’s visit to the University of Warwick; he is grateful to this institution and to Professor Miles Reid for their warm hospitality.
Author information
Authors and Affiliations
Corresponding author
Additional information
Evgeny Smirnov was partially supported by the Russian Science Foundation grant 142100053 within AG Laboratory NRUHSE.
Rights and permissions
About this article
Cite this article
Merzon, G., Smirnov, E. Determinantal identities for flagged Schur and Schubert polynomials. European Journal of Mathematics 2, 227–245 (2016). https://doi.org/10.1007/s4087901500789
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s4087901500789
Keywords
 Schur polynomial
 Schubert polynomial
 Young tableau
 Plane partition
Mathematics Subject Classification
 05E05
 14N15