# Properties of the nonsymmetric Robinson–Schensted–Knuth algorithm

- 199 Downloads

## Abstract

We introduce a generalization of the Robinson–Schensted–Knuth insertion algorithm for semi-standard augmented fillings whose basement is an arbitrary permutation *σ*∈*S*_{n}. If *σ* is the identity, then our insertion algorithm reduces to the insertion algorithm introduced by the second author (Sémin. Lothar. Comb. 57:B57e, 2006) for semi-standard augmented fillings and if *σ* is the reverse of the identity, then our insertion algorithm reduces to the original Robinson–Schensted–Knuth row insertion algorithm. We use our generalized insertion algorithm to obtain new decompositions of the Schur functions into nonsymmetric elements called generalized Demazure atoms (which become Demazure atoms when *σ* is the identity). Other applications include Pieri rules for multiplying a generalized Demazure atom by a complete homogeneous symmetric function or an elementary symmetric function, a generalization of Knuth’s correspondence between matrices of non-negative integers and pairs of tableaux, and a version of evacuation for composition tableaux whose basement is an arbitrary permutation *σ*.

### Keywords

Symmetric functions Permutations Nonsymmetric Macdonald polynomials Demazure atoms Permuted basement fillings## 1 Introduction

*γ*=(

*γ*

_{1},

*γ*

_{2},…,

*γ*

_{n}) is a weak composition of

*m*into

*n*parts if each

*γ*

_{i}∈ℕ and \(\sum_{i=1}^{n} \gamma_{i} =m\). Letting |

*γ*|=∑

_{i}

*γ*

_{i}, the (

*column*)

*diagram of*

*γ*is the figure

*dg*′(

*γ*) consisting of |

*γ*|

*cells*arranged into columns so that the

*i*th column contains

*γ*

_{i}cells. For example, the diagram of

*γ*=(2,0,1,0,3) is pictured in Fig. 1. The

*augmented diagram of*

*γ*, denoted by \(\widehat{dg}(\gamma)\), consists of the diagram of

*γ*together with an extra row of

*n*cells attached below. These extra cells are referred to as the

*basement*of the augmented diagram. We let

*λ*(

*γ*) be the partition that results by taking the weakly decreasing rearrangement of the parts of

*γ*. Thus if

*γ*=(2,0,1,0,3), then

*λ*(

*γ*)=(3,2,1,0,0).

*P*

_{λ}(

*x*

_{1},

*x*

_{2},…,

*x*

_{n};

*q*,

*t*), which have important applications to a variety of areas. In [6], Macdonald showed that many of the properties of the

*P*

_{λ}, such as satisfying a multivariate orthogonality condition, are shared by a family of nonsymmetric polynomials

*E*

_{γ}(

*x*

_{1},…,

*x*

_{n};

*q*,

*t*), where

*γ*is a weak composition with

*n*parts. Haglund, Haiman and Loehr [1] obtained a combinatorial formula for

*E*

_{γ}(

*x*

_{1},…,

*x*

_{n};

*q*,

*t*) in terms of fillings of \(\widehat {dg}(\gamma)\) by positive integers satisfying certain constraints. It will be simpler for us to phrase things in terms of a transformed version of the

*E*

_{γ}studied by Marshall [8] which we denote by \({\widehat{E}}_{\gamma}(x_{1},\ldots ,x_{n};q,t)\). The \(\widehat{E}_{\gamma}\) can be obtained from the

*E*

_{γ}by sending

*q*→1/

*q*,

*t*→1/

*t*, reversing the

*x*-variables, and reversing the parts of

*γ*. The corresponding combinatorial expression for \(\widehat{E}_{\gamma }(x_{1},\ldots,x_{n};0;0)\) from [1] involves what the second author [9, 10] later called semi-standard augmented fillings. It was previously known that \(\widehat{E}_{\gamma}(x_{1},\ldots,x_{n}; 0, 0)\) (hereafter denoted more simply by \(\widehat{E}_{\gamma}(x_{1},\ldots,x_{n})\)), equals the “standard bases” of Lascoux and Schützenberger [5], which are also referred to as Demazure atoms. The second author introduced a generalization of the RSK insertion algorithm involving semi-standard augmented fillings, and used this to give combinatorial proofs of several results involving Demazure atoms. For example, this generalized RSK insertion algorithm gives a bijective proof that for any partition

*β*,

This extended Robinson–Schensted–Knuth insertion algorithm is also instrumental in work of Haglund, Luoto, Mason, and van Willigenburg, who developed the theory of a new basis for the ring of quasisymmetric functions called quasisymmetric Schur functions [3, 4]. In particular, these authors use it in proving a generalization of the Littlewood–Richardson rule, where the product of a Schur function and a Demazure atom (Demazure character, quasisymmetric Schur function) is expanded in terms of Demazure atoms (Demazure characters, quasisymmetric Schur functions), respectively, with positive coefficients.

*ϵ*

_{n}denote the identity 1 2 ⋯

*n*in

*S*

_{n}and \(\bar{\epsilon}_{n}\) the reverse of the identity

*n*

*n*−1 ⋯ 1. In [9, 10] and in [3, 4], the basements of the diagrams \(\widehat{dg}(\gamma)\) are always filled by either

*ϵ*

_{n}(i.e.,

*i*is in the

*i*th column of the basement), or by \(\bar{\epsilon}_{n}\). In this article, we show that many of the nice properties of the extended RSK insertion algorithm hold with the basement consisting of an arbitrary permutation

*σ*∈

*S*

_{n}. In particular, we define a weight preserving bijection which shows

*γ*such that

*λ*(

*γ*)=

*β*and

*γ*

_{i}≥

*γ*

_{j}whenever

*i*<

*j*and

*σ*

_{i}>

*σ*

_{j}. Here \(\widehat{E}^{\sigma}_{\gamma}(x_{1},\ldots,x_{n})\) is the version of \(\widehat{E}_{\gamma}(x_{1},\ldots,x_{n})\) with basement

*σ*which we call a generalized Demazure atom. In the special case when \(\sigma= \bar{\epsilon}_{n}\), there is only one term in the sum above so that \(s_{\beta} = E^{\bar{\epsilon}_{n}}_{\beta}\), while if

*σ*is

*ϵ*

_{n}then (2) reduces to (1).

*ϵ*

_{n}by

*σ*

_{1}

*σ*

_{2}⋯

*σ*

_{n}, and keeping other aspects of the formula the same. Then if

*i*+1 occurs to the left of

*i*in the basement

*σ*

_{1}

*σ*

_{2}⋯

*σ*

_{n}, we have Here

*A*equals one if the height of the column of \(\widehat{dg}(\gamma )\) above

*i*+1 in the basement is greater than or equal to the height of the column above

*i*in the basement, and equals zero otherwise. Also,

*σ*′ is the permutation obtained by interchanging

*i*and

*i*+1 in

*σ*. The

*T*

_{i}are generators for the affine Hecke algebra which act on monomials in the

*X*variables by with \(x^{\alpha_{i}} = x_{i}/x_{i+1}\). See [1] for a more detailed description of the

*T*

_{i}and their relevance to nonsymmetric Macdonald polynomials. Our \(\widehat{E}^{\sigma}_{\gamma}(x_{1}, \ldots, x_{n})\) can be obtained by setting

*q*=

*t*=0 in \(\widehat{E}^{\sigma}_{\gamma}(x_{1}, \ldots, x_{n};q,t)\), and hence are a natural generalization of the \(\widehat{E}_{\gamma}(x_{1}, \ldots, x_{n})\) to investigate. If we set

*q*=

*t*=0 in the Hecke operator

*T*

_{i}, it reduces to a divided difference operator similar to those appearing in the definition of Schubert polynomials. By (3), \(\widehat{E}^{\sigma}_{\gamma}(x_{1}, \ldots, x_{n})\) can be expressed (up to a power of

*t*) as a series of the divided difference operators applied to the Demazure character \(\widehat{E}^{\bar{\epsilon}_{n}}_{\gamma}(x_{1}, \ldots, x_{n})\).

As with the extended insertion algorithm, we shall see that our insertion algorithm with general basements also commutes in a natural way with the RSK insertion algorithm. This useful fact will allow us to extend the results of the second author to our more general setup. Moreover, we shall give a precise characterization of how the results of our insertion algorithm vary as the basement *σ* varies. If \(\sigma= {\bar {\epsilon}_{n}}\) our algorithm becomes essentially equivalent to the ordinary RSK row insertion algorithm, while if *σ*=*ϵ*_{n}, it reduces to the extended insertion algorithm.

The outline of this paper is as follows. In Sect. 2, we formally define the objects we will be working with, namely permuted basement semi-standard augmented fillings relative to a permutation *σ* (\({\operatorname {PBF}}\)s). In Sects. 3 and 4, we describe our insertion algorithm for \({\operatorname {PBF}}\)s and derive its general properties. In Sect. 5, we use it to prove analogues of the Pieri rules for the product of a homogeneous symmetric function *h*_{n}(*x*_{1},…,*x*_{n}) times an \(\widehat{E}^{\sigma }_{\gamma}(x_{1}, \ldots, x_{n})\) and the product of an elementary symmetric function *e*_{n}(*x*_{1},…,*x*_{n}) times an \(\widehat{E}^{\sigma }_{\gamma}(x_{1}, \ldots, x_{n})\). In Sect. 6, we define a generalization of the RSK correspondence between ℕ-valued matrices and pairs of column strict tableaux for permuted basement fillings and prove several of its basic properties. Finally, in Sect. 7, we study the analogue of evacuation for \({\operatorname {PBF}}\)s.

## 2 Permuted basement semi-standard augmented fillings

*n*is fixed throughout, while

*γ*will always denote a weak composition into

*n*parts and

*σ*a permutation in

*S*

_{n}. We let (

*i*,

*j*) denote the cell in the

*i*th column, reading from left to right, and the

*j*th row, reading from bottom to top, of \(\widehat{dg}(\gamma)\). The basement cells of \(\widehat{dg}(\gamma)\) are considered to be in row 0 so that \(\widehat{dg}(\gamma)=dg^{\prime}(\gamma) \cup\{ (i,0) : 1 \le i \le n \}\). The

*reading order*of the cells of \(\widehat{dg}(\gamma)\) is obtained by reading the cells in rows from left to right, beginning at the highest row and reading from top to bottom. Thus a cell

*a*=(

*i*,

*j*) is less than a cell

*b*=(

*i*′,

*j*′) in the reading order if either

*j*>

*j*′ or

*j*=

*j*′ and

*i*<

*i*′. For example, if

*γ*=(0,2,0,3,1,2,0,0,1), then \(\widehat{dg}(\gamma)\) is pictured in Fig. 2 where we have placed the number

*i*in the

*i*th cell in reading order. An

*augmented filling*,

*F*, of an augmented diagram \(\widehat {dg}(\gamma)\) is a function \(F: \widehat{dg}(\gamma) \rightarrow\mathbb{P}\), which we picture as an assignment of positive integers to the cells of \(\widehat{dg}(\gamma)\). We let

*F*(

*i*,

*j*) denote the entry in cell (

*i*,

*j*) of

*F*. The reading word of

*F*,

*read*(

*F*), is obtained by recording the entries of

*F*in the reading order of

*dg*′(

*γ*). The

*content*of

*F*is the multiset of entries which appear in the filling

*F*. Throughout this article, we will only be interested in fillings

*F*such that entries in each column are weakly increasing reading from top to bottom and the basement entries form a permutation in the symmetric group

*S*

_{n}.

*A*and

*B*triples as in [9]. A

*type A triple*in an augmented diagram of shape

*γ*is a set of three cells

*a*,

*b*,

*c*of the form (

*i*,

*k*),(

*j*,

*k*),(

*i*,

*k*−1) for some pair of columns

*i*<

*j*of the diagram and some row

*k*>0, where

*γ*

_{i}≥

*γ*

_{j}. A

*type B triple*is a set of three cells

*a*,

*b*,

*c*of the form (

*j*,

*k*+1),(

*i*,

*k*),(

*j*,

*k*) for some pair of columns

*i*<

*j*of the diagram and some row

*k*≥0, where

*γ*

_{i}<

*γ*

_{j}. Note that basement cells can be elements of triples. As noted above, in this article our fillings

*F*have weakly increasing column entries reading from top to bottom, so we always have the entry values satisfying

*F*(

*a*)≤

*F*(

*c*). We say that a triple of either type is an

*inversion*triple if the relative order of the entries is either

*F*(

*b*)<

*F*(

*a*)≤

*F*(

*c*) or

*F*(

*a*)≤

*F*(

*c*)<

*F*(

*b*). Otherwise we say that the triple is a

*coinversion*triple, i.e., if

*F*(

*a*)≤

*F*(

*b*)≤

*F*(

*c*). Figure 3 pictures type

*A*and

*B*triples.

A *semi-standard augmented filling* is a filling of an augmented diagram with positive integer entries so that (i) the column entries are weakly increasing from top to bottom, (ii) the basement entries form a permutation of 1,2,…,*n* where *n* is the number of cells in the basement, and (iii) every Type *A* or *B* triple is an inversion triple. We say that cells *c*_{1}=(*x*_{1},*y*_{1}) and *c*_{2}=(*x*_{2},*y*_{2}) are *attacking* if either *c*_{1} and *c*_{2} lie in the same row, i.e., *y*_{1}=*y*_{2}, or if *c*_{1} lies strictly to the left and one row below *c*_{2}, i.e., if *x*_{1}<*x*_{2} and *y*_{2}=*y*_{1}+1. We say that filling *F* is *non-attacking* if *F*(*c*_{1})≠*F*(*c*_{2}) whenever *c*_{1} and *c*_{2} are attacking. It is easy to see from our definition of inversion triples that a semi-standard augmented filling *F* must be non-attacking. A superscript *σ* on a filling *F*, as in *F*^{σ}, means the basement entries form the permutation *σ*.

*F*

^{σ}is a

*permuted basement semi-standard augmented filling*(\(\operatorname {PBF}\)) of shape

*γ*with basement permutation

*σ*if

- (I)
*F*^{σ}is a semi-standard augmented filling of \(\widehat{dg}(\gamma)\), - (II)
*F*^{σ}((*i*,0))=*σ*_{i}for*i*=1,…,*n*, and - (III)
for all cells

*a*=(*i*_{2},*j*),*b*=(*i*_{1},*j*−1) such that*i*_{1}<*i*_{2}and \(\gamma_{i_{1}} < \gamma_{i_{2}}\), we have*F*^{σ}(*b*)<*F*^{σ}(*a*).

*B*-

*increasing*condition, as pictured in Fig. 4.

We note that the fact that a \(\operatorname {PBF}\)*F*^{σ} has weakly increasing columns, reading from top to bottom, and satisfies the *B*-increasing condition automatically implies that every *B*-triple in *F*^{σ} is an inversion triple. That is, suppose that *γ*_{i}<*γ*_{j} where *i*<*j* and *a*=(*j*,*k*+1), *b*=(*i*,*k*) and *c*=(*j*,*k*) is *B*-triple. Then *F*^{σ}(*b*)<*F*^{σ}(*a*)≤*F*^{σ}(*c*) since the *B*-increasing condition forces *F*^{σ}(*b*)<*F*^{σ}(*a*) and the weakly increasing column condition forces *F*^{σ}(*a*)≤*F*^{σ}(*c*). Thus {*a*,*b*,*c*} is an inversion triple.

*F*

^{σ}of shape

*γ*, we define the weight of

*F*

^{σ},

*W*(

*F*

^{σ}), to be

*F*

^{σ}of shape

*γ*with basement

*σ*. We then define

The following fact about \(\operatorname {PBF}\)s will be used frequently in the sequel.

### Lemma 1

*Let*

*F*

^{σ}

*be a*\(\operatorname {PBF}\)

*of shape*

*γ*

*and assume that*

*i*<

*m*.

- (i)
*Suppose that**F*^{σ}(*i*,*j*)<*F*^{σ}(*m*,*j*)*for some**j*>0.*Then**F*^{σ}(*i*,*j*−1)<*F*^{σ}(*m*,*j*).*Moreover*,*for all*0≤*k*<*j*,*F*^{σ}(*i*,*k*)<*F*^{σ}(*m*,*k*+1)≤*F*^{σ}(*m*,*k*). - (ii)
*Suppose that**F*^{σ}(*i*,*j*)>*F*^{σ}(*m*,*j*)*for some**j*≥0.*Then**γ*_{i}≥*γ*_{m}*and*,*for all**j*≤*k*≤*γ*_{m},*F*^{σ}(*i*,*k*)>*F*^{σ}(*m*,*k*).

### Proof

For (i), we consider two cases. First, if *γ*_{i}<*γ*_{m}, then the *B*-increasing condition forces *F*^{σ}(*i*,*j*−1)<*F*^{σ}(*m*,*j*). Second, if *γ*_{i}≥*γ*_{m}, then consider the *A*-triple *a*=(*i*,*j*), *b*=(*m*,*j*), and *c*=(*i*,*j*−1). As we are assuming that *F*^{σ}(*a*)<*F*^{σ}(*b*), it must be the case that *F*^{σ}(*i*,*j*−1)=*F*^{σ}(*c*)<*F*^{σ}(*b*)=*F*^{σ}(*m*,*j*) since otherwise {*a*,*b*,*c*} would be coinversion triple in *F*^{σ}. Thus it is always the case that *F*^{σ}(*i*,*j*−1)<*F*^{σ}(*m*,*j*). But then we know that *F*^{σ}(*i*,*j*−1)<*F*^{σ}(*m*,*i*)≤*F*^{σ}(*m*,*j*−1) so that *F*^{σ}(*i*,*j*−1)<*F*^{σ}(*m*,*j*−1). Thus we can repeat our argument to show that for all 0≤*k*<*j*, *F*^{σ}(*i*,*k*)<*F*^{σ}(*m*,*k*+1)≤*F*^{σ}(*m*,*k*).

*F*

^{σ}(

*i*,

*j*)>

*F*

^{σ}(

*m*,

*j*). Then we claim that it cannot be the case that

*γ*

_{i}<

*γ*

_{m}since otherwise (

*m*,

*j*+1) must be a cell in

*F*

^{σ}which would mean that

*F*

^{σ}(

*i*,

*j*)>

*F*

^{σ}(

*m*,

*j*)≥

*F*

^{σ}(

*m*,

*j*+1). But then

*a*=(

*m*,

*j*+1) and

*b*=(

*i*,

*j*) would violate the

*B*-increasing condition. Thus it must be the case that

*γ*

_{i}≥

*γ*

_{m}. We claim that it also must be the case that

*F*

^{σ}(

*i*,

*k*)>

*F*

^{σ}(

*m*,

*k*) for all

*j*<

*k*≤

*γ*

_{m}. If this is not the case, then let

*k*be the smallest

*ℓ*such that

*ℓ*≥

*j*and

*F*

^{σ}(

*i*,

*ℓ*)≤

*F*

^{σ}(

*m*,

*ℓ*). This implies the triple {(

*i*,

*k*),(

*m*,

*k*),(

*i*,

*k*−1)} is a type

*A*coinversion triple since

*F*

^{σ}has no type

*A*coinversion triples, there can be no such

*k*. □

Note that part (ii) of Lemma 1 tells us that the basement permutation *σ* restricts the possible shapes of a \(\operatorname {PBF}\)*F*^{σ} with basement *σ*. That is, if *σ*_{i}>*σ*_{m}, then it must be the case that height of column *i* in *F*^{σ} is greater than or equal to the height of column *m* in *F*^{σ}.

We end this section by considering the two special cases of \(\operatorname {PBF}\)s where the basement is either the identity or the reverse of the identity. In the special case where the basement permutation *σ*=*ϵ*_{n}, a \(\operatorname {PBF}\) is a semi-standard augmented filling as defined in [9]. Next consider the case where \(F^{\bar{\epsilon}_{n}}\) is a \(\operatorname {PBF}\) of shape *γ*=(*γ*_{1},…,*γ*_{n}) with basement \(\bar{\epsilon}_{n}\). In that case, Lemma 1 implies that *γ*_{1}≥*γ*_{2}≥⋯≥*γ*_{n} and that *F*^{σ} must be strictly decreasing in rows. Since the entries of \(F^{\bar{\epsilon}_{n}}\) must weakly decrease in columns reading from bottom to top, we see that \(F^{\bar{\epsilon}_{n}}\) is what could be called a reverse row strict tableau with basement \(\bar{\epsilon}_{n}\) attached. It follows that for *γ* a partition, \(\widehat{E}_{\gamma}^{\bar{\epsilon }_{n}}(x_{1},x_{2}, \ldots,x_{n})\) is equal to the Schur function *s*_{γ}(*x*_{1},*x*_{2},…,*x*_{n}).

## 3 An analogue of Schensted insertion

In [9], the second author defined a procedure *k*→*F* to insert a positive integer *k* into a semi-skyline augmented filling, which is a \(\operatorname {PBF}\) with basement permutation equal to the identity. In this section, we shall describe an extension of this insertion procedure which inserts a positive integer into a \(\operatorname {PBF}\) with an arbitrary basement permutation.

Let *F*^{σ} be a \(\operatorname {PBF}\) with basement permutation *σ*∈*S*_{n}. We shall define a procedure *k*→*F*^{σ} to insert a positive integer *k* into *F*^{σ}. Let \(\bar{F}^{\sigma }\) be the extension of *F*^{σ} which first extends the basement permutation *σ* by adding *j* in cell (*j*,0) for *n*<*j*≤*k* and then adds a cell which contains a 0 on top of each column. Let (*x*_{1},*y*_{1}),(*x*_{2},*y*_{2}),… be the cells of this extended diagram listed in reading order. Formally, we shall define the insertion procedure of *k*→(*x*_{1},*y*_{1}),(*x*_{2},*y*_{2}),… of *k* into the sequence of cells (*x*_{1},*y*_{1}),(*x*_{2},*y*_{2}),….

Let *k*_{0}=*k* and look for the first *i* such that \(\bar{F}^{\sigma }(x_{i},y_{i}) < k_{0} \leq\bar{F}^{\sigma }(x_{i},y_{i}-1)\). Then there are two cases.

**Case 1**. If \(\bar{F}^{\sigma }(x_{i},y_{i}) =0\), then place *k*_{0} in cell (*x*_{i},*y*_{i}) and terminate the procedure.

**Case 2**. If \(\bar{F}^{\sigma }(x_{i},y_{i}) \neq0\), then place *k*_{0} in cell (*x*_{i},*y*_{i}), set \(k_{0} := \bar{F}^{\sigma }(x_{i},y_{i})\) and repeat the procedure by inserting *k*_{0} into the sequence of cells (*x*_{i+1},*y*_{i+1}),(*x*_{i+2},*y*_{i+2}),…. In such a situation, we say that \(\bar{F}^{\sigma }(x_{i},y_{i})\) was *bumped* in the insertion *k*→*F*^{σ}.

The output of *k*→*F*^{σ} is the filling that keeps only the cells that are filled with positive integers. That is, we remove any cells of \(\bar{F}^{\sigma }\) that still have a 0 in them.

*k*→

*F*

^{σ}plus the final cell which is added when the procedure is terminated will be called the

*bumping path*of the insertion. For example, Fig. 5 shows an extended diagram of a \(\operatorname {PBF}\) with basement permutation equal to 6 1 3 4 2 5. If we insert 5 into this \(\operatorname {PBF}\), then it is easy to see that the first element bumped is the 4 in column 1. Thus that 4 will be replaced by 5 and we will insert 4 into the remaining sequence of cells. The first element that 4 can bump is the 2 in column 4. Thus that 4 will replace the 2 in column 4 and 2 will be inserted in the remaining cells. But then that 2 will bump the 0 in column 5 so that the procedure will terminate. Thus the circled elements in Fig. 5 correspond to the bumping path of this insertion. Clearly, the entries of \(\bar{F}^{\sigma }\) in the bumping path must strictly decrease as we proceed in reading order.

*k*→

*F*

^{σ}, we will assume that

*σ*∈

*S*

_{n}where

*n*is greater than or equal to

*k*and all of the entries in

*F*

^{σ}.

The following lemmas are needed in order to prove that the insertion procedure terminates and the result is a \(\operatorname {PBF}\).

### Lemma 2

*Let**c*_{1}=(*i*_{1},*j*_{1}) *and**c*_{2}=(*i*_{2},*j*_{2}) *be two cells in a*\(\operatorname {PBF}\)*F*^{σ}*such that**F*^{σ}(*c*_{1})=*F*^{σ}(*c*_{2})=*a*, *assume**c*_{1}*appears before**c*_{2}*in reading order*, *and no cell between**c*_{1}*and**c*_{2}*in reading order contains the entry**a*. *Let*\(c_{1}'=(i_{1}',j_{1}')\)*and*\(c_{2}'=(i_{2}',j_{2}')\)*be the cells in**k*→*F*^{σ}*containing the entries from**c*_{1}*and**c*_{2}, *respectively*. *Then*\(j_{1}' > j_{2}'\).

### Proof

Consider the cell \(\underline{c_{1}}=(i_{1},j_{1}-1)\) immediately below *c*_{1} in the diagram *F*^{σ}. Note that *c*_{1} attacks all cells of *F*^{σ} to its right that lie in same row as well as all cells to its left that lie one row below the row of *c*_{1}. Since entries in cells which are attacked by *c*_{1} must be different from *F*^{σ}(*c*_{1}), it follows that *c*_{2} must appear weakly after \(\underline{c_{1}}\) in reading order. If \(c_{2} = \underline{c_{1}}=(i_{1},j_{1}-1)\), then the entry in cell *c*_{1} cannot be bumped because that would require *F*^{σ}(*i*_{1},*j*_{1})<*k*_{0}≤*F*^{σ}(*i*_{1},*j*_{1}−1). Thus either *c*_{2} is not bumped in which case the lemma automatically holds or *c*_{2} is bumped in which case its entry ends up in a cell which is later in reading order so that \(j_{1} =j_{1}' > j_{1}-1 \geq j_{2}'\).

Thus we may assume that \(F^{\sigma }(\underline{c_{1}}) > F^{\sigma }(c_{1})\) and that *c*_{2} follows \(\underline{c_{1}}\) in reading order. This means that the element \(\overline{c_{2}} = (i_{2},j_{2}+1)\) which lies immediately above *c*_{2} follows *c*_{1} in reading order and the entry in cell \(\overline{c_{2}}\) must be strictly less than *a* by our choice of *c*_{2}. If the entry in *c*_{1} is not bumped, then again we can conclude as above that the entry in *c*_{2} will end up in a cell which follows \(\underline {c_{1}}\) in reading order so that again \(j_{1} =j_{1}' > j_{1}-1 \geq j_{2}'\). Finally, suppose that the entry *a* in cell *c*_{1} is bumped. Since \(F^{\sigma }(\,\overline{c_{2}}\,) < a = F^{\sigma }(c_{2})\), it follows that \(F^{\sigma }(\,\overline{c_{2}}\,)\) is a candidate to be bumped by *a*. Thus the *a* that was bumped out of cell *c*_{1} must end up in a cell which weakly precedes \(\overline{c_{2}}\) in reading order and hence it ends up in a row which is higher than the row of *c*_{2}. Since the elements in a bumping path strictly decrease, the *a* in cell *c*_{2} cannot be part of the bumping path. Thus the lemma holds. □

### Lemma 3

*Suppose that**F*^{σ}*is a*\(\operatorname {PBF}\)*and**k**is a positive integer*. *Then every type A triple in**k*→*F*^{σ}*is an inversion triple*.

### Proof

Suppose that *F*^{σ} is of shape *γ*=(*γ*_{1},…,*γ*_{n}) where *n*≥*k*. Consider an arbitrary type *A* triple {*a*=(*x*_{1},*y*_{1}),*b*=(*x*_{2},*y*_{1}),*c*=(*x*_{1},*y*_{1}−1)} in \(\tilde{F}^{\sigma} : = k \rightarrow F^{\sigma}\). Suppose for a contradiction that {*a*,*b*,*c*} is a coinversion triple so that \(\mbox {$\tilde {F}^{\sigma }$}(a) \le \mbox {$\tilde {F}^{\sigma }$}(b) \le \mbox {$\tilde {F}^{\sigma }$}(c)\). Since the entries in the bumping path in the insertion *k*→*F*^{σ} form a strictly decreasing sequence when read in reading order, only one of {*F*^{σ}(*a*),*F*^{σ}(*b*),*F*^{σ}(*c*)} can be bumped by the insertion procedure *k*→*F*^{σ}. Let \(\bar{F}^{\sigma }\) be the extended diagram corresponding to *F*^{σ} as defined in our definition of the insertion *k*→*F*^{σ}. We claim that the triple conditions for *F*^{σ} imply that either \(\bar{F}^{\sigma}(b) < \bar{F}^{\sigma}(a) \leq \bar{F}^{\sigma}(c)\) or \(\bar{F}^{\sigma}(a) \leq \bar{F}^{\sigma}(c) < \bar{F}^{\sigma}(b)\). This follows from the fact that *F*^{σ} is a \(\operatorname {PBF}\) if *a*,*b*,*c* are cells in *F*^{σ}. Since the shape of \(\tilde{F}^{\sigma }\) arises from *γ* by adding a single cell on the outside of *γ*, we know that *c* is a cell in *F*^{σ}. However, it is possible that exactly one of *a* or *b* is not in *F*^{σ} and is filled with a 0 in \(\bar{F}^{\sigma }\). If it is *b*, then we automatically have \(\bar{F}^{\sigma}(b) < \bar {F}^{\sigma}(a) \leq \bar{F}^{\sigma}(c)\). If it is *a*, then the column that contains *a* is strictly shorter than the column that contains *b* because in \(\tilde{F}^{\sigma }\), it must be the case that the height of column *x*_{1} is greater than or equal to the height of column *x*_{2} since {*a*,*b*,*c*} is a type *A* triple in \(\tilde{F}^{\sigma }\). But then the *B*-increasing condition for *F*^{σ} forces \(\bar{F}^{\sigma }(c) < \bar{F}^{\sigma }(b)\) and, hence, \(\bar{F}^{\sigma}(a) \leq \bar{F}^{\sigma}(c) < \bar{F}^{\sigma}(b)\) must hold.

We now consider two cases.

**Case 1**. \(\bar{F}^{\sigma}(b) < \bar{F}^{\sigma}(a) \leq\bar {F}^{\sigma}(c)\).

Note in this case, \(0 < \bar{F}^{\sigma}(a)\) so that *a* is a cell in *F*^{σ}. Moreover the entries in *a* and *c* cannot be bumped in the insertion *k*→*F*^{σ} since their replacement by a larger value would not produce the desired ordering \(\mbox {$\tilde {F}^{\sigma }$}(a) \le \mbox {$\tilde {F}^{\sigma }$}(b) \le \mbox {$\tilde {F}^{\sigma }$}(c)\). Thus it must be the case that \(\mbox {$\tilde {F}^{\sigma }$}(b)\) was bumped in the insertion *k*→*F*^{σ}. We now consider two subcases.

**Subcase 1(a)**. \(\bar{F}^{\sigma}(a) < \mbox {$\tilde {F}^{\sigma }$}(b)\).

*k*→

*F*

^{σ}. It cannot be that \(\mbox {$\tilde {F}^{\sigma }$}(b)= k\) or that it was bumped from a cell that comes before

*a*in the reading order since it would then meet the conditions to bump the entry

*F*

^{σ}(

*a*) in cell

*a*as \(F^{\sigma}(a) < \mbox {$\tilde {F}^{\sigma }$}(b)\leq F^{\sigma }(c)\). Thus it must have been bumped from a cell after

*a*but before

*b*in reading order. That is, \(\mbox {$\tilde {F}^{\sigma }$}(b) = F^{\sigma }(d)\) where

*d*=(

*x*

_{3},

*y*

_{1}) and

*x*

_{1}<

*x*

_{3}<

*x*

_{2}. Thus we have the situation pictured in Fig. 7.

However, this is not possible since if \(\gamma_{x_{1}} \geq\gamma_{x_{3}}\), then the entries in cells *a*, *d*, and *c* would violate the *A*-triple condition for *F*^{σ} and, if \(\gamma_{x_{1}} < \gamma_{x_{3}}\), then the entries in cells *c* and *d* would violate the *B*-increasing condition on *F*^{σ}.

**Subcase 1(b)**. \(\bar{F}^{\sigma}(a) = \mbox {$\tilde {F}^{\sigma }$}(b)\).

*k*→

*F*

^{σ}. To this end, let

*r*be the least row such that

*r*>

*y*

_{1}and \(\bar{F}^{\sigma }(x_{1},r) < \bar{F}^{\sigma }(x_{1},r-1)\). Then we will have the situation pictured in Fig. 8 where

*d*is the cell in column

*x*

_{1}and row

*r*. Thus all the entries of

*F*

^{σ}in the cells in column

*x*

_{1}between

*a*and

*d*are equal to

*F*

^{σ}(

*a*).

Now the region of shaded cells pictured in Fig. 8 are cells which are attacked or attack some cell which is equal to *F*^{σ}(*a*) and hence their entries in *F*^{σ} must all be different from *F*^{σ}(*a*). Hence \(\mbox {$\tilde {F}^{\sigma }$}(b)\) cannot have come from any of these cells since we are assuming that \(\bar{F}^{\sigma }(a) = \mbox {$\tilde {F}^{\sigma }$}(b)\). Thus \(\mbox {$\tilde {F}^{\sigma }$}(b)\) must have come from a cell before *d* in reading order. But this is also impossible because \(\mbox {$\tilde {F}^{\sigma }$}(b)\) would then meet the conditions to bump \(\bar{F}^{\sigma }(d)\) which would violate our assumption that it bumps *F*^{σ}(*b*).

**Case 2**. \(\bar{F}^{\sigma}(a) \leq\bar{F}^{\sigma}(c) < \bar {F}^{\sigma}(b)\).

The entry in cell *c* is the only entry which could be bumped in the insertion *k*→*F*^{σ} if we are to end up with the relative ordering \(\mbox {$\tilde {F}^{\sigma }$}(a) \le \mbox {$\tilde {F}^{\sigma }$}(b) \le \mbox {$\tilde {F}^{\sigma }$}(c)\). Since *F*^{σ}(*c*) is bumped, this means that *c* is not in the basement. But if we do not bump either *a* or *b* in the insertion *k*→*F*^{σ} and *a* and *b* are cells in \(\tilde{F}^{\sigma }\), it must be the case that *a* and *b* are cells in *F*^{σ} and that there is no change in the heights of columns *x*_{1} and *x*_{2}. Thus \(\gamma_{x_{1}} \geq\gamma_{x_{2}}\). Let \(\underline{c}\) be the cell immediately below *c* and \(\underline{b}\) be the cell immediately below *b*. Thus we must have \(F^{\sigma }(c) < \mbox {$\tilde {F}^{\sigma }$}(c) \leq F^{\sigma }(\underline{c})\). We now consider two subcases.

**Subcase 2(a)**. \(\mbox {$\tilde {F}^{\sigma }$}(c) = F^{\sigma }(b)\).

*r*be the least row such that

*r*>

*y*

_{1}and \(\bar{F}^{\sigma }(x_{2},r) < \bar{F}^{\sigma }(x_{2},r-1)\). Then we will have the situation pictured in Fig. 9 where

*d*is the cell in column

*x*

_{2}and row

*r*. Thus all the entries of

*F*

^{σ}in the cells on column

*x*

_{2}between

*b*and

*d*are equal to

*F*

^{σ}(

*b*).

Now the region of shaded cells pictured in Fig. 9 are cells which are attacked or attack some cell which is equal to *F*^{σ}(*b*) and hence their entries in *F*^{σ} must all be different from *F*^{σ}(*b*). Thus \(\mbox {$\tilde {F}^{\sigma }$}(c)\) cannot have come from any of these cells since we are assuming that \(F^{\sigma }(b) = \mbox {$\tilde {F}^{\sigma }$}(c)\). Hence \(\mbox {$\tilde {F}^{\sigma }$}(c)\) must have come from a cell before *d* in reading order. But this is also impossible because \(\mbox {$\tilde {F}^{\sigma }$}(c)\) would then meet the conditions to bump \(\bar{F}^{\sigma }(d)\) which would violate our assumption that it bumps *F*^{σ}(*c*).

**Subcase 2(b)**. \(F^{\sigma }(b)< \mbox {$\tilde {F}^{\sigma }$}(c)\).

First, consider the *A*-triple \(c,\underline{c},\underline{b}\) in *F*^{σ}. We cannot have that \(F^{\sigma }(\underline{b}) < F^{\sigma }(c) \leq F^{\sigma }(\underline{c})\) since that would imply \(F^{\sigma }(b) \leq F^{\sigma }(\underline{b}) < F^{\sigma }(c)\), which would violate our assumption that *F*^{σ}(*a*)<*F*^{σ}(*c*)<*F*^{σ}(*b*). Thus it must be the case that \(F^{\sigma }(c) \leq F^{\sigma }(\underline{c}) < F^{\sigma }(\underline{b})\). But then we would have \(F^{\sigma }(b)< \mbox {$\tilde {F}^{\sigma }$}(c) \leq F^{\sigma }(\underline{c}) < F^{\sigma }(\underline{b})\) which would mean that \(\mbox {$\tilde {F}^{\sigma }$}(c)\) satisfies the conditions to bump *F*^{σ}(*b*). Since it does not bump *F*^{σ}(*b*), it must be the case that \(\mbox {$\tilde {F}^{\sigma }$}(c)\) came from a cell which is after *b* in the reading order. We now consider two more subcases.

**Subcase 2(bi)**. \(\mbox {$\tilde {F}^{\sigma }$}(c)\) is in the same row as *F*^{σ}(*b*).

Assume that \(\mbox {$\tilde {F}^{\sigma }$}(c) = F^{\sigma }(d)\) where *d*=(*x*_{3},*y*_{1}) and *x*_{2}<*x*_{3}. It cannot be that \(\gamma_{x_{2}} < \gamma_{x_{3}}\) since then the *B*-increasing condition would force that \(F^{\sigma }(\underline{b}) < F^{\sigma }(d) = \mbox {$\tilde {F}^{\sigma }$}(c)\). But that would mean that \(F^{\sigma }(\underline{b}) < \mbox {$\tilde {F}^{\sigma }$}(c) \leq F^{\sigma }(\underline{c})\) which violates the fact that \(F^{\sigma }(c) \leq F^{\sigma }(\underline{c}) < F^{\sigma }(\underline{b})\). Thus it must be the case that \(\gamma_{x_{2}} \geq\gamma_{x_{3}}\) and, hence, \(b,\underline{b},d\) is a type *A* triple. As we cannot have \(F^{\sigma }(\underline{b}) < F^{\sigma }(d) = \mbox {$\tilde {F}^{\sigma }$}(c)\), it must be the case that \(\mbox {$\tilde {F}^{\sigma }$}(c) = F^{\sigma }(d) < F^{\sigma }(b) \leq F^{\sigma }(\underline{b})\). But this is also impossible because we are assuming that \(F^{\sigma }(b)< \mbox {$\tilde {F}^{\sigma }$}(c)\).

**Subcase 2(bii)**. \(\mbox {$\tilde {F}^{\sigma }$}(c)\) is in the same row as *F*^{σ}(*c*).

*e*

_{1},…,

*e*

_{s},

*e*

_{s+1}=

*c*be the cells in the bumping path of the insertion of

*k*→

*F*

^{σ}in row

*y*

_{1}−1, reading from left to right. Thus we are assuming that \(\mbox {$\tilde {F}^{\sigma }$}(c) =F^{\sigma }(e_{s})\). For each

*e*

_{i}, we let \(\underline{e}_{i}\) be the cell directly below

*e*

_{i}and \(\overline{e}_{i}\) be the cell directly above

*e*

_{i}. Thus we have the picture in Fig. 10 where we are assuming that

*s*=3 and we have circled the elements in the bumping path.

Since the elements in the bumping path strictly decrease, we have that *F*^{σ}(*e*_{1})>⋯>*F*^{σ}(*e*_{s})>*F*^{σ}(*e*_{s+1})=*F*^{σ}(*c*) and that for each *i*, \(F^{\sigma }(e_{i+1}) < F^{\sigma }(e_{i}) \leq F^{\sigma }(\underline{e}_{i+1})\). Let *e*_{j}=(*z*_{j},*y*_{1}−1). Thus *z*_{s+1}=*x*_{1}. By Lemma 1, we must have \(\gamma_{z_{1}} \geq\cdots\geq\gamma_{z_{s}} \geq\gamma_{x_{1}}\). This means that the \(\overline{e}_{i}\)s are cells in *F*^{σ} so that Lemma 1 also implies that \(F^{\sigma }(\overline{e}_{1}) > \cdots> F^{\sigma }(\overline{e}_{s})\). Note that in this case, we have \(\gamma_{x_{1}} \geq\gamma_{x_{2}}\) so that we know that \(\gamma_{z_{1}} \geq\cdots\geq\gamma_{z_{s}} \geq\gamma_{x_{1}} \geq\gamma_{x_{2}}\). Now consider the *A* triples \(\{e_{i}, \underline{e}_{i},\underline{b}\}\). We are assuming that \(F^{\sigma }(c) = F^{\sigma }(e_{s+1}) \leq F^{\sigma }(\underline{e}_{s+1}) = F^{\sigma }(\underline{c}) < F^{\sigma }(\underline{b})\). But since \(F^{\sigma }(e_{s+1}) < F^{\sigma }(e_{s}) \leq F^{\sigma }(\underline{e}_{s+1})\), the \(\{e_{s},\underline{e}_{s},\underline{b}\}\) A-triple condition must be that \(F^{\sigma }(e_{s}) \leq F^{\sigma }(\underline{e}_{s}) < F^{\sigma }(\underline{b})\). Now if *e*_{s−1} exists, then we know that \(F^{\sigma }(e_{s}) < F^{\sigma }(e_{s-1}) \leq F^{\sigma }(\underline{e}_{s})\) and, hence, the \(\{e_{s-1},\underline{e}_{s-1},\underline{b}\}\)*A*-triple condition must also be that \(F^{\sigma }(e_{s-1}) \leq F^{\sigma }(\underline{e}_{s-1}) < F^{\sigma }(\underline{b})\). If *e*_{s−2} exists, then we know that \(F^{\sigma }(e_{s-1}) < F^{\sigma }(e_{s-2}) \leq F^{\sigma }(\underline{e}_{s-1})\) and, hence, the \(\{e_{s-2},\underline{e}_{s-2},\underline{b}\}\)*A*-triple condition must also be that \(F^{\sigma }(e_{s-2}) \leq F^{\sigma }(\underline{e}_{s-2}) < F^{\sigma }(\underline{b})\). Continuing on in this way, we can conclude that for all *j*, \(F^{\sigma }(e_{j}) \leq F^{\sigma }(\underline{e}_{j}) < F^{\sigma }(\underline{b})\). Next consider the \(\overline{e}_{i}, e_{i}, b\)*A*-triple conditions. We are assuming that \(F^{\sigma }(b) < \mbox {$\tilde {F}^{\sigma }$}(c) = F^{\sigma }(e_{s})\). Thus it must be the case that \(F^{\sigma }(b) < F^{\sigma }(\overline{e}_{s}) \leq F^{\sigma }({e}_{s})\). Since \(F^{\sigma }(\overline{e}_{s}) < F^{\sigma }(\overline{e}_{s-1}) < \cdots< F^{\sigma }(\overline{e}_{1})\), it must be the case that for all *j*, \(F^{\sigma }(b) < F^{\sigma }(\overline{e}_{j}) \leq F^{\sigma }(e_{j})\).

Thus in this case, we must have \(F^{\sigma }(b) < F^{\sigma }(\overline{e}_{1}) \leq F^{\sigma }(e_{1}) \leq F^{\sigma }(\underline{e}_{1}) < F^{\sigma }(\underline{b})\). Now the question is where can the element *z* which bumps *F*^{σ}(*e*_{1}) come from? We claim that *z* cannot equal *k* or come from a cell before *b* in reading order since it satisfies the condition to bump *b* and *b* is not bumped. Thus it must have come from a cell *d*=(*x*_{3},*y*_{1}) which lies in the same row as *b* but comes after *b* in reading order. In that case, we must have \(F^{\sigma }(e_{1}) < F^{\sigma }(d) \leq F^{\sigma }(\underline{e_{1}}) < F^{\sigma }(\underline{b})\). Thus it cannot be that \(\gamma_{x_{2}} < \gamma_{x_{3}}\) since the *B*-increasing condition would force \(F^{\sigma }(\underline{b}) < F^{\sigma }(d)\). Thus \(\gamma_{x_{2}} \geq\gamma_{x_{3}}\). But in that case, we would have \(F^{\sigma }(b) < F^{\sigma }(d) < F^{\sigma }(\underline{b})\) which would be a coinversion *A* triple in *F*^{σ}.

Thus we have shown that in Subcase 2, *c* could not have been bumped and, hence, there can be no coinversion *A* triples in *k*→*F*^{σ}. □

It is obvious that our insertion algorithm ensures that the columns of *k*→*F*^{σ} are weakly increasing when read from top to bottom. Thus if we can show that *k*→*F*^{σ} satisfies the *B*-increasing condition, we know that all *B* triples in *k*→*F*^{σ} will be inversion triples.

### Lemma 4

*If**F*^{σ}*is a PBF*, *then*\(\tilde{F}^{\sigma }= k \rightarrow F^{\sigma}\)*satisfies the B*-*increasing condition*.

### Proof

*F*

^{σ}is of shape

*γ*=(

*γ*

_{1},…,

*γ*

_{n}) where

*n*≥

*k*. Suppose that \(\tilde{F}^{\sigma}\) does not satisfy the

*B*-increasing condition. Thus there must be a type

*B*triple {

*b*=(

*x*

_{1},

*y*

_{1}),

*a*=(

*x*

_{2},

*y*

_{1}+1),

*c*=(

*x*

_{2},

*y*

_{1})} in \(\tilde{F}^{\sigma} : = k \rightarrow F^{\sigma}\) as depicted in Fig. 11 such that \(\mbox {$\tilde {F}^{\sigma }$}(b) \geq \mbox {$\tilde {F}^{\sigma }$}(a)\). Assume that we have picked

*a*and

*b*so that

*b*is as far left as possible. Let \(\overline{b}\) denote the cell immediately above

*b*and \(\underline{b}\) denote the cell immediately below

*b*. Then there are two possibilities, namely, it could be that \(\gamma_{x_{1}} < \gamma_{x_{2}}\) so that {

*a*,

*b*,

*c*} forms a type

*B*triple in

*F*

^{σ}or it could be that \(\gamma_{x_{1}} = \gamma_{x_{2}}\) and we added an element on the top of column

*x*

_{2}during the insertion

*k*→

*F*

^{σ}so that in \(\mbox {$\tilde {F}^{\sigma }$}\), the height of column

*x*

_{1}is strictly less than the height of column

*x*

_{2}.

**Case 1**. \(\gamma_{x_{1}} < \gamma_{x_{2}}\).

In this case, the *B*-increasing condition for *F*^{σ} implies that *F*^{σ}(*b*)<*F*^{σ}(*a*) and \(F^{\sigma }(\underline{b})< F^{\sigma }(c)\). As the elements in the bumping path strictly decrease, it must be the case that *F*^{σ}(*b*) is bumped and *F*^{σ}(*a*) is not bumped. Thus we must have that \(F^{\sigma }(\underline{b})\geq \mbox {$\tilde {F}^{\sigma }$}(b) > F^{\sigma }(b)\).

*r*be the least row such that

*r*>

*y*

_{1}+1 and \(\bar{F}^{\sigma }(x_{2},r) < \bar{F}^{\sigma }(x_{2},r-1)\). Then we will have the situation pictured in Fig. 12 where

*d*is the cell in column

*x*

_{2}and row

*r*. Thus all the entries of

*F*

^{σ}in the cells in column

*x*

_{2}between

*a*and

*d*are equal to

*F*

^{σ}(

*a*). Now the region of shaded cells pictured in Fig. 12 are cells which are attacked or attack some cell which is equal to

*F*

^{σ}(

*a*) and hence their entries in

*F*

^{σ}must all be different from

*F*

^{σ}(

*a*). Hence \(\mbox {$\tilde {F}^{\sigma }$}(b)\) cannot have come from any of these cells since we are assuming that \(F^{\sigma }(a) = \mbox {$\tilde {F}^{\sigma }$}(b)\). Thus \(\mbox {$\tilde {F}^{\sigma }$}(b)\) must be either equal to

*k*or have come from a cell in

*F*

^{σ}which precedes

*d*in reading order. But this is also impossible because \(\mbox {$\tilde {F}^{\sigma }$}(b)\) would then meet the conditions to bump \(\bar{F}^{\sigma }(d)\) which would violate our assumption that it bumps

*F*

^{σ}(

*b*).

Thus we can assume that *F*^{σ}(*a*)<*F*^{σ}(*b*). Now the question is where did \(\mbox {$\tilde {F}^{\sigma }$}(b)\) come from?

*k*or was equal to

*F*

^{σ}(

*d*) where

*d*comes before

*a*in reading order since then we have that

*F*

^{σ}(

*a*) which would violate our assumption that \(\mbox {$\tilde {F}^{\sigma }$}(b)\) bumps

*F*

^{σ}(

*b*).

Similarly, it cannot be the case that \(\mbox {$\tilde {F}^{\sigma }$}(b) = F^{\sigma }(d)\) where *d* is a cell to the right of *a* and in the same row as *a*. That is, if *d*=(*x*_{3},*y*_{1}+1) where *x*_{2}<*x*_{3}, then either (i) \(\gamma_{x_{2}} < \gamma_{x_{3}}\) in which case the fact that \(F^{\sigma }(c) > F^{\sigma }(d) = \mbox {$\tilde {F}^{\sigma }$}(b)\) would mean that cells *d* and *c* violate the *B*-increasing condition for *F*^{σ} or (ii) \(\gamma_{x_{2}} \geq \gamma_{x_{3}}\) in which case the triple {*a*,*c*,*d*} would be a type *A* coinversion triple in *F*^{σ}.

*b*and in the same row as

*b*in

*F*

^{σ}. So let

*e*

_{1},…,

*e*

_{s},

*e*

_{s+1}=

*b*be the cells in the bumping path of the insertion of

*k*→

*F*

^{σ}in row

*y*

_{1}, reading from left to right. Thus we are assuming that \(\mbox {$\tilde {F}^{\sigma }$}(b) =F^{\sigma }(e_{s})\). For each

*e*

_{i}, we let \(\underline{e}_{i}\) be the cell directly below

*e*

_{i}. Thus we have the situation pictured in Fig. 13 where we are assuming that

*s*=3 and we have circled the elements in the bumping path.

Since the elements in the bumping path strictly decrease, we have that *F*^{σ}(*e*_{1})>⋯>*F*^{σ}(*e*_{s})>*F*^{σ}(*e*_{s+1})=*F*^{σ}(*b*)>*F*^{σ}(*a*). Moreover, for each 1≤*i*≤*s*, we have \(F^{\sigma }(e_{i+1}) < F^{\sigma }(e_{i}) \leq F^{\sigma }(\underline{e}_{i+1})\). Let *e*_{j}=(*z*_{j},*y*_{1}) for *j*=1,…,*s*+1. Thus *z*_{s+1}=*x*_{1}. By Lemma 1, we must have \(\gamma_{z_{1}} \geq\cdots\geq\gamma_{z_{s}} \geq\gamma_{x_{1}}\). Note that the fact that we chose *b* to be as far left as possible means that it must be the case that \(\gamma_{z_{j}} \geq\gamma_{x_{2}}\) for 1≤*j*≤*s*. That is, if for some 1≤*j*≤*s*, \(\gamma_{z_{j}} < \gamma_{x_{2}}\), then the entries in cells *a* and *e*_{j} would violate the *B*-increasing condition in *F*^{σ} which would violate our choice of *b*. Thus \(\{e_{j},\underline{e}_{j},c\}\) is a type *A* triple for 1≤*j*≤*s*. Since \(F^{\sigma }(c)> F^{\sigma }(\underline{b}) = F^{\sigma }(\underline{e}_{s+1}) \geq F^{\sigma }(e_{s})\), it must be the case that the \(\{c,e_{s},\underline{e}_{s}\}\). A triple condition is \(F^{\sigma }(e_{s}) \leq F^{\sigma }(\underline{e}_{s}) < F^{\sigma }(c)\). Now assume by induction that we have shown that \(F^{\sigma }(e_{j}) \leq F^{\sigma }(\underline{e}_{j}) < F^{\sigma }(c)\). Then since \(F^{\sigma }(e_{j-1}) \leq F^{\sigma }(\underline{e}_{j}) \), the \(\{a,e_{j-1},\underline{e}_{j-1}\}\) A triple condition must be that \(F^{\sigma }(e_{j-1}) \leq F^{\sigma }(\underline{e}_{j-1}) < F^{\sigma }(c)\). It thus follows that \(F^{\sigma }(e_{1}) \leq F^{\sigma }(\underline{e}_{1}) < F^{\sigma }(c)\).

*k*or is equal to

*F*

^{σ}(

*d*) for some cell

*d*which precedes

*a*in reading order since then \(\mbox {$\tilde {F}^{\sigma }$}(e_{1})\) would bump

*F*

^{σ}(

*a*). By our choice of

*e*

_{1}, the only other possibility is that \(\mbox {$\tilde {F}^{\sigma }$}(e_{1}) = F^{\sigma }(d)\) for some cell

*d*to the right of

*a*and in the same row as

*a*. Say

*d*=(

*x*

_{3},

*y*

_{1}+1) where

*x*

_{2}<

*x*

_{3}. Then it cannot be that \(\gamma_{x_{2}}< \gamma_{x_{3}}\) since then the cells

*d*and

*c*would violate the

*B*-increasing condition in

*F*

^{σ}and it cannot be that \(\gamma_{x_{2}}\geq \gamma_{x_{3}}\) since then the triple {

*a*,

*c*,

*d*} would be a type

*A*coinversion triple in

*F*

^{σ}.

Thus we have shown that \(\gamma_{x_{1}} < \gamma_{x_{2}}\) is impossible.

**Case 2**. \(\gamma_{x_{1}} = \gamma_{x_{2}} =y\)

Thus we must have added an element on the top of column *x*_{2} during the insertion *k*→*F*^{σ} so that in \(\mbox {$\tilde {F}^{\sigma }$}\), the height of column *x*_{1} is strictly less than the height of column *x*_{2}. In this case, neither *b* nor *c* were involved in the bumping path of *k*→*F*^{σ} so that \(F^{\sigma }(b) = \tilde{F}^{\sigma }(b)\) and \(F^{\sigma }(c) = \tilde{F}^{\sigma }(c)\). We claim that it must be the case that \(\tilde{F}^{\sigma }(x_{1},y) \geq\tilde{F}^{\sigma }(x_{2},y+1)\). That is, if *y*=*y*_{1}, then \(\tilde{F}^{\sigma }(x_{1},y) = \tilde{F}^{\sigma }(b) \geq \tilde{F}^{\sigma }(a) = \tilde{F}^{\sigma }(x_{2},y+1)\) since we are assuming that \(\tilde{F}^{\sigma }(b) \geq \tilde{F}^{\sigma }(a)\). If *y*>*y*_{1}, then the triple \(\{ \overline{b},a,b \}\) is a type *A* triple in *F*^{σ} and \(F^{\sigma }(a) = \tilde{F}^{\sigma }(a)\). We now have two possibilities, namely, either (i) \(F^{\sigma }(a) < F^{\sigma }(\overline{b}) \leq F^{\sigma }(b)\) or (ii) \(F^{\sigma }(\overline{b}) \leq F^{\sigma }(b) < F^{\sigma }(a)\). Note that (ii) is inconsistent with our assumption that \(\tilde{F}^{\sigma }(b) \geq \tilde {F}^{\sigma }(a)\) so that it must be the case that \(F^{\sigma }(\overline{b}) > F^{\sigma }(a)\). But then we know by part (ii) of Lemma 1 that *F*^{σ}(*x*_{1},*y*)>*F*^{σ}(*x*_{2},*y*). Our insertion algorithm ensures that \(F^{\sigma }(x_{2},y) \geq\tilde{F}^{\sigma }(x_{2},y+1)\) so that \(F^{\sigma }(x_{1},y) > \tilde{F}^{\sigma }(x_{2},y+1)\) in this case.

Now consider the question of where \(\tilde{F}^{\sigma }(x_{2},y+1)\) came from in the bumping process. It cannot be the case that \(\tilde{F}^{\sigma }(x_{2},y+1) = k\) or was bumped from a cell before (*x*_{1},*y*+1) in the reading order because then \(\tilde{F}^{\sigma }(x_{2},y+1)\) could be placed on top of *F*^{σ}(*x*_{1},*y*) and \(\bar{F}^{\sigma }(x_{1},y+1)=0\) in this case. Thus \(\tilde{F}^{\sigma }(x_{2},y+1)\) must have been bumped from some cell *d* between (*x*_{1},*y*+1) and (*x*_{2},*y*+1) in reading order. But this is impossible since \(F^{\sigma }(x_{1},y) \geq F^{\sigma }(d) = \tilde{F}^{\sigma }(x_{2},y+1)\) would mean that (*x*_{1},*y*) and *d* do not satisfy the *B*-increasing condition in *F*^{σ}. Thus we have shown that the assumption that \(\mbox {$\tilde {F}^{\sigma }$}(b) \geq \mbox {$\tilde {F}^{\sigma }$}(a)\) leads to a contradiction in all cases and, hence, \(\mbox {$\tilde {F}^{\sigma }$}\) must satisfy the *B*-increasing condition. □

### Proposition 5

*The insertion procedure**k*→*F*^{σ}*is well*-*defined and produces a*\(\operatorname {PBF}\).

### Proof

Let *F*^{σ} be an arbitrary \(\operatorname {PBF}\) of shape *γ* and basement *σ*∈*S*_{n} and let *k* be an arbitrary positive integer less than or equal to *n*. We must show that the procedure *k*→*F*^{σ} terminates and that the resulting filling is indeed a \(\operatorname {PBF}\). Lemma 2 implies that at most one occurrence of any given value will be bumped to the first row. Therefore each entry *i* in the first row will be inserted into a column at or before the column *σ*^{−1}(*i*). This means that the insertion procedure terminates and hence is well-defined.

Lemmas 3 and 4 imply that *k*→*F*^{σ} is a semi-standard augmented filling which satisfies the *B*-increasing condition. Thus *k*→*F*^{σ} is a \(\operatorname {PBF}\). □

Before proceeding, we make two remarks. Our first remark is concerned with the process of inverting our insertion procedure. That is, the last cell or terminal cell in the bumping path of *k*→*F*^{σ} must be a cell that originally contained 0 in \(\bar {F}^{\sigma }\). Such a cell was not in *F*^{σ} so that the shape of \(\mbox {$\tilde {F}^{\sigma }$}\) is the result of adding one new cell *c* on the top of some column of the shape of *F*^{σ}. However, there are restrictions as to where this new cell may be placed. That is, we have the following proposition which says that if *c* is the top cell of a column in a sequence of columns which have the same height in *k*→*F*^{σ}, then *c* must be in the rightmost of those columns.

### Proposition 6

*Suppose that**σ*∈*S*_{n}*and**F*^{σ}*is a*\(\operatorname {PBF}\)*with basement**σ**and**k*≤*n*. *Suppose that**F*^{σ}*has shape**γ*=(*γ*_{1},…,*γ*_{n}), *k*→*F*^{σ}*has shape**δ*=(*δ*_{1},…,*δ*_{n}), *and* (*x*,*y*) *is the cell in**δ*/*γ*. *Then it must be case that if**x*<*n*, *then* 1+*γ*_{x}≠*γ*_{x+j}*for* 1≤*j*≤*n*−*x*. *In particular*, *if**x*<*n*, *then**δ*_{x}≠*δ*_{x+j}*for* 1≤*j*≤*n*−*x*.

### Proof

*x*<

*n*and 1+

*γ*

_{x}=

*γ*

_{x+j}=

*y*for some

*j*such that 1≤

*j*≤

*n*−

*x*. Let

*G*

^{σ}=

*k*→

*F*

^{σ}. and let \(\bar{F}^{\sigma }\) and \(\bar{G}^{\sigma }\) be the fillings which result by placing 0s on top of the columns of

*F*

^{σ}and

*G*

^{σ}, respectively. Thus we would have the situation pictured in Fig. 14 for the end of the bumping path in the insertion

*k*→

*F*

^{σ}.

*b*is at the top of column

*x*+

*j*in both

*F*

^{σ}and

*G*

^{σ}and neither

*F*

^{σ}(

*b*) nor

*F*

^{σ}(

*c*) are bumped during the insertion of

*k*→

*F*

^{σ}. Note that

*B*-increasing condition in

*F*

^{σ}forces that

*F*

^{σ}(

*c*)<

*F*

^{σ}(

*b*). Thus the {

*a*,

*b*,

*c*}

*A*-triple condition in

*G*

^{σ}must be that

*G*

^{σ}(

*a*) came from in the bumping path of the insertion

*k*→

*F*

^{σ}. It cannot be that

*G*

^{σ}(

*a*)=

*k*or

*G*

^{σ}(

*a*) was bumped from a cell before (

*x*+

*j*,

*y*+1) because of the fact that

*G*

^{σ}(

*a*)<

*G*

^{σ}(

*b*)=

*F*

^{σ}(

*b*) would allow

*G*

^{σ}(

*a*) to be inserted on top of cell

*b*. Thus either (i)

*G*

^{σ}(

*a*)=

*F*

^{σ}(

*z*,

*y*+1) for some

*z*>

*x*+

*j*or (ii)

*G*

^{σ}(

*a*)=

*F*

^{σ}(

*z*,

*y*) for some

*z*<

*x*. Case (i) is impossible since then we would have

*γ*

_{x+j}<

*γ*

_{z}and the

*B*-increasing condition in

*F*

^{σ}would force

*G*

^{σ}(

*b*)=

*F*

^{σ}(

*b*)<

*F*

^{σ}(

*z*,

*y*+1)=

*G*

^{σ}(

*a*).

*e*

_{1},…,

*e*

_{s},

*e*

_{s+1}=(

*x*,

*y*) be the cells in row

*y*of the bumping path of the insertion of

*k*→

*F*

^{σ}, reading from left to right. Thus we are assuming that

*G*

^{σ}(

*a*)=

*F*

^{σ}(

*e*

_{s}). For each

*e*

_{i}, we let \(\underline{e}_{i}\) be the cell directly below

*e*

_{i}. Thus we have the picture in Fig. 15 where we are assuming that

*s*=3 and we have circled the elements in the bumping path.

Since the elements in the bumping path strictly decrease, we have that *F*^{σ}(*e*_{1})>⋯>*F*^{σ}(*e*_{s})=*G*^{σ}(*a*) and that for each *i*, \(F^{\sigma }(e_{i+1}) < F^{\sigma }(e_{i}) \leq F^{\sigma }(\underline{e}_{i+1})\). Let *e*_{j}=(*z*_{j},*y*). Thus *z*_{s+1}=*x*. It follows from Lemma 1 that \(\gamma_{z_{1}} \geq\cdots\geq\gamma_{z_{s}} > \gamma_{x}\).

Now consider the *A*-triples \(\{e_{i},\underline{e}_{i},(x+j,y)\}\) for *i*=1,…,*s* in *F*^{σ}. We have established that *F*^{σ}(*e*_{s})=*G*^{σ}(*a*)≤*G*^{σ}(*c*)<*G*^{σ}(*b*)=*F*^{σ}(*x*+*j*,*y*). Thus it follows from the \(\{e_{s},\underline{e}_{s},(x+j,y)\}\)*A*-triple condition that \(F^{\sigma }(e_{s}) \leq F^{\sigma }(\underline{e}_{s}) < F^{\sigma }(b)\). But then \(F^{\sigma }(e_{s}) < F^{\sigma }(e_{s-1})\leq F^{\sigma }(\underline{e}_{s})\) so that the \(\{e_{s-1},\underline{e}_{s-1},(x+j,y)\}\)*A*-triple condition also implies that \(F^{\sigma }(e_{s-1}) \leq F^{\sigma }(\underline{e}_{s-1}) < F^{\sigma }(b)\). Continuing on in this way, we can conclude from the \(\{e_{i},\underline{e}_{i},(x+j,y)\}\)*A*-triple condition that \(F^{\sigma }(e_{i}) \leq F^{\sigma }(\underline{e}_{i}) < F^{\sigma }(b)\) for *i*=1,…,*s*.

Now consider the element *z* that bumps *F*^{σ}(*e*_{1}) in the insertion *k*→*F*^{σ}. We must have \(F^{\sigma }(e_{1}) < z \leq F^{\sigma }(\underline{e}_{1})< F^{\sigma }(b)\). Thus it cannot be that *z*=*k* or *z*=*F*^{σ}(*d*) for some cell *d* which precedes (*x*+*j*,*y*+1) in reading order because that would mean that *z* meets the conditions to be placed on top of *b*. Thus it must be that *z*=*F*^{σ}(*d*) for some cell *d* which follows (*x*+*j*,*y*+1) in reading order. Suppose that *d*=(*t*,*y*+1) where *t*>*x*+*j*. But we are assuming that (*x*+*j*,*y*) is the top cell in column *x*+*j*. Thus it must be the case that *γ*_{x+j}<*γ*_{t}. But then the *B*-increasing condition in *F*^{σ} would force *F*^{σ}(*b*)<*F*^{σ}(*d*)=*z* which is a contradiction. Thus case (ii) cannot hold either which implies 1+*γ*_{x}≠*γ*_{x+j}. □

Except for the restrictions determined by Proposition 6, we can invert the insertion procedure. That is, to invert the procedure *k*→*F*^{σ}, begin with the entry *r*_{j} contained in the new cell appended to *F*^{σ} and read backward through the reading order beginning with this cell until an entry is found which is greater than *r*_{j} and immediately below an entry less than or equal to *r*_{j}. Let this entry be *r*_{j−1}, and repeat. When the first cell of *k*→*F*^{σ} is passed, the resulting entry is *r*_{1}=*k* and the procedure has been inverted.

Our second remark concerns the special case where \(\sigma = \bar{\epsilon}_{n}\) and *k*≤*n*. In that case, we claim that our insertion procedure is just a twisted version of the usual RSK row insertion algorithm. That is, we know that *F*^{σ} must be of shape *γ*=(*γ*_{1},…,*γ*_{n}) where *γ*_{1}≥*γ*_{2}≥⋯≥*γ*_{n} and that *F*^{σ} is weakly decreasing in columns, reading from bottom to top, and is strictly decreasing in rows, reading from left to right. Now if *k*≤*F*^{σ}(1,*γ*_{1}), then we just add *k* to the top of column 1 to form *k*→*F*^{σ}. Otherwise suppose that *F*^{σ}(1,*y*_{1})≥*k*>*F*^{σ}(1,*y*_{1}+1). Then all the elements in \(\bar{F}^{\sigma }\) that lie weakly above row *y*_{1}+1 and strictly to the right of column 1 must be less than or equal to *F*^{σ}(1,*y*_{1}+1). Thus the first place that we can insert *k* is in cell (1,*y*_{1}+1). Thus it will be that case that *k* bumps *F*^{σ}(1,*y*_{1}+1). Since elements in the bumping path are decreasing and all the elements in column 1 below row *y*_{1}+1 are strictly larger than *F*^{σ}(1,*y*_{1}+1), it follows that none of them can be involved in the bumping path of the insertion *k*→*F*^{σ}. It is then easy to check that since *F*^{σ}(1,*y*_{1}+1)≤*n*−1, the result of the insertion *k*→*F*^{σ} is the same as the result of the insertion of *F*^{σ}(1,*y*_{1}+1) into the \(\operatorname {PBF}\) formed from *F*^{σ} by removing the first column and then adding back column 1 of *F*^{σ} with *F*^{σ}(1,*y*_{1}+1) replaced by *k*. Thus our insertion process satisfies the usual recursive definition of the RSK row insertion algorithm. Hence, in the special case where the basement permutation is \(\bar{\epsilon}_{n}\) and *k*≤*n*, our insertion algorithm is just the usual RSK row insertion algorithm subject to the condition that we have weakly decreasing columns and strictly decreasing rows.

## 4 General properties of the insertion algorithm

In this section, we shall prove several fundamental properties of the insertion algorithm *k*→*F*^{σ}. In particular, our results in this section will allow us to prove that our insertion algorithm can be factored through the twisted version of RSK row insertion described in the previous section.

*σ*, let

*E*

^{σ}be the empty filling which just consists of the basement whose entries are

*σ*

_{1},…,

*σ*

_{n}reading from left to right. Let

*s*

_{i}denote the transposition (

*i*,

*i*+1) so that if

*σ*=

*σ*

_{1}⋯

*σ*

_{n}, then

*w*into

*E*

^{σ}versus inserting

*w*into \(E^{s_{i}\sigma }\). If

*w*=

*w*

_{1}…

*w*

_{t}, then let

### Theorem 7

*Let*

*w*

*be an arbitrary word whose letters are less than or equal to*

*n*

*and suppose that*

*σ*=

*σ*

_{1}⋯

*σ*

_{n}

*is a permutation in*

*S*

_{n}

*such that*

*σ*

_{i}<

*σ*

_{i+1}.

*Let*

*F*

^{σ}=

*w*→

*E*

^{σ},

*γ*=(

*γ*

_{1},…,

*γ*

_{n})

*be the shape of*

*F*

^{σ}, \(F^{s_{i} \sigma} = w \rightarrow E^{s_{i} \sigma}\),

*and*

*δ*=(

*δ*

_{1},…,

*δ*

_{n})

*be the shape of*\(F^{s_{i} \sigma }\).

*Then*

- 1.
{

*γ*_{i},*γ*_{i+1}}={*δ*_{i},*δ*_{i+1}}*and**δ*_{i}≥*δ*_{i+1}, - 2.
\(F^{s_{i} \sigma} (i,j) > F^{s_{i} \sigma} ({i+1},j)\),

*for**j*≤*δ*_{i}*where we let*\(F^{s_{i} \sigma}(i+1,j)=0\)*if*(*i*+1,*j*)*is not a cell in**F*^{σ}. - 3.
\(F^{s_{i} \sigma}(j,k) = F^{\sigma}(j,k)\)

*for**j*≠*i*,*i*+1*so that**γ*_{j}=*δ*_{j}*for all**j*≠*i*,*i*+1, - 4.
*For all**j*, \(\{F^{s_{i} \sigma} (i,j) , F^{s_{i} \sigma} ({i+1},j)\} = \{ F^{\sigma} (i,j) , F^{\sigma} ({i+1},j) \}\).

### Proof

Note that since (*s*_{i}*σ*)_{i}=*σ*_{i+1}>*σ*_{i}=(*s*_{i}*σ*)_{i+1}, Lemma 1 implies claims 1 and 2. Thus we need only prove claims 3 and 4.

*w*. The theorem clearly holds when

*w*is the empty word. Now suppose that the theorem holds for all words of length less than

*t*. Then let

*G*=

*w*

_{1}…

*w*

_{t−1}→

*E*

^{σ}and \(H = w_{1} \ldots w_{t-1} \rightarrow E^{s_{i}\sigma }\) and suppose

*G*has shape

*α*=(

*α*

_{1},…,

*α*

_{n}) and

*H*has shape

*β*=(

*β*

_{1},…,

*β*

_{n}). Let \(\bar{G}\) and \(\bar{H}\) be the fillings with 0s added to the tops of the columns of

*G*and

*H*, respectively. Let \(\tilde{G} = w_{t} \rightarrow G\) and \(\tilde{H} = w_{t} \rightarrow H\) and suppose that \(\tilde{G}\) has shape

*γ*=(

*γ*

_{1},…,

*γ*

_{n}) and \(\tilde{H}\) has shape

*δ*=(

*δ*

_{1},…,

*δ*

_{n}). We compare the bumping path of

*w*

_{t}→

*H*to the bumping path in

*w*

_{t}→

*G*. That is, in the insertion process

*w*

_{t}→

*H*, suppose we come to a point were we are inserting some element

*c*which is either

*w*

_{t}or some element bumped in the insertion

*w*

_{t}→

*H*into the cells (

*i*,

*j*) and (

*i*+1,

*j*). Assume by a second inner reverse induction on the size of

*i*, that the insertion of

*w*

_{t}→

*G*will also insert

*c*into the cells (

*i*,

*j*) and (

*i*+1,

*j*). This will certainly be true the first time the bumping paths interact with elements in columns

*i*and

*i*+1 since our induction assumption ensures that \(\bar{G}\) restricted to columns 1,…,

*i*−1 equals \(\bar{H}\) restricted to columns 1,…,

*i*−1. Let \(x = \bar{H}(i,j)\), \(y = \bar{H}(i+1,j)\), \(\underline{x} = \bar{H}(i,j-1)\), and \(\underline{y} = \bar {H}(i+1,j-1)\) (see Fig. 16). Our inductive assumption implies that if

*x*>0, then

*x*>

*y*and if \(\underline{x} >0 \), then \(\underline{x} > \underline{y}\). Our goal is to analyze how the insertion of

*c*interacts with elements in cells (

*i*,

*j*) and (

*i*+1,

*j*) during the insertions

*w*

_{t}→

*H*and

*w*

_{t}→

*G*. We will show that either

(\(\mathbb{A}\)) The bumping path does not interact with cells (*i*,*j*) and (*i*+1,*j*) during either the insertions *w*_{t}→*H* or *w*_{t}→*G*,

(\(\mathbb{B}\)) The insertion of *c* into cells (*i*,*j*) and (*i*+1,*j*) results in inserting some *c*′ into the next cell in reading order after (*i*+1,*j*) in both *w*_{t}→*H* and *w*_{t}→*G*, or

(ℂ) Both insertions end up terminating in one of (*i*,*j*) or (*i*+1,*j*).

This will ensure that *w*_{t}→*H* and *w*_{t}→*G* are identical outside of columns *i* and *i*+1 thus proving condition 3 of the theorem and that {*H*(*i*,*j*),*H*(*i*+1,*j*)}={*G*(*i*,*j*),*G*(*i*+1,*j*)} which will prove condition 4 of the theorem.

Now suppose that the elements \(\bar{H}\) are in cells (*i*,*j*), (*i*+1,*j*), (*i*,*j*−1), and (*i*+1,*j*−1) are *x*, *y*, \(\underline{x}\) and \(\underline{y}\), respectively, as pictured on the left in Fig. 16. If \(\bar{G}\) and \(\bar{H}\) agree on those cells, then there is nothing to prove. Thus we have to consider three cases (I), (II), or (III) for the entries in \(\bar{G}\) in those cells which are pictured on the right in Fig. 16. We can assume that \(\underline{x} \neq0\). Now if \(y = \underline{y} =0\), then it is easy to see that one of (\(\mathbb{A}\)), (\(\mathbb{B}\)), or (ℂ) will hold since the insertion procedure sees the same elements possibly in different columns. Thus we can assume that \(\underline{y} \neq0\) and hence, \(\underline{x} > \underline{y}\).

We now consider several cases.

**Case A**. *x*=*y*=0.

This means that *x* and *y* sit on top of columns *i* and *i*+1, respectively, in *H*.

First, suppose that \(c \leq\underline{x}\). Then in *w*_{t}→*H*, the insertion will terminate by putting *c* on top of \(\underline{x}\). In case (I), the insertion *w*_{t}→*G* will terminate by placing *c* on top of \(\underline{x}\) and in cases (II) and (III), the insertion *w*_{t}→*G* will terminate by placing *c* on top of \(\underline{y}\) if \(c \leq\underline{y}\), or by placing *c* on top of \(\underline{x}\) if \(\underline{y} < c \leq\underline{x}\). In either situation, (ℂ) holds.

Next suppose that \(\underline{x} < c\), Then in *w*_{t}→*H*, *c* will not be placed in either cell (*i*,*j*) or (*i*+1,*j*) so that the result is that *c* will end up being inserted in the next cell in reading order after (*i*+1,*j*). But then in cases (I), (II), and (III), *c* will not be placed in either cell (*i*,*j*) or (*i*+1,*j*) in the insertion *w*_{t}→*G* so that the result is that *c* will end up being inserted in the next cell in reading order after (*i*+1,*j*). Thus (\(\mathbb{B}\)) holds in all cases.

**Case B**. *x*>0, and *y*=0.

Note that case (I) is impossible since then *x* and \(\underline{x}\) would violate the *B*-increasing condition in *G*.

First, consider the case where *c* does not bump *x* in the insertion *w*_{t}→*H*, and the insertion terminates with *c* being placed on top of \(\underline{y}\). Then it must be the case \(c \leq\underline{y}\). Moreover, it is the case that *x*>*c* by Lemma 1. Hence in case (II), *c* will not bump *x* and instead will be placed on top of \(\underline{x}\) since \(c \leq\underline{y} < \underline{x}\) and in case (III), *c* will be put on top of \(\underline{y}\). However, this is not possible because then the insertion would violate Proposition 6. Thus, we know that condition (ℂ) holds.

Next consider the case where in the insertion *w*_{t}→*H*, *c* bumps *x* and *x* terminates the insertion by being placed on top of \(\underline{y}\). Thus we know that \(x < c \leq\underline{x}\) and \(x \leq\underline{y}\). This rules out case (III) since then *x* and \(\underline{y}\) would violate the *B*-increasing condition and case (I) since then *x* and \(\underline{x}\) would violate the *B*-increasing condition. Now in case (II), *c* will bump *x* and *x* will be placed on top of \(\underline{x}\) if \(c \leq\underline{y}\). If \(c > \underline{y}\), then *c* will not bump *x* and *c* will be placed on top of \(\underline{x}\). In either situation, condition (ℂ) holds.

Next consider the case where in the insertion *w*_{t}→*H*, *c* bumps *x* and *x* cannot be placed on top of \(\underline{y}\) so that *x* is inserted in the next cell in reading order after (*i*+1,*j*). Then we must have \(\underline{y} < x < c \leq\underline{x}\). This rules out cases (I) and (II) since *x* cannot sit on top of \(\underline{y}\). In case (III), *c* cannot sit on top of \(\underline{y}\) so *c* will bump *x*. Thus condition (\(\mathbb{B}\)) holds in this case.

Finally, consider the case where *c* does not bump *x* and *c* cannot be placed on top of \(\underline{y}\) in the insertion *w*_{t}→*H* so that *c* is inserted in the next cell in reading order after (*i*+1,*j*). The fact that *c* does not bump *x* means that either \(c > \underline{x}\) or *c*≤*x*. The fact that *c* cannot be placed on top of \(\underline{y}\) means that \(c > \underline{y}\). If \(c > \underline{x} > \underline{y}\), then in cases (II) and (III), *c* does not meet the conditions for the entries in cells (*i*,*j*) and (*i*+1,*j*) to change so that the result is that *c* will be inserted in the next cell in reading order after (*i*+1,*j*). If *c*≤*x*, then we know that \(\underline{y} < c \leq x\). This rules out cases (I) and (II) since *x* cannot sit on \(\underline{y}\). In case (III), *c* cannot be placed on \(\underline{y}\) and *c* cannot bump *x* so that Case \(\mathbb{A}\) holds in either situation.

**Case C.**\(x,\underline{x},y, \underline{y} >0\).

Then we know that *x*>*y* and \(\underline{x} > \underline{y}\).

First, suppose that in the insertion *w*_{t}→*H*, *c* bumps *x*, but *x* does not bump *y* so that the result is that *x* will be inserted into the cell following (*i*+1,*j*) in reading order. Since *y*<*x*, the reason that *x* does not bump *y* must be that \(x > \underline{y}\). Thus it must be the case that \(\underline{x} \geq x > \underline{y} \geq y\). This means that cases (I) and (II) are impossible since *x* cannot sit on top of \(\underline{y}\) in *G*. But then \(c > x > \underline{y}\) so that in the insertion *w*_{t}→*G*, *c* cannot bump *y* in case (III). Thus in case (III), *c* will bump *x* so that the result is that *x* will be inserted into the cell following (*i*+1,*j*) in reading order as desired. Hence, condition (\(\mathbb{B}\)) holds in this case.

Next consider the case where *c* does not bump *x* but *c* bumps *y*. Since *c* does not bump *x* then we either have (i) \(c> \underline{x}\) or (ii) *c*≤*x*. If (i) holds, then \(c > \underline{x} > \underline{y}\) which means that *c* cannot bump *y*. Thus (ii) must hold. Since *c* bumps *y*, \(y < c \leq\underline{y}\). Thus we have two possibilities, namely, \(y < c \leq\underline{y} < x\) or \(y < c \leq x \leq\underline{y}\). First suppose that \(y < c \leq\underline{y} < x\). Then cases (I) and (II) are impossible since *x* cannot sit on top of \(\underline{y}\). In case (III), *c* will bump *y* but *y* cannot bump *x* since *y*<*x* so that *y* is inserted in the next cell after (*i*+1,*j*). Next suppose that \(y < c \leq x \leq\underline{y}\). Then in case (*I*), *c* will bump but *y* cannot bump *x* since *y*<*x* so that *y* is inserted in the next cell after (*i*+1,*j*). In case (II), *c* does not bump *x* since *c*≤*x* so that *c* will bump *y* and *y* will be inserted in the next cell after (*i*+1,*j*). In case (III), *c* will bump *y* but *y* cannot bump *x* since *y*<*x* so that *y* is inserted in the next cell after (*i*+1,*j*). Thus in every case, *y* will be inserted in the next cell after (*i*+1,*j*). Hence, condition (\(\mathbb{B}\)) holds in this case.

Next consider the case where in the insertion *w*_{t}→*H*, *c* bumps *x* and then *x* bumps *y* so that the result is that *y* will be inserted into the cell following (*i*+1,*j*) in reading order. In this case, we must have \(y < x \leq \underline{y} < \underline{x}\) and \(x < c \leq \underline{x}\). In case (I), it is easy to see that in the insertion *w*_{t}→*G*, *c* will bump *y* since \(y < x < c \leq\underline{x}\), but *y* will not bump *x* so that the result is that *y* will be inserted into the cell following (*i*+1,*j*) in reading order. In case (II), *c* will bump *x* and then *x* will bump *y* if \(c \leq\underline{y}\). However, if \(c > \underline{y}\), then *c* will not bump *x* but it will bump *y*. Thus in either situation, the result is that *y* will be inserted into the cell following (*i*+1,*j*) in reading order. Finally, consider case (III). If \(c \leq\underline{y}\), then *c* will bump *y* but *y* will not bump *x* so that again the result is that *y* will be inserted into the cell following (*i*+1,*j*) in reading order. Now if \(c > \underline{y}\), then we must have that \(y < x \leq\underline{y} < c \leq\underline{x}\). We claim that this is impossible. Recall that *α*_{i} and *α*_{i+1} are the heights of column *i* and *i*+1 in *G*, respectively. Now if *α*_{i}≥*α*_{i+1}, then \(\{G(i,j) =y, G(i,j-1) = \underline{y}, G(i+1,j)=x\}\) would be a type *A* coinversion triple in *G* and if *α*_{i}<*α*_{i+1}, then \(\{G(i,j-1) = \underline{y}, G(i+1,j)=x, G(i+1,j-1) =\underline {x}\}\) would be a type *B* coinversion triple in *G*.

Finally, consider the case where *c* does not bump either *x* or *y* in the insertion *w*_{t}→*H* so that *c* is inserted into the cells following (*i*+1,*j*) in reading order. Then either *c*≤*y*<*x* so that *c* cannot bump either *x* or *y* in cases (I)–(III) or \(c > \underline{x} > \underline{y}\) so again *c* cannot bump either *x* or *y* in cases (I)–(III). Thus in all cases, the result is that *c* will be inserted into the cells following (*i*+1,*j*) in reading order.

Thus we have shown that conditions (\(\mathbb{A}\)), (\(\mathbb{B}\)), and (ℂ) always hold which, in turn, implies that conditions 3 and 4 always hold. □

Before we proceed, we pause to make one technical remark which will be important for our results in Sect. 5. That is, a careful check of the proof of Theorem 7 will show that we actually proved the following.

### Corollary 8

*Suppose that**σ*=*σ*_{1}⋯*σ*_{n}∈*S*_{n}, *σ*_{i}<*σ*_{i+1}, *and**w*=*w*_{1}…*w*_{t}∈{1,…,*n*}^{t}. *For**j*=1,…,*t*, *let*\(F^{\sigma }_{j} = w_{1} \ldots w_{j} \rightarrow E^{\sigma }\)*and*\(F^{s_{i}\sigma }_{j} = w_{1} \ldots w_{j} \rightarrow E^{s_{i}\sigma }\). *Let*\(F^{\sigma }_{0} = E^{\sigma }\)*and*\(F^{s_{i}\sigma }_{0} = E^{s_{i}\sigma }\). *Let**α*^{(j)}*be the shape of*\(F^{\sigma }_{j}\)*and**β*^{(j)}*be the shape of*\(F^{s_{i}\sigma }_{j}\). *Then for all**i*≥1, *the cells in**α*^{(i)}/*α*^{(i−1)}*and**β*^{(i)}/*β*^{(i−1)}*lie in the same row*.

### Proof

It is easy to prove the corollary by induction on *t*. The corollary is clearly true for *t*=1 since inserting *w*_{1} into either *E*^{σ} or \(E^{s_{i}\sigma }\) will create a new cell in the first row. Then it is easy to check that our proof of Theorem 7 establishing properties (\(\mathbb{A}\)), (\(\mathbb{B}\)), and (ℂ) for the insertions \(w_{t} \rightarrow F^{\sigma }_{t-1}\) and \(w_{t} \rightarrow F^{s_{i}\sigma }_{t-1}\) implies that the cells in *α*^{(t)}/*α*^{(t−1)} and *β*^{(t)}/*β*^{(t−1)} must lie in the same row. □

For any alphabet *A*, we let *A*^{∗} denote the set of all words over the alphabet *A*. If *w*∈{1,…,*n*}^{∗}, then let *P*^{σ}(*w*)=*w*→*E*^{σ}, which we call the *σ*-insertion tableau of *w*, and let \(\gamma^{\sigma }(w) = (\gamma^{\sigma }_{1}(w), \ldots,\gamma^{\sigma }_{n}(w))\) be the composition corresponding to the shape of *P*^{σ}(*w*). Theorem 7 has a number of fundamental consequences about the set of *σ*-insertion tableaux of *w* as *σ* varies over the symmetric group *S*_{n}. Note that \(P^{\bar{\epsilon}_{n}}(w)\) arises from *w* by performing a twisted version of the RSK row insertion algorithm. Hence \(\gamma^{\bar{\epsilon}_{n}}(w)\) is always a partition. Then we have the following corollary.

### Corollary 9

*Suppose that*

*w*∈{1,…,

*n*}

^{∗}.

- 1.
*P*^{σ}(*w*)*is completely determined by*\(P^{\epsilon_{n}}(w)\). - 2.
*For all**σ*∈*S*_{n},*γ*^{σ}(*w*)*is a rearrangement of*\(\gamma^{\bar{\epsilon}_{n}}(w)\). - 3.
*For all**σ*∈*S*_{n},*the set of elements that lie in row**j**of**P*^{σ}(*w*)*equals the set of elements that lie in row**j**of*\(P^{\bar{\epsilon}_{n}}(w)\)*for all**j*≥1.

### Proof

For claim 1, note that if *σ*=*σ*_{1}⋯*σ*_{n} where *σ*_{i}<*σ*_{i+1}, then Theorem 7 tells us that *P*^{σ}(*w*) completely determines \(P^{s_{i}\sigma }(w)\). That is, to obtain \(P^{s_{i}\sigma }(w)\) from *P*^{σ}(*w*), Lemma 1 tells us that we need only ensure that when both (*i*,*j*) and (*i*+1,*j*) are cells in *P*^{σ}(*w*), then the elements in those two cells in *P*^{σ}(*w*) must be arranged in decreasing order in \(P^{s_{i}\sigma }(w)\). If only one of the cells (*i*,*j*) and (*i*+1,*j*) is in *P*^{σ}(*w*), then the element in the cell that is occupied in *P*^{σ}(*w*) must be placed in cell (*i*,*j*) in \(P^{s_{i}\sigma }(w)\). Since we can get from *ϵ*_{n} to any *σ*∈*S*_{n} by applying a sequence of adjacent transpositions where we increase the number of inversions at each step, it follows that *P*^{σ}(*w*) is completely determined by \(P^{\epsilon_{n}}(w)\).

For claims 2 and 3, note that for any *σ*∈*S*_{n}, we can get from *σ* to \(\bar{\epsilon}_{n}\) by applying a sequence of adjacent transpositions where we increase the number of inversions at each step. Thus it follows from Theorem 7 that the set of column heights in \(\gamma^{\bar{\epsilon}_{n}}(w)\) must be a rearrangement of the set of column heights of *γ*^{σ}(*w*).

Moreover, it also follows that the set of elements in row *j* of \(P^{\bar{\epsilon}_{n}}(w)\) must be the same as the set of elements in row *j* of *P*^{σ}(*w*). Note that all the elements in a row of a \(\operatorname {PBF}\) must be distinct by the non-attacking properties of a \(\operatorname {PBF}\). □

*ρ*which takes any \(\operatorname {PBF}\)

*F*with basement equal to

*ϵ*

_{n}to a reverse row strict tableau

*ρ*(

*F*) by simply putting the elements which appear in row

*j*of

*F*(where

*j*≥1) in decreasing order in row

*j*of

*ρ*(

*F*), reading from left to right. This map is pictured at the top of Fig. 17. We can then add a basement below

*ρ*(

*F*) which contains the permutation \(\bar{\epsilon}_{n}\) to obtain a \(\operatorname {PBF}\) with basement equal to \(\bar{\epsilon}_{n}\).

We can extend this map to \(\operatorname {PBF}\)s with an arbitrary basement *σ*. That is, if *F*^{σ} is a \(\operatorname {PBF}\) with basement *σ*∈*S*_{n}, let *ρ*_{σ}(*F*^{σ}) be the \(\operatorname {PBF}\) with basement \(\bar{\epsilon}_{n}\) by simply putting the elements which appear in row *j* of *F*^{σ} in decreasing order in row *j* of *ρ*_{σ}(*F*^{σ}) for *j*≥0, reading from left to right. This map is pictured at the bottom of Fig. 17. To see that *ρ*_{σ}(*F*^{σ}) is a reverse row strict tableau, we need only check that *ρ*_{σ}(*F*^{σ}) is weakly decreasing in columns from bottom to top. But this property is an immediate consequence of the fact that every element in row *j* of *F*^{σ} where *j*≥1 is less than or equal to the element it sits on top of in *F*^{σ}.

The second author [9] showed that for any reverse row strict tableaux *T*, there is a unique \(\operatorname {PBF}\)*F*_{T} with basement equal to *ϵ*_{n} such that *ρ*(*F*_{T})=*T*. Thus *ρ*^{−1} is uniquely defined. In fact, there is a natural procedure for constructing *ρ*^{−1}(*T*). That is, assume that *T* has *k* rows and *P*_{i} is the set of elements of *T* that lie in row *i*.

### Definition of *ρ*^{−1} [9]

Inductively assume that the first *i* rows of *T*, {*P*_{1},…,*P*_{i−1}}, have been mapped to a \(\operatorname {PBF}\)*F*^{(i−1)} with basement *ϵ*_{n} in such a way the elements in row *j* of *F*^{(i−1)} are equal to *P*_{j} for *j*=1,…,*i*−1. Let \(P_{i} =\{\alpha_{1} > \alpha_{2} > \cdots> \alpha_{s_{i}}\}\). There exists an element greater than or equal to *α*_{1} in row *i*−1 since *α*_{1} sits on top of some element in *T*. Place *α*_{1} on top of the left-most such element in row *i*−1 of *F*^{(i−1)}. Next assume that we have placed *α*_{1},…,*α*_{k−1}. Then there are at least *k* elements of *P*_{i−1} that are greater than or equal to *α*_{k} since each of *α*_{1},…,*α*_{k} sit on top of some element in row *i*−1 of *T*. Place *α*_{k} on top of the left-most element in row *i*−1 of *F*^{(i−1)} which is greater than or equal to *α*_{k} which does not have one of *α*_{1},…,*α*_{k−1} on top of it.

*w*∈{1,…,

*n*}

^{∗}. We let

*rr*(

*w*) be the word that results by reading the cells of \(P^{\bar{\epsilon}_{n}}(w)\) in reverse reading order excluding the cells in the basement. Thus

*rr*(

*w*) is just the word which consists of the elements in the first row of \(P^{\bar{\epsilon}_{n}}(w)\) in increasing order, followed by the elements in the second row of \(P^{\bar{\epsilon}_{n}}(w)\) in increasing order, etc. For example, if

*w*=1 3 2 4 3 2 1 4, then \(P^{\bar{\epsilon}_{4}}(w)\) is pictured in Fig. 18 so that

*rr*(

*w*)=1 2 3 4 3 4 1 2 2.

*j*≥1, the elements in the

*j*th row of \(P^{\epsilon_{n}}(rr(w)) = rr(w) \rightarrow E^{\epsilon_{n}}\) is equal to the set of elements in the

*j*th row of \(P^{\epsilon_{n}}(w)\) since both sets are equal to the set of elements in the

*j*th row of \(P^{\bar{\epsilon}_{n}}(w) = P^{\bar{\epsilon}_{n}}(rr(w))\). Thus

*F*with basement

*ϵ*

_{n}such that \(\rho(F) = P^{\bar{\epsilon}_{n}}(w)\), we can conclude that \(P^{\epsilon_{n}}(w) = P^{\epsilon_{n}}(rr(w))\). But then by part 1 of Corollary 9, it must be the case that

*P*

^{σ}(

*w*)=

*P*

^{σ}(

*rr*(

*w*)) for all

*σ*∈

*S*

_{n}. Thus we have the following theorem.

### Theorem 10

- 1.
*If**u*,*v*∈{1,…,*n*}^{∗}*and*\(P^{\bar{\epsilon}_{n}}(w) = P^{\bar{\epsilon}_{n}}(u)\),*then**P*^{σ}(*u*)=*P*^{σ}(*w*)*for all**σ*∈*S*_{n}. - 2.
*For any*\(\operatorname {PBF}\)*T**with basement equal to*\(\bar{\epsilon}_{n}\)*and any**σ*∈*S*_{n},*there is a unique*\(\operatorname {PBF}\)*F*^{σ}*with basement**σ**such that**ρ*_{σ}(*F*^{σ})=*T*.

Theorem 10 says that we can construct *P*^{σ}(*w*)=*w*→*E*^{σ} by first constructing \(P^{\bar{\epsilon}_{n}}(w) = w \rightarrow E^{\bar{\epsilon}_{n}}\) by our twisted version of RSK row insertion, then find *rr*(*w*) which is the reverse reading word of \(P^{\bar{\epsilon}_{n}}(w)\), and then compute *rr*(*w*)→*E*^{σ}. However, it is easy to construct \(rr(w) \rightarrow E^{\sigma }= \rho_{\sigma }^{-1}(P^{\bar {\epsilon}_{n}}(w))\). That is, suppose that *w*=*w*_{1}*w*_{2}…*w*_{s} is the strictly increasing word that results by reading the first row of \(P^{\bar{\epsilon}_{n}}(w)\) in reverse reading order. Now consider inserting *w*=*w*_{1}*w*_{2}…*w*_{s} into *E*^{σ}. It is easy to see *w*_{s} will end up sitting on top of *σ*_{i} in the basement where *i* is the least *j* such that *σ*_{j}≥*w*_{s}. Next consider the entry *w*_{s−1}. Before the insertion of *w*_{s}, *w*_{s−1} sat on top *σ*_{a} in the basement where *a* is the least *b* such that *σ*_{b}≥*w*_{s−1}. Now if *a* equals *i*, then *w*_{s} will bump *w*_{s−1} and *w*_{s−1} will move to *σ*_{c} where *c* is the least *d*>*i* such that *σ*_{d}≥*w*_{s−1}. Thus once *w*_{s} is placed, *w*_{s−1} will be placed on top of *σ*_{a} in the basement where *a* is the least *b* such that *σ*_{b}≥*w*_{s−1} and *w*_{s} is not on top of *σ*_{b}. We continue this reasoning and show that *w*=*w*_{1}*w*_{2}…*w*_{s}→*E*^{σ} can be constructed inductively as follows.

**Procedure to construct**\(\rho_{\sigma }^{-1}(P^{\bar{\epsilon}_{n}}(w)) = rr(w) \rightarrow E^{\sigma }\).

**Step 1**. Let *w*_{1}…*w*_{s} be the first row of \(P^{\bar{\epsilon}_{n}}(w)\) in increasing order. First, place *w*_{s} on top of *σ*_{i} in the basement where *i* is the least *j* such that *σ*_{j}≥*w*_{s}. Then having placed *w*_{s},…,*w*_{r+1}, place *w*_{r} on top *σ*_{u} in the basement where *u* is the least *v* such that *σ*_{v}≥*w*_{r} and none of *w*_{s},…,*w*_{r+1} are on top of *σ*_{v}.

**Step***i*>0. Inductively assume that the first *i* rows of \(P^{\bar{\epsilon}_{n}}(w)\), {*P*_{1},…,*P*_{i−1}}, have been mapped to a \(\operatorname {PBF}\)*F*^{(i−1)} with basement *σ* in such a way that the elements in row *j* of *F*^{(i−1)} are equal to *P*_{j} for *j*=1,…,*i*−1. Let \(P_{i} =\{\alpha_{1} > \alpha_{2} > \cdots> \alpha_{s_{i}}\}\) be the *i*th row of \(P^{\bar{\epsilon}_{n}}(w)\). There exists an element greater than or equal to *α*_{1} in row *i*−1 since there *α*_{1} sits on top of some element in \(P^{\bar{\epsilon}_{n}}(w)\). Place *α*_{1} on top of the left-most such element in row *i*−1 of *F*^{(i−1)}. Next assume that we have placed *α*_{1},…,*α*_{k−1}. Then there are at least *k* elements of *P*_{i−1} that are greater than or equal to *α*_{k} since each of *α*_{1},…,*α*_{k} sit on top of some element in row *i*−1 of \(P^{\bar{\epsilon}_{n}}(w)\). Place *α*_{k} on top of the left-most element in row *i*−1 of *F*^{(i−1)} which is greater than or equal to *α*_{k} which does not have one of *α*_{1},…,*α*_{k−1} on top of it.

We then have the following theorem which shows that the insertion *w*→*E*^{σ} can be factored through the twisted RSK row insertion algorithm used to construct \(w \rightarrow E^{\bar{\epsilon}_{n}}\).

### Theorem 11

*If**w*∈{1,…,*n*}^{∗}*and**σ*∈*S*_{n}, *then**P*^{σ}(*w*)=*w*→*E*^{σ}*equals*\(\rho_{\sigma }^{-1}(P^{\bar{\epsilon}_{n}}(w))\)*where*\(P^{\bar{\epsilon}_{n}}(w) = w \rightarrow E^{\bar{\epsilon}_{n}}\).

*u*,

*v*∈{1,2,…}

^{∗}and

*x*,

*y*,

*z*∈{1,2,…}. The two types of Knuth relations are

- 1.
*uyxzv*∽*uyzxv*if*x*<*y*≤*z*and - 2.
*uxzyv*∽*uzxyv*if*x*≤*y*<*z*.

*w*,

*w*′∈{1,2,…}

^{∗}are Knuth equivalent,

*w*∽

*w*′, if

*w*can be transformed into

*w*′ by repeated use of relations 1 and 2. If

*w*∽

*w*′, then

*w*and

*w*′ give us the same insertion tableau under row insertion. In our twisted version of row insertion the two types of Knuth relations become

- 1
^{∗} *uyxzv*∽^{∗}*uyzxv*if*z*≤*y*<*x*and- 2
^{∗} *uxzyv*∽^{∗}*uzxyv*if*z*<*y*≤*x*.

*w*,

*w*′∈{1,2,…,

*n*}

^{∗}are twisted Knuth equivalent,

*w*∽

^{∗}

*w*′, if

*w*can be transformed to

*w*′ by repeated use of relations 1

^{∗}and 2

^{∗}. Therefore, if

*w*∽

^{∗}

*w*′, then \(P^{\bar{\epsilon}_{n}}(w) = P^{\bar{\epsilon}_{n}}(w^{\prime})\). Then Theorem 11 immediately implies the following.

### Theorem 12

*Suppose that**w*,*w*′∈{1,2,…,*n*}^{∗}*and**w*∽^{∗}*w*′. *Then for all**σ*∈*S*_{n}, *P*^{σ}(*w*)=*P*^{σ}(*w*′).

It also follows from Theorem 11 that for every partition *γ*, the map \(\rho_{\sigma }^{-1}\) gives a one-to-one correspondence between the set of reverse row strict tableaux of shape *γ* whose entries are less than or equal to *n* and the set of \(\operatorname {PBF}\)s with basement *σ* whose entries are less than or equal to *n* and whose shape (*δ*_{1},…,*δ*_{n}) is a rearrangement of *γ* compatible with basement *σ*. That is, we say that a weak composition *δ*=(*δ*_{1},…,*δ*_{n}) *is compatible with basement**σ*=*σ*_{1}⋯*σ*_{n}∈*S*_{n} if *δ*_{i}≥*δ*_{j} whenever *σ*_{i}>*σ*_{j} and *i*<*j*. Note that Lemma 1 implies that the shape of any \(\operatorname {PBF}\)*F*^{σ} with entries from {1,…,*n*} and basement *σ* must have a shape which is a weak composition compatible with basement *σ*. Then we have the following theorem.

### Theorem 13

*Let*

*λ*=(

*λ*

_{1},…,

*λ*

_{n})

*be a partition of*

*n*.

*Then*

*where the sum runs over all weak compositions*

*δ*=(

*δ*

_{1},…,

*δ*

_{n})

*which are rearrangements of*

*λ*

*that are compatible with basement*

*σ*.

*s*

_{(2,1,0)}(

*x*

_{1},

*x*

_{2},

*x*

_{3}). In Fig. 19, we have listed the eight \(\operatorname {PBF}\)s with basement \(\bar{\epsilon}_{3} = 3~2~1\) over the alphabet {1,2,3}. Below each of these \(\operatorname {PBF}\)s

*G*, we have pictured \(\rho^{-1}_{123}(G)\), \(\rho^{-1}_{132}(G)\) and \(\rho^{-1}_{312}(G)\). One can see that and

*σ*∈

*S*

_{n}and a partition

*λ*of

*n*, then we can view the set of generalized Demazure atoms

*λ*. That is, let

*RRT*(

*λ*) denote the set of reverse row strict tableaux of shape

*λ*with entries from {1,…,

*n*}. Then if

*λ*(

*γ*)=

*λ*and

*γ*is compatible with basement

*σ*, we can identify \(\widehat{E}^{\sigma }_{\gamma}(x_{1}, \ldots, x_{n})\) with the set

*σ*such

*ρ*

_{σ}(

*P*)=

*T*for any reverse row strict tableau

*T*of shape

*λ*with entries from {1,…,

*n*} implies that

*RRT*(

*λ*). For example, if

*T*

_{1},…,

*T*

_{8}are the reverse row strict tableaux with basement 321 pictured at the top of Fig. 19, reading from left to right, then Then the collection of such set partitions \(\mathcal{STP}_{\lambda}= \{Stp^{\sigma }_{\lambda}: \sigma \in S_{n}\}\) can be partially ordered by refinement. We can show that if

*σ*<

_{L}

*τ*in the (left) weak Bruhat order on

*S*

_{n}, then \(Stp^{\sigma }_{\lambda}\) is a refinement of \(Stp^{\tau}_{\lambda}\). Moreover, if

*λ*has

*n*distinct parts then \(\mathcal{STP}_{\lambda}\) under refinement is isomorphic to the (left) weak Bruhat order on

*S*

_{n}. These results will appear in a subsequent paper [2].

We end this section with a simple characterization for when \(\widehat{E}^{\sigma }_{\alpha}(x_{1}, \ldots, x_{n}) \neq0\).

### Proposition 14

*Suppose that**α*=(*α*_{1},…,*α*_{n}) *is a weak composition of length**n**and**σ*=*σ*_{1}⋯*σ*_{n}∈*S*_{n}. *Then*\(\widehat{E}^{\sigma }_{\alpha}(x_{1}, \ldots, x_{n}) \neq0\)*if and only if**α**is compatible with basement**σ*.

### Proof

If \(\widehat{E}^{\sigma }_{\alpha}(x_{1}, \ldots, x_{n}) \neq0\), then there must be a \(\operatorname {PBF}\)*F*^{σ} of shape *α* with basement *σ*. Then Lemma 1 tells us that if 1≤*i*<*j*≤*n* and *σ*_{i}>*σ*_{j}, then *α*_{i}≥*α*_{j} so that *α* is compatible with basement *σ*.

Vice versa, suppose that *α* is compatible with basement *σ*. Then let *F*^{σ} be the filling of \(\hat{dg}(\alpha)\) such that the elements in column *i* are all equal to *σ*_{i}. The elements of *F*^{σ} are weakly increasing in columns, reading from top to bottom. If 1≤*i*<*j*≤*n* and *α*_{i}<*α*_{j}, then we know that *σ*_{i}<*σ*_{j} so that *F*^{σ} automatically satisfies the *B*-increasing condition. Finally, if *α*_{i}≥*α*_{i} and *a*=(*i*,*y*), *b*=(*j*,*y*) and *c*=(*i*,*j*−1) is a type *A* triple, then we cannot have *F*^{σ}(*a*)≤*F*^{σ}(*b*)≤*F*^{σ}(*c*), so every type *A* triple in *F*^{σ} will be an inversion triple. Thus *F*^{σ} is a \(\operatorname {PBF}\) of shape *α* with basement *σ* so that \(\widehat{E}^{\sigma }_{\alpha}(x_{1}, \ldots, x_{n}) \neq0\). □

## 5 Pieri rules

*h*

_{k}(

*x*

_{1},…,

*x*

_{n}) and the elementary symmetric function

*e*

_{k}(

*x*

_{1},…,

*x*

_{n}) are defined by

*λ*of

*k*+|

*μ*| such that

*μ*⊆

*λ*and

*λ*/

*μ*does not contain two elements in the same column and that

*λ*of

*k*+|

*μ*| such that

*μ*⊆

*λ*and

*λ*/

*μ*does not contain two elements in the same row. In our case, we think of the Schur function

*s*

_{μ}(

*x*

_{1},…,

*x*

_{n}) as \(\widehat{E}^{\bar{e}_{n}}_{\mu}(x_{1}, \ldots, x_{n})\). Since we work with reverse row strict tableaux

*T*,

*μ*corresponds to the column heights of the \(\operatorname {PBF}\)\(T^{\bar{\epsilon}_{n}}\). Thus we say that

*λ*/

*μ*is a

*transposed skew row*of size

*k*if

*dg*′(

*μ*)⊆

*dg*′(

*λ*), |

*λ*|=

*k*+|

*μ*| and no two elements in

*dg*′(

*λ*/

*μ*) lie in the same column. Similarly, we say that

*λ*/

*μ*is a

*transposed skew column*of size

*k*if

*dg*′(

*μ*)⊆

*dg*′(

*λ*), |

*λ*|=

*k*+|

*μ*| and no two elements in

*dg*′(

*λ*/

*μ*) lie in the same row. Thus in this language, the Pieri rules become

*λ*such that

*μ*⊆

*λ*and

*λ*/

*μ*is a transposed skew row of size

*k*and

*λ*such that

*μ*⊆

*λ*and

*λ*/

*μ*is a transposed skew column of size

*k*.

The main goal of this section is to prove an analogue of the Pieri rules (7) and (8) for the products \(h_{k}(x_{1}, \ldots x_{n}) \widehat{E}^{\sigma }_{\gamma}(x_{1}, \ldots, x_{n})\) and \(e_{k}(x_{1}, \ldots x_{n}) \widehat{E}^{\sigma }_{\gamma}(x_{1}, \ldots, x_{n})\).

We start with a simple lemma about the effect of inserting two letters into a \(\operatorname {PBF}\). If *α* and *β* are weak compositions of length *n* and *dg*′(*α*)⊆*dg*′(*β*), then *dg*′(*β*/*α*) will denote the cells of *dg*′(*β*) which are not in *dg*′(*α*).

### Lemma 15

*Suppose that*

*F*

^{σ}

*is a*\(\operatorname {PBF}\),

*G*

^{σ}=

*k*→

*F*

^{σ}

*and*

*H*

^{σ}=

*k*′→

*G*

^{σ}.

*Suppose*

*F*

^{σ}

*is of shape*

*α*,

*G*

^{σ}

*is of shape*

*β*,

*H*

^{σ}

*is of shape*

*γ*,

*T*

*is the cell in*

*dg*′(

*β*/

*α*),

*and*

*T*′

*is the cell in*

*dg*′(

*γ*/

*β*).

*Then*

- 1.
*If**k*≥*k*′,*then**T**is strictly below**T*′*and* - 2.
*If**k*<*k*′,*then**T**appears before**T*′*in reading order*.

### Proof

There is no loss in generality in assuming that *σ*=*σ*_{1}⋯*σ*_{n}∈*S*_{n} where *n*≥max(*k*,*k*′). Assume \(\bar{F}^{\sigma }\) is the diagram that results by adding 0s on top of the cells of *F*^{σ} as in the definition of the insertion *k*→*F*^{σ}. Let *c*_{1},*c*_{2},… be the cells in reading order that are in \(\bar{F}^{\sigma }\) but not in the basement. We will prove this result by induction on the number of cells *p* in the list *c*_{1},*c*_{2},….

First, suppose that *p*=0 so that *F*^{σ} just consists of the basement permutation *σ* and thus the cells *c*_{1},*c*_{2},… are simply the zero entries on top of the basement in \(\bar{F}^{\sigma }\). Then *k* will be inserted in cell (*i*,1) where *i* is the least *j* such that *k*≤*σ*_{j}. Now if *k*′≤*k*, then it is easy to see that *k*′ will be inserted on top of *k* in the insertion *k*′→*G*^{σ} so that *T* will be strictly below *T*′.

If *k*′>*k*, then suppose that *k* is in cell (*i*,1) in *G*^{σ}. Then it is clear that *k*′ cannot be placed in any of the cells (*j*,1) with *j*<*i* since *k* could not be placed in any of those cells. Hence *k*′ either bumps *k* or is placed in the first row in some cell to the right of *k*. In either case, *T* precedes *T*′ in reading order.

Now if *p*>0, there are two cases.

**Case 1**. *k* is placed in cell *c*_{i} which is either equal to *c*_{1} or in the same row as *c*_{1}.

In either case, *c*_{i} is a cell on top of a column in \(\overline{F}^{\sigma }\). Let \(\overline{c_{i}}\) be the cell immediately above *c*_{i}. Then cell \(\overline{c_{i}}\) will be the first cell in reading order in \(\bar{G}^{\sigma }\). If *k*′≤*k*, then *k*′ will be placed in \(\overline{c_{i}}\) so that *c*_{i}=*T* and \(\overline{c_{i}} =T^{\prime}\). Thus *T* will occur below *T*′.

If *k*<*k*′, then *k*′ cannot be placed in \(\overline{c_{i}}\). Moreover, *k*′ cannot bump any of the entries in cells *c*_{1},…,*c*_{i−1} since *k* does not bump any of those elements. That is, for 1≤*j*<*i*, let \(\underline{c}_{j}\) be the cell immediately below *c*_{j}. Then the reason that *k* does not bump *F*^{σ}(*c*_{j}) was that \(k > F^{\sigma }(\underline{c}_{j})\) in which case \(k' > F^{\sigma }(\underline{c}_{j})\) since the entries in cells *c*_{1},…,*c*_{i−1} are all zero. Thus either *k*′ bumps *k* in cell *c*_{i} or it is inserted into cells of \(\bar{F}^{\sigma }\) after *c*_{i}. In either case, it is easy to see that *T*′ must follow *c*_{i}=*T* in reading order.

**Case 2**. *k* is placed in a cell *c*_{i} which is not in the same row as *c*_{1}.

Let *c*_{j} be the first cell in our list which is not in the same row as cell *c*_{1}. If *k*′ is not placed in any of the cells *c*_{1},…*c*_{j−1}, then we are inserting *k* followed by *k*′ into the sequence \(\bar{F}^{\sigma }(c_{j}), \bar{F}^{\sigma }(c_{j}+1), \ldots\) so the result follows by induction. However, the only way that *k*′ can be placed in a cell *c*_{i} in the same row as *c*_{1} is if *k*′<*k* in which case *T*′ = *c*_{i}. In that case, *T* lies in a row below the row of *c*_{1} so that *T* lies strictly below *T*′. □

Suppose that *γ* and *δ* are weak compositions such that *dg*′(*γ*)⊆*dg*′(*δ*) and *dg*′(*δ*/*γ*) consists of a single cell *c*=(*x*,*y*). Then we say that *c* is a *removable cell* from *δ* if there is no *j* such that *x*<*j*≤*ℓ*(*δ*) and *δ*_{j}=1+*y*. The idea is that if *F*^{σ} is a \(\operatorname {PBF}\) of shape *γ* and basement *σ*, and *δ* is the shape of *k*→*F*^{σ}, then Proposition 6 tells us that the cell *c* in *dg*′(*δ*/*γ*) must be a removable cell.

*γ*and

*δ*are weak compositions of length

*m*such that

*dg*′(

*γ*) is contained in

*dg*′(

*δ*) and

*σ*=

*σ*

_{1}⋯

*σ*

_{m}is a permutation in

*S*

_{m}. Let

*c*

_{1}=(

*x*

_{1},

*y*

_{1}),…,

*c*

_{k}=(

*x*

_{k},

*y*

_{k}) be the cells of

*dg*′(

*δ*/

*γ*) listed in reverse reading order. Let

*dg*′(

*δ*

^{(i)}) consist of the diagram of

*γ*plus the cells

*c*

_{1},…,

*c*

_{i}. Then we say that

*δ*/

*γ*is a

*γ*-

*transposed skew row relative to basement*

*σ*if

- 1.
*y*_{1}<*y*_{2}<⋯<*y*_{k}, - 2.
For

*i*=1,…,*k*,*dg*′(*δ*^{(i)}) is the diagram of weak composition*δ*^{(i)}which is compatible with basement*σ*, - 3.
*dg*′(*γ*)⊂*dg*′(*δ*^{(1)})⊂*dg*′(*δ*^{(2)})⊂⋯⊂*dg*′(*δ*^{(k)}), and - 4.
*c*_{i}is a removable square from*δ*^{(i)}for*i*=1,…,*k*.

*γ*and

*ϵ*are weak compositions of length

*m*such that

*dg*′(

*γ*) is contained in

*dg*′(

*ϵ*). Let

*d*

_{1}=(

*x*

_{1},

*y*

_{1}),…,

*d*

_{k}=(

*x*

_{k},

*y*

_{k}) be the cells of

*dg*′(

*ϵ*/

*γ*) listed in reading order. Let

*dg*′(

*ϵ*

^{(i)}) consist of the diagram of

*γ*plus the cells

*d*

_{1},…,

*d*

_{i}. We say that

*ϵ*/

*γ*is a

*γ*-

*transposed skew column relative to basement*

*σ*if

- 1.
For

*i*=1,…,*k*,*dg*′(*ϵ*^{(i)}) is the diagram of weak composition*ϵ*^{(i)}which is compatible with basement*σ*, - 2.
*dg*′(*γ*)⊂*dg*′(*ϵ*^{(1)})⊂*dg*′(*ϵ*^{(2)})⊂⋯⊂*dg*′(*ϵ*^{(k)}), and - 3.
*d*_{i}is a removable square from*ϵ*^{(i)}for*i*=1,…,*k*.

*m*=9,

*σ*=127346589, and

*γ*=(2,0,3,1,1,3,1,0,0), then, in Fig. 20, we have pictured a

*γ*-transposed skew row relative to basement

*σ*in the top left and a

*γ*-transposed skew column relative to basement

*σ*in the bottom left. The diagram on the top right is not a

*γ*-transposed skew column relative to basement

*σ*since

*c*

_{3}is not a removable cell from

*δ*

^{(3)}=(2,0,3,3,1,3,1,1,0) and the diagram on the bottom right is not a

*γ*-skew row since the diagram consisting of

*γ*plus cells

*d*

_{1}and

*d*

_{2}does not correspond to the diagram of a weak composition. It is easy to check that if \(\sigma = \bar{\epsilon}_{n}\) and

*γ*is of partition shape, then a

*γ*-transposed skew row

*δ*/

*γ*relative to basement \(\bar{\epsilon}_{n}\) implies that

*δ*is a partition containing

*γ*such that no two cells

*δ*/

*γ*can lie in the same column. In this case, the removable cell condition is automatic since there are no cells to the right of any cell in

*δ*/

*γ*. Similarly, it is easy to check that a

*γ*-transposed skew column

*ϵ*/

*γ*relative to basement \(\bar{\epsilon}_{n}\) is just a transposed skew column.

### Theorem 16

*Let*

*γ*=(

*γ*

_{1},…,

*γ*

_{n})

*be a weak composition of*

*p*

*and*

*σ*∈

*S*

_{n}.

*Then*

*where the sum runs over all weak compositions*

*δ*=(

*δ*

_{1},…,

*δ*

_{n})

*of size*

*p*+

*k*

*such that*

*dg*′(

*γ*)⊆

*dg*′(

*δ*)

*and*

*δ*/

*γ*

*is a*

*γ*-

*transposed skew row relative to basement*

*σ*.

*where the sum runs over all weak compositions*

*ϵ*=(

*ϵ*

_{1},…,

*ϵ*

_{n})

*of size*

*p*+

*k*

*such that*

*dg*′(

*γ*)⊆

*dg*′(

*ϵ*)

*and*

*ϵ*/

*γ*

*is a*

*γ*-

*transposed skew column relative to basement σ*.

### Proof

The left hand side of (9) can be interpreted as the weight of the set of pairs (*w*,*F*^{σ}) where *w*=*w*_{1}…*w*_{k} and *n*≥*w*_{1}≥⋯≥*w*_{k}≥1, *F*^{σ} is a \(\operatorname {PBF}\) of shape *γ* with basement *σ*, and the weight *W*(*w*,*F*^{σ}) of the pair (*w*,*F*^{σ}) is equal to \(W(F^{\sigma })= \prod_{i=1}^{k} x_{w_{i}}\). The right hand side of (9) can be interpreted as the sum of the weights of all \(\operatorname {PBF}\)s *G*^{σ} with basement *σ* such *G*^{σ} has shape *δ*=(*δ*_{1},…,*δ*_{n}) for some *δ* which is a weak composition of size *p*+*k* such that *δ*/*γ* is a *γ*-transposed skew row relative to basement *σ*.

*Θ*which takes such a pair (

*w*,

*F*

^{σ}) to

*i*=1,…,

*k*and let \(G^{\sigma }_{0} =F^{\sigma }\). Let

*δ*

^{(i)}be the shape of \(G^{\sigma }_{i}\). It follows that each

*δ*

^{(i)}is a weak composition of size

*p*+

*i*which is compatible with basement

*σ*. Then let

*c*

_{i}be the cell in

*dg*′(

*δ*

^{(i)}/

*δ*

^{(i−1)}) for

*i*=1,…,

*k*. By Lemma 15, we know that

*c*

_{i+1}must be strictly above

*c*

_{i}for

*i*=1,…,

*k*−1. Also, by Proposition 6, we know that

*c*

_{i}must be a removable cell for

*δ*

^{(i)}. It follows that

*G*

^{σ}is a \(\operatorname {PBF}\) of some shape

*δ*such that

*δ*/

*γ*is a

*γ*-transposed skew row relative to basement

*σ*. Moreover, it is clear that

*W*(

*G*

^{σ})=

*W*(

*w*,

*F*

^{σ}). Since our insertion procedure can be reversed, it is easy to see that

*Θ*is one-to-one.

To see that *Θ* is a bijection between the pairs (*w*,*F*^{σ}) contributing to the left hand side of (9) and the \(\operatorname {PBF}\)s *G*^{σ} contributing to the right hand side (9), we must show that for each *G*^{σ} contributing to the right hand side (9), there is a pair (*w*,*F*^{σ}) contributing to the left hand side (9) such that *w*→*F*^{σ}=*G*^{σ}. Suppose that *G*^{σ} is a \(\operatorname {PBF}\) with basement *σ* such that *G*^{σ} has shape *δ*=(*δ*_{1},…,*δ*_{n}) where *δ*/*γ* is *γ*-transposed skew row relative to basement *σ*. Let *c*_{k},…,*c*_{1} be the cells of *dg*′(*α*/*γ*) reading from top to bottom. Because *c*_{k} is a removable square for *δ*, it follows from our remarks following Proposition 6 that we can reverse the insertion procedure starting at cell *c*_{k}. Similarly, *c*_{k−1} is a removable cell for shape consisting of *δ* with *c*_{k} removed, and, in general, *c*_{i} must be a removable cell for the shape of *δ* with *c*_{k},…,*c*_{i+1} removed. Thus we can first reverse the insertion process for the element in cell *c*_{k} in *G*^{σ} to produce a \(\operatorname {PBF}\)\(F^{\sigma }_{k-1}\) with basement *σ* and shape *δ* with cell *c*_{k} removed and a letter *w*_{k} such that \(w_{k} \rightarrow F^{\sigma }_{k-1} = G^{\sigma }\). Then we can reverse our insertion process for the element in cell *c*_{k−1} of \(F^{\sigma }_{k-1}\) to produce a \(\operatorname {PBF}\)\(F^{\sigma }_{k-2}\) with basement *σ* and shape *δ* with cells *c*_{k} and *c*_{k−1} removed and a letter *w*_{k−1} such that \(w_{k-1}w_{k} \rightarrow F^{\sigma }_{k-2} = G^{\sigma }\). Continuing on in this manner we can produce a sequence of \(\operatorname {PBF}\)s \(F^{\sigma }_{0}, \ldots, F^{\sigma }_{k-1}\) with basement *σ* and a word *w*=*w*_{1}…*w*_{k} such that \(w_{i} \ldots w_{k} \rightarrow F^{\sigma }_{i-1} =G^{\sigma }\) and the shape of \(F^{\sigma }_{i-1}\) equals *δ* with the cells *c*_{i},*c*_{i+1},…,*c*_{k} removed. Thus \(F^{\sigma }_{0}\) will be a \(\operatorname {PBF}\) with basement *σ* and shape *γ* such that \(w \rightarrow F^{\sigma }_{0} =G^{\sigma }\). The only thing that we have to prove is that *w*_{1}≥⋯≥*w*_{k}. But it cannot be that *w*_{i}<*w*_{i+1} for some *i* because Lemma 15 would imply that *c*_{i} appears before *c*_{i+1} in reading order which it does not. Thus *Θ* is a bijection which proves that (9) holds.

The left hand side of (10) can be interpreted as the weight of the set of pairs (*u*,*H*^{σ}) where *u*=*u*_{1}…*u*_{k} and 1≤*u*_{1}<⋯<*u*_{k}≤*n*, *H*^{σ} is a \(\operatorname {PBF}\) of shape *γ* with basement *σ*, and the weight *W*(*u*,*H*^{σ}) of the pair (*u*,*H*^{σ}) is equal to \(W(H^{\sigma })\prod_{i=1}^{k} x_{u_{i}}\). The right hand side of (10) can be interpreted as the sum of the weights of all \(\operatorname {PBF}\)s *K*^{σ} with basement *σ* such that *K*^{σ} has shape *ϵ*=(*ϵ*_{1},…,*ϵ*_{n}) for some weak composition *ϵ* of size *p*+*k* such that *ϵ*/*γ* is a *γ*-transposed skew column relative to basement *σ*.

*Θ*which takes such a pair (

*u*,

*H*

^{σ}) to

*i*=1,…,

*k*and let \(K^{\sigma }_{0} =H^{\sigma }\). Then let

*ϵ*

^{(i)}be the shape of \(K^{\sigma }_{i}\) so that

*ϵ*

^{(i)}is a weak composition of size

*p*+

*i*which is compatible with basement

*σ*. Then let

*d*

_{i}be the cell in

*dg*′(

*ϵ*

^{(i)}/

*ϵ*

^{(i−1)}) for

*i*=1,…,

*k*. By Lemma 15, we know that

*d*

_{i}must appear before

*d*

_{i+1}in reading order for

*i*=1,…,

*k*−1. Moreover,

*d*

_{i}must be a removable cell from

*ϵ*

^{(i)}by Proposition 6.

It follows that *K*^{σ} is a \(\operatorname {PBF}\) of some shape *ϵ*=(*ϵ*_{1},…,*ϵ*_{n}) such that *ϵ*/*γ* is a *γ*-transposed skew column relative to basement *σ*. Moreover, it is clear that *W*(*G*^{σ})=*W*(*w*,*F*^{σ}). Since our insertion procedure can be reversed, it is easy to see that *Θ* is one-to-one.

To see that *Θ* is a bijection between the pairs (*u*,*H*^{σ}) contributing to the left hand side of (10) and the \(\operatorname {PBF}\)s *K*^{σ} contributing to the right hand side (10), we must show that for each *K*^{σ} contributing to the right hand side of (10), there is a pair (*u*,*H*^{σ}) contributing to the left hand side of (10) such that *u*→*H*^{σ}=*K*^{σ}. So suppose that *K*^{σ} is a \(\operatorname {PBF}\) with basement *σ* such that *K*^{σ} has shape *ϵ*=(*ϵ*_{1},…,*ϵ*_{n}) such that *ϵ*/*γ* is a *γ*-transposed skew column relative to basement *σ* of size *k*. Let *d*_{k},…,*d*_{1} be the cells of *dg*′(*ϵ*/*γ*) read in reverse reading order. Since *d*_{k} is a removable cell for *ϵ*, we can reverse our insertion process starting at cell *d*_{k}. Similarly, *d*_{k−1} is removable cell for shape consisting of *ϵ* with *c*_{k} removed, and, in general, *d*_{i} must be a removable cell for the shape of *ϵ* with *d*_{k},…,*d*_{i+1} removed. This means that we can reverse our insertion process staring with cell *d*_{i} after we have reversed the insertion process starting at cells *d*_{k},…,*d*_{i+1}. Then we first reverse the insertion process for the element in cell *d*_{k} in *K*^{σ} to produce a \(\operatorname {PBF}\)\(H_{k-1}^{\sigma }\) with basement *σ* and shape *ϵ* with cell *d*_{k} removed and a letter *u*_{k} such that \(u_{k} \rightarrow H^{\sigma }_{k-1} = K^{\sigma }\). Then we can reverse our insertion process for the element in cell *d*_{k−1} of \(H^{\sigma }_{k-1}\) to produce a \(\operatorname {PBF}\)\(H^{\sigma }_{k-2}\) with basement *σ* and shape *ϵ* with cells *d*_{k} and *d*_{k−1} removed and a letter *u*_{k−1} such that \(u_{k-1}u_{k} \rightarrow H^{\sigma }_{k-2} = K^{\sigma }\). Continuing in this manner, we can produce a sequence of \(\operatorname {PBF}\)s \(H^{\sigma }_{0}, \ldots, H^{\sigma }_{k-1}\) with basement *σ* and a word *u*=*u*_{1}…*u*_{k} such that \(u_{i} \ldots u_{k} \rightarrow H^{\sigma }_{i-1} =G^{\sigma }\) and the shape of \(H^{\sigma }_{i-1}\) equals *ϵ* with the cells *d*_{i},*d*_{i+1},…,*d*_{k} removed. Thus \(H^{\sigma }_{0}\) will be a \(\operatorname {PBF}\) with basement *σ* and shape *γ* such that \(u \rightarrow H^{\sigma }_{0} =K^{\sigma }\). The only thing that we have to prove is that *u*_{1}<⋯<*u*_{k}. But it cannot be that *u*_{i}≥*u*_{i+1} for some *i* because Lemma 15 would force *d*_{i+1} to appear in a row which is strictly above the row in which *d*_{i} appears which would mean that *d*_{i+1} does not follow *d*_{i} in reading order. Thus *Θ* is a bijection which proves that (10) holds. □

We can show that there is an analogue of the Littlewood–Richardson rule for the product of a Schur function *s*_{λ}(*x*_{1},…,*x*_{n}) times a \(\widehat{E}^{\sigma }_{\gamma}(x_{1}, \ldots, x_{n})\) for all *γ* and *σ*∈*S*_{n}. This rule will appear in a subsequent paper.

## 6 A permuted basements analogue of the Robinson–Schensted–Knuth algorithm

We are now ready to state an analogue of the Robinson–Schensted–Knuth Algorithm for \(\operatorname {PBF}\)s.

*A*=(

*a*

_{i,j}) be an arbitrary

*n*×

*n*-matrix with nonnegative integer entries and let

*σ*=

*σ*

_{1}⋯

*σ*

_{n}∈

*S*

_{n}. For each pair

*i*,

*j*such that

*a*

_{i,j}>0, create a sequence of

*a*

_{i,j}biletters \(\begin{array}{c} \scriptstyle i \\[-3pt] \scriptstyle j \end{array} \). Let

*w*

_{A}be the unique two-line array consisting of such biletters so the top letters are weakly increasing and for all pairs with the same top letter the bottoms letters are weakly increasing. For example, if then Let

*u*

_{A}be the word consisting of the top row of

*w*

_{A}and

*v*

_{A}be the word consisting of the bottom row of

*w*

_{A}. Let \({P}^{\sigma}_{0} = Q^{\sigma}_{0} = E^{\sigma }\) be empty \(\operatorname {PBF}\)s with basement

*σ*. We say that \({P}^{\sigma}_{0}\) is the initial insertion \(\operatorname {PBF}\) and \({Q}^{\sigma}_{0}\) is the initial recording \(\operatorname {PBF}\) relative to

*σ*.

*u*

_{A}=

*i*

_{1}…

*i*

_{t}and

*v*

_{A}=

*j*

_{1}…

*j*

_{t}. Then insert the biletters of

*w*

_{A}into the insertion and recording \(\operatorname {PBF}\)s using the following inductive procedure. Assume that the last

*k*biletters of

*w*

_{A}have already been inserted and the resulting pair of \(\operatorname {PBF}\)s is \((\mbox {${P_{k}}^{\sigma }$}, \mbox {${Q_{k}}^{\sigma }$})\) such that the partitions obtained by rearranging the shapes of \(\mbox {${P_{k}}^{\sigma }$}\) and \(\mbox {${Q_{k}}^{\sigma }$}\) are the same. Insert the entry

*j*

_{t−k}into \(\mbox {${P_{k}}^{\sigma }$}\) according to the procedure \(j_{t-k} \rightarrow \mbox {${P_{k}}^{\sigma }$}\). Suppose the new cell created in the insertion \(j_{t-k} \rightarrow \mbox {${P_{k}}^{\sigma }$}\) lies in row

*r*. Record the position of the new entry by placing the entry

*i*

_{t−k}into the leftmost empty cell in row

*r*of \(\mbox {${Q_{k}}^{\sigma }$}\) which lies immediately above a cell greater than or equal to

*i*

_{t−k}. Note there will be such a cell since all the elements of \(\mbox {${Q_{k}}^{\sigma }$}\) are greater than or equal to

*i*

_{t−k}and there is at least one cell in row

*r*which is not occupied that lies above an occupied cell in row

*r*−1 in \(\mbox {${Q_{k}}^{\sigma }$}\) since there is such a cell in \(\mbox {${P_{k}}^{\sigma }$}\). The resulting filling is \({Q}^{\sigma}_{k+1}\). Repeat this procedure until all of the biletters from

*w*

_{A}have been inserted. The resulting pair \((\mbox {${P_{n}}^{\sigma }$}, \mbox {${Q_{n}}^{\sigma }$}):=(\mbox {${P}^{\sigma }$}, \mbox {${Q}^{\sigma }$})\) is denoted by

*Ψ*

_{σ}(

*A*). For example, if

*σ*=1 4 2 5 3 and

*A*is the matrix given above, then

*Ψ*

_{σ}(

*A*)=(

*P*

^{σ},

*Q*

^{σ}) is pictured in Fig. 21.

Next consider the special case where \(\sigma = \bar{\epsilon}_{n}\). Note that \(P^{\bar{\epsilon}_{n}} = j_{t} j_{t-1} \ldots j_{1} \rightarrow E^{\bar {\epsilon}_{n}}\) is constructed by a twisted version of the usual RSK row insertion algorithm. In that case, the recording \(\operatorname {PBF}\)\(Q^{\bar{\epsilon}_{n}}\) is constructed in the same way that the usual RSK recording tableau is constructed except that we are constructing tableaux such that columns are weakly decreasing reading from bottom to top and the rows are strictly decreasing reading from left to right. Thus \(\varPsi_{\bar {\epsilon}_{n}}\) is just a twisted version of the usual RSK correspondence between ℕ-valued *n*×*n*-matrices and pairs of column strict tableaux of the same shape. In particular, we know that if *A* is an ℕ-valued *n*×*n*-matrix and *A*^{T} is its transpose, then \(\varPsi_{\bar{\epsilon}_{n}}(A) = (P^{\bar{\epsilon}_{n}},Q^{\bar{\epsilon}_{n}})\) if and only if \(\varPsi_{\bar{\epsilon}_{n}}(A^{T}) = (Q^{\bar{\epsilon }_{n}},P^{\bar{\epsilon}_{n}})\).

### Theorem 17

*Let**σ*=*σ*_{1}⋯*σ*_{n}∈*S*_{n}. *The map**Ψ*_{σ}*is a bijection between* ℕ-*valued**n*×*n**matrices and pairs*\((\mbox {${P}^{\sigma }$}, \mbox {${Q}^{\sigma }$})\)*of*\(\operatorname {PBF}\)*s with basement**σ**such that if**α**is the shape of*\(\mbox {${P}^{\sigma }$}\)*and**β**is the shape of*\(\mbox {${Q}^{\sigma }$}\), *then**λ*(*α*)=*λ*(*β*) *and**α**and**β**are compatible with basement σ*.

### Proof

Suppose that *A* is an ℕ-valued *n*×*n* matrix and \(\varPsi_{\sigma }(A) = (\mbox {${P}^{\sigma }$},\mbox {${Q}^{\sigma }$})\). The filling \(\mbox {${P}^{\sigma }$}\) is a \(\operatorname {PBF}\) by Lemma 5. The shape *α* of \(\mbox {${P}^{\sigma }$}\) satisfies *α*_{i}≥*α*_{j} for all inversions *i*<*j* of *σ* by Lemma 1. It is also easy to see that our definition of *Ψ*_{σ} ensures that *λ*(*α*)=*λ*(*β*).

We must prove that the filling \(\mbox {${Q}^{\sigma }$}\) is a \(\operatorname {PBF}\). The columns of \(\mbox {${Q}^{\sigma }$}\) are weakly decreasing from bottom to top by construction. For any given *i*, the bottom elements of biletters whose top elements are *i* are inserted in weakly decreasing order. It then follows from Lemma 15 that *i* cannot occur twice in the same row in \(\mbox {${Q}^{\sigma }$}\).

*A*triple consisting of the cells

*a*=(

*x*

_{1},

*y*

_{1}),

*b*=(

*x*

_{2},

*y*

_{1}), and

*c*=(

*x*

_{1},

*y*

_{1}−1) where

*x*

_{1}<

*x*

_{2}as depicted below. This triple would be a type

*A*coinversion triple only if

*Q*

^{σ}(

*a*)≤

*Q*

^{σ}(

*b*)≤

*Q*

^{σ}(

*c*). Since we cannot have two equal elements in \(\mbox {${Q}^{\sigma }$}\) in the same row, it must be that

*Q*

^{σ}(

*a*)<

*Q*

^{σ}(

*b*)≤

*Q*

^{σ}(

*c*). There are now two cases. First if

*Q*

^{σ}(

*b*)<

*Q*

^{σ}(

*c*), then under the

*Ψ*

_{σ}map,

*Q*

^{σ}(

*c*) was placed first in \(\mbox {${Q}^{\sigma }$}\), then

*Q*

^{σ}(

*b*) was placed, and then

*Q*

^{σ}(

*a*) was placed. But this means at the time

*Q*

^{σ}(

*b*) was placed, it could have been placed on top of

*Q*

^{σ}(

*c*) which is a contradiction since the

*Ψ*

_{σ}map requires that

*Q*

^{σ}(

*b*) be placed in the left-most possible position subject to the requirement that the columns are weakly decreasing. The second case is when

*Q*

^{σ}(

*b*)=

*Q*

^{σ}(

*c*). In that case, Lemma 15 ensures that the cells created by the insertion of the bottoms of biletters whose tops equal

*Q*

^{σ}(

*b*) are created from bottom to top. This means that the biletter which created cell

*c*in \(\mbox {${Q}^{\sigma }$}\) must have been processed before the biletter which created cell

*b*. But this means that under the

*Ψ*

_{σ}map,

*Q*

^{σ}(

*c*) was placed first in \(\mbox {${Q}^{\sigma }$}\), then

*Q*

^{σ}(

*b*) was placed, and then

*Q*

^{σ}(

*a*) was placed, which we have already determined is impossible. Thus there are no type

*A*coinversion triples in \(\mbox {${Q}^{\sigma }$}\).

*a*=(

*x*

_{2},

*y*),

*b*=(

*x*

_{1},

*y*−1), and

*c*=(

*x*

_{2},

*y*−1), where

*x*

_{1}≤

*x*

_{2}, which form a type

*B*coinversion triple in \(\mbox {${Q}^{\sigma }$}\) as depicted below. We know that

*Q*

^{σ}(

*b*)≠

*Q*

^{σ}(

*c*) since we cannot have two equal elements in the same row in \(\mbox {${Q}^{\sigma }$}\). Thus we must have

*Q*

^{σ}(

*a*)≤

*Q*

^{σ}(

*b*)<

*Q*

^{σ}(

*c*). Now if

*Q*

^{σ}(

*a*)<

*Q*

^{σ}(

*b*), then under the

*Ψ*

_{σ}map,

*Q*

^{σ}(

*c*) was placed first in \(\mbox {${Q}^{\sigma }$}\), then

*Q*

^{σ}(

*b*) was placed, and then

*Q*

^{σ}(

*a*) was placed. However, if

*Q*

^{σ}(

*a*)=

*Q*

^{σ}(

*b*), then Lemma 15 ensures that the cells created by the insertion of the bottoms of biletters whose tops equal

*Q*

^{σ}(

*b*) are created from bottom to top. This means that the biletter which created cell

*b*in \(\mbox {${Q}^{\sigma }$}\) must have been processed before the biletter which created cell

*a*. Thus in either case, under the

*Ψ*

_{σ}map,

*Q*

^{σ}(

*c*) was placed first in \(\mbox {${Q}^{\sigma }$}\), then

*Q*

^{σ}(

*b*) was placed, and then

*Q*

^{σ}(

*a*) was placed. The only reason that

*Q*

^{σ}(

*a*) was not placed on top of

*Q*

^{σ}(

*b*) is that there must have already existed an element

*e*which was on top of

*Q*

^{σ}(

*b*) at the time

*Q*

^{σ}(

*a*) was placed. This means that

*Q*

^{σ}(

*a*)≤

*e*since the cells in \(\mbox {${Q}^{\sigma }$}\) are created by adding elements in weakly decreasing order. However, since we cannot have two equal elements in the same row, we must have that

*Q*

^{σ}(

*a*)<

*e*. Thus we know \(\mbox {${Q}^{\sigma }$}(x_{1},y) > \mbox {${Q}^{\sigma }$}(x_{2},y)\). But this means that if we added an element

*z*in cell (

*x*

_{2},

*y*+1) which sits on top of

*Q*

^{σ}(

*a*), then the only reason that

*z*was not placed on top of \(e = \mbox {${Q}^{\sigma }$}(x_{1},y)\) is that there must have already been an element in \(\mbox {${Q}^{\sigma }$}(x_{1},y+1)\) at the time we added

*z*. But then we can argue as above that it must be the case that \(\mbox {${Q}^{\sigma }$}(x_{1},y+1) > \mbox {${Q}^{\sigma }$}(x_{2},y+1)\). But then we can repeat the argument for row

*y*+2 so that if (

*x*

_{2},

*y*+2) is a cell in \(\mbox {${Q}^{\sigma }$}\), then (

*x*

_{1},

*y*+2) must have already been filled at the time we added an element to (

*x*

_{2},

*y*+2) and that \(\mbox {${Q}^{\sigma }$}(x_{1},y+2) > \mbox {${Q}^{\sigma }$}(x_{2},y+2)\). Continuing on in this way, we conclude that the height of column

*x*

_{1}in \(\mbox {${Q}^{\sigma }$}\) is greater than or equal to the height of column

*x*

_{2}in \(\mbox {${Q}^{\sigma }$}\). But that is a contradiction, since if {

*a*,

*b*,

*c*} is a type

*B*triple, the height of column

*x*

_{1}in \(\mbox {${Q}^{\sigma }$}\) must be less than the height of column

*x*

_{2}in \(\mbox {${Q}^{\sigma }$}\). Thus there can be no type

*B*coinversion triples in \(\mbox {${Q}^{\sigma }$}\).

Note that our argument above did not really use any properties of *Q*^{σ}(*c*), but only relied on the fact that *Q*^{σ}(*a*)≤*Q*^{σ}(*b*). That is, we proved that if *x*_{1}<*x*_{2} and \(\mbox {${Q}^{\sigma }$}(x_{1},y-1) \geq \mbox {${Q}^{\sigma }$}(x_{2},y)\), then the height of column *x*_{1} in \(\mbox {${Q}^{\sigma }$}\) must be greater that or equal to the height of column *x*_{2} in \(\mbox {${Q}^{\sigma }$}\). But this means that if the height of column *x*_{1} in \(\mbox {${Q}^{\sigma }$}\) is less than the height of column *x*_{2} in \(\mbox {${Q}^{\sigma }$}\), then \(\mbox {${Q}^{\sigma }$}(x_{1},y-1) < \mbox {${Q}^{\sigma }$}(x_{2},y)\), which is precisely the *B*-increasing condition. Thus \(\mbox {${Q}^{\sigma }$}\) is a \(\operatorname {PBF}\).

Next consider the shape *β* of \(\mbox {${Q}^{\sigma }$}\). We must prove that *β*_{i}≥*β*_{j} for all inversions *i*<*j* of *σ*. Consider the shape *β*^{(1)} of \(\mbox {${Q}^{\sigma }$}\) after we have placed *j*_{n} into *Q*^{σ}. Since *j*_{n} is placed on top of the leftmost entry *σ*_{k} such that *σ*_{k}≥*j*_{n}, the first *k*−1 entries of *σ* are less than *σ*_{k} and hence the claim is satisfied after the initial insertion.

Assume that the claim is satisfied after the insertion of each of the last *k*−1 letters of *w*_{A} and consider the placement of the entry *j*_{n−k} in *Q*^{σ}. Let *s* be the index of the column into which *j*_{n−k} is placed. Let *t* be an integer less than *s* such that *σ*_{t}>*σ*_{s}. Then column *t* is weakly taller than column *s* before the placement of *j*_{n−k} by assumption. If column *t* is strictly taller, then the placement of *j*_{n−k} on top of column *s* will not alter the relative orders of the columns. If the heights of columns *t* and *s* are equal, then the highest entry in column *t* was inserted before the highest entry in column *s*, for otherwise the columns would violate the condition immediately after the highest entry of column *s* was inserted. But then *j*_{n−k} would be inserted on top of column *t*, a contradiction. Therefore, the shape *β* of \(\mbox {${Q}^{\sigma }$}\) satisfies the condition that *β*_{i}≥*β*_{j} for all pairs (*i*,*j*) satisfying *i*<*j* and *σ*_{i}>*σ*_{j}.

*Ψ*

_{σ}maps any

*n*×

*n*matrix

*A*to a pair of \(\operatorname {PBF}\)s (

*P*

^{σ},

*Q*

^{σ}). Now suppose that

*u*

_{A}=

*i*

_{1}…

*i*

_{t}and

*v*

_{A}=

*j*

_{1}…

*j*

_{t}and

*σ*=

*σ*

_{1}⋯

*σ*

_{n}where

*σ*

_{i}<

*σ*

_{i+1}. We would like to determine the relationship between

*Q*

^{σ}and \(Q^{s_{i}\sigma }\). We established in Corollary 8 that as we consider the sequence of insertions the new cells that we created by the insertions at each stage were in the same row of

*E*

^{σ}as in \(E^{s_{i}\sigma }\). This implies that for all

*j*, the elements in row

*j*of

*Q*

^{σ}and \(Q^{s_{i}\sigma }\) are the same. But then it is easy to prove by induction on the number of inversions of

*σ*that for all

*j*, the elements in row

*j*of

*Q*

^{σ}and \(Q^{\bar{\epsilon}_{n}}\) are the same. That is, \(\rho(Q^{\sigma }) = Q^{\bar{\epsilon}_{n}}\). Since there is a unique \(\operatorname {PBF}\)

*Q*with basement

*σ*such that for all

*j*, the elements in row

*j*of

*Q*and \(Q^{\bar{\epsilon}_{n}}\) are the same, it follows that \(Q^{\sigma }= \rho^{-1}_{\sigma }(Q^{\bar{\epsilon}_{n}})\) for all

*σ*. Since

*P*

^{σ}=

*j*

_{t}

*j*

_{t−1}…

*j*

_{1}→

*E*

^{σ}for all

*σ*, we know by the results of Sect. 3 that \(P^{\sigma }= \rho^{-1}_{\sigma }(P^{\bar{\epsilon}_{n}})\) for all

*σ*. Thus it follows that for any ℕ-valued

*n*×

*n*matrix

*A*,

*Ψ*

_{σ}is also a bijection between ℕ-valued

*n*×

*n*matrices

*A*and pairs (

*P*,

*Q*) of \(\operatorname {PBF}\)s with basement

*σ*.

We note that another way to define the inverse of *Ψ*_{σ} is given by choosing the first occurrence (in reading order) of the smallest value in \(\mbox {${Q}^{\sigma }$}\), removing it from \(\mbox {${Q}^{\sigma }$}\), and labeling this entry *j*_{1}. Then choose the rightmost entry in this row of \(\mbox {${P}^{\sigma }$}\) which sits at the top of its column and apply the inverse of the insertion procedure to remove this cell from \(\mbox {${P}^{\sigma }$}\). The resulting entry is then *i*_{1}. Repeat this procedure to obtain the array *w*_{A}. □

Note that our proof of Theorem 17 allows us to prove the following corollary which says that for any *σ*∈*S*_{n}, the map *Ψ*_{σ} can be factored through our twisted version of the RSK correspondence.

### Corollary 18

*For any*ℕ-

*valued*

*n*×

*n*

*matrix*

*A*,

*where the map*

*is a twisted version of the usual RSK correspondence*.

Corollary 18 allows us to prove that our permuted basement version of the RSK correspondence *Ψ*_{σ} satisfies many of the properties that are satisfied by the RSK correspondence. For example, we have the following theorem.

### Theorem 19

*Suppose that*

*A*

*is an*ℕ-

*valued*

*n*×

*n*

*matrix and*

*A*

^{T}

*is its transpose*.

*Then for all*

*σ*∈

*S*

_{n},

### Proof

### 6.1 Standardization

Let *w*=*w*_{1}…*w*_{n}∈{1,…,*n*}^{∗} be a word and let *P*^{σ}(*w*)=*w*_{1}…*w*_{n}→*E*^{σ} where *σ*=*σ*_{1}…*σ*_{n}∈*S*_{n}. One can standardize *w* in the usual manner. That is, if *w* has *i*_{j}*j*s for *j*=1,…,*n*, then the *standardization of**w*, *st*(*w*), is the permutation that results by replacing the 1’s in *w* by 1,…,*i*_{1}, reading from right to left, then replacing the 2s in *w* by *i*_{1}+1,…,*i*_{1}+*i*_{2}, reading from right to left, etc. If *st*(*w*_{1}…*w*_{n})=*s*_{1}…*s*_{n}, then we define the standardization of *P*^{σ}(*w*) by letting *st*(*P*^{σ}(*w*))=*s*_{1}…*s*_{n}→*E*^{σ}.

*st*(

*P*

^{σ}(

*w*)). That is, we can compute \(st(P^{\bar{\epsilon}_{n}}) = st(w) \rightarrow E^{\bar{\epsilon}_{n}}\) directly or we can compute \(P^{\bar{\epsilon}_{n}} = w\rightarrow E^{\bar{\epsilon}_{n}}\) and then standardize the reverse row strict tableau \(P^{\bar{\epsilon}_{n}}\). Here, for any reverse row strict tableau

*T*,

*st*(

*F*) is the standard reverse row strict tableau obtained by replacing the 1s in

*T*1,…,

*i*

_{1}in order from top to bottom, then replacing the 2s in

*T*by

*i*

_{1}+1,…,

*i*

_{1}+

*i*

_{2}, reading from top to bottom, etc. This follows from the fact that the usual standardization operation for words and column strict tableaux commutes with RSK row insertion; see [11]. Thus our standardization operation for words and reverse row strict tableaux commutes with our twisted version of RSK row insertion. That is, suppose

*w*=

*w*

_{1}…

*w*

_{n}∈{1,…,

*n*}

^{∗}and

*st*(

*w*)=

*s*

_{1}…

*s*

_{n}. Then \(w \rightarrow E^{\bar{\epsilon}_{n}} = T\) if and only if \(s_{1} \ldots s_{n} \rightarrow E^{\bar{\epsilon}_{n}} = st(T)\). Because our insertion algorithm where the basement permutation is \({\bar{\epsilon}}_{n}\) can be factored through our twisted version of RSK row insertion, the same thing happens when the basement is

*σ*. That is,

We can summarize the above discussion in the following two propositions.

### Proposition 20

*Let**σ*=*σ*_{1}⋯*σ*_{n}∈*S*_{n}, *w*=*w*_{1}…*w*_{n}∈{1,…,*n*}^{∗}, *and**st*(*w*)=*s*_{1}…*s*_{n}. *If**P*^{σ}(*w*)=*w*_{1}…*w*_{n}→*E*^{σ}*is of shape**γ**where**γ**is a weak composition of n*, *then the*\(\operatorname {PBF}\)*st*(*P*^{σ}(*w*))=*s*_{1}…*s*_{n}→*E*^{σ}*is a*\(\operatorname {PBF}\)*whose shape is a rearrangement of**γ*.

### Proof

We have proved above that \(st(P^{\sigma }(w)) = \rho^{-1}_{\sigma }(st(P^{\bar{\epsilon}_{n}}(w)))\). Thus since the shape of \(st(P^{\bar{\epsilon}_{n}}(w))\) is *λ*(*γ*), we know that \(\rho^{-1}_{\sigma }(st(P^{\bar{\epsilon}_{n}}(w)))\) is a rearrangement of *γ*. □

### Proposition 21

*The standardization of words and*\(\operatorname {PBF}\)

*s commutes with our insertion algorithm relative to the basement*

*σ*=

*σ*

_{1}⋯

*σ*

_{n}∈

*S*

_{n}

*in the sense that for any*

*w*=

*w*

_{1}…

*w*

_{n}∈{1,…,

*n*}

^{∗},

*we have the following commutative diagram*.

*w*=4 3 1 3 2 3 4 1 is pictured in Fig. 22.

*σ*also commutes with standardization. That is, suppose that we are given an ℕ-valued matrix

*n*×

*n*matrix

*A*such that the sum of then entries of

*A*is less than or equal to

*n*. Then if \(w_{A} = \begin{array}{c} \scriptstyle u_{A} \\[-3pt] \scriptstyle v_{A} \end{array} \) and it will be the case that

## 7 Evacuation

The evacuation procedure on reverse semi-standard Young tableaux associates to each reverse SSYT *T* a new reverse SSYT *evac*(*T*) through a deletion process coupled with jeu de taquin. Specifically, let *T* be a reverse SSYT with *n* cells whose largest entry is *m* and let *a* be the entry in cell (1,1). Remove the entry *a* from *T* and apply jeu de taquin to create a new reverse SSYT, *T*′, with *n*−1 cells. The skew shape *sh*(*T*)/*sh*(*T*′) therefore consists of one cell which is then filled with the complement, *m*+1−*a*, of *a* relative to *m*. Repeat this procedure with *T*′ (but without changing the value of *m*) and continue until all of the cells from *T* have been evacuated and their complements relative to *m* have been placed into the appropriate locations in the diagram consisting of the union of all the one-celled skew shapes. This resulting diagram is a reverse semi-standard Young tableau called *evac*(*T*).

We define an evacuation procedure on standard \(\operatorname {PBF}\)s with basement *σ* as follows. Given a standard \(\operatorname {PBF}\)*F*^{σ} with basement *σ*, we define \(\mathit{evac}(F^{\sigma }) = \rho^{-1}_{\sigma }(\mathit{evac}(\rho_{\sigma }(F^{\sigma })))\). That is, we first use the *ρ*_{σ} map to send *F*^{σ} to a reverse standard tableau *ρ*_{σ}(*F*^{σ}). Then we apply the usual evacuation procedure to produce a reverse standard tableau *evac*(*ρ*_{σ}(*F*^{σ})) and next apply \(\rho^{-1}_{\sigma }\) to map *evac*(*ρ*(*F*^{σ})) back to a standard \(\operatorname {PBF}\) with basement *σ*. We claim that in the special case where *σ*=*ϵ*_{n} is the identity, then we can define the evacuation procedure directly on the standard \(\operatorname {PBF}\) which will allow us to compute evacuation without using jeu de taquin.

### Procedure 22

Let \(F^{\epsilon_{n}}\) be an arbitrary \(\operatorname {PBF}\) of size *n* whose largest entry is *m*, and let *R*_{i} be the collection of entries appearing in the *i*th row of \(F^{\epsilon_{n}}\), reading from bottom to top. Let *e*_{1} be the largest entry in the first row of \(F^{\epsilon_{n}}\), *C*_{1} be the column containing *e*_{1}, and let *h*_{1} be the height of *C*_{1} in \(F^{\epsilon _{n}}\). Assign *m*+1−*e*_{1} row \(R_{h_{1}}\) in \(\mathit{evac}(F^{\epsilon_{n}})\). Remove *e*_{1} and shift the remaining entries in column *C*_{1} down by one position so that there are no gaps in the column. Next rearrange the entries in the rows in the resulting figure according to the same procedure that we used in defining the \(\rho^{-1}_{\epsilon _{n}}\) map to produce a \(\operatorname {PBF}\)\(F_{1}^{\epsilon_{n}}\). Repeat the procedure on the new diagram \(F_{1}^{\epsilon_{n}}\). That is, let *e*_{2} be the largest entry in the first row of \(F_{1}^{\epsilon_{n}}\), *C*_{2} be the column that contains *e*_{2}, and *h*_{2} be the height of column *C*_{2} in \(F_{1}^{\epsilon_{n}}\). Assign *m*+1−*e*_{2} row \(R_{h_{2}}\) in \(\mathit{evac}(F^{\epsilon_{n}})\). Remove *e*_{2} and shift the remaining entries in column *C*_{2} down by one position so that there are no gaps in the column. Next rearrange the entries in the rows of the resulting figure according to same procedure that we used in defining the \(\rho^{-1}_{\epsilon_{n}}\) map to produce a \(\operatorname {PBF}\)\(F_{2}^{\epsilon_{n-2}}\). Continue in this manner until all of the entries have been removed. The \(\operatorname {PBF}\)\(\mathit{evac}(F^{\epsilon_{n}})\) is produced by letting row *i* contain the complements of each entry relative to *m* associated with a column of height *i* and applying the map \(\rho^{-1}_{\epsilon_{n}}\) to send the resulting entries in the given rows to their appropriate places.

### Theorem 23

*If*\(F^{\epsilon_{n}}\)*is a*\(\operatorname {PBF}\), *then one can construct*\(\mathit{evac}(F^{\epsilon_{n}})= \rho^{-1}_{\epsilon_{n}}(\mathit{evac}(\rho(F^{\epsilon_{n}})))\)*by Procedure* 22.

### Proof

Let *F* be a \(\operatorname {PBF}\) with basement \(\bar{\epsilon}_{n}\) and let \(G^{\epsilon_{n}} = \rho^{-1}_{\epsilon_{n}}(F)\). Let *e*_{1}=*F*(1,1) so that *e*_{1} is the largest entry in the first row of *F* and hence it will be the largest entry in the first row of \(G^{\epsilon_{n}}\). Now consider the jeu de taquin path of the empty space created by the removal of *e*_{1} from *F*. That is, in jeu de taquin, we move the empty space to cell (2,1) and put *F*(2,1) in cell (1,1) if *F*(2,1) is defined and either *F*(2,1)≥*F*(1,2) or *F*(1,2) is not defined. Otherwise, we put *F*(1,2) in cell (1,1) and move the empty space to cell (1,2). In general, if the empty space is in cell (*i*,*j*), then we move the empty space to cell (*i*+1,*j*) and put *F*(*i*+1,*j*) into cell (*i*,*j*) if *F*(*i*+1,*j*) is defined and either *F*(*i*+1,*j*)≥*F*(*i*,*j*+1) or *F*(*i*,*j*+1) is not defined. Otherwise, we put *F*(*i*,*j*+1) in cell (*i*,*j*) and move the empty space to cell (*i*,*j*+1). The jeu de taquin path ends at cell (*i*,*j*) when both *F*(*i*,*j*+1) and *F*(*i*+1,*j*) are undefined.

Now suppose in the evacuation of *e*_{1}=*F*(1,1), the path of the empty space ends in row *s* and that *c*_{i} is the right-most column involved in the jeu de taquin path in row *i* for *i*=1,…,*s*. Thus the jeu de taquin path involves cells (1,1),…,(*c*_{1},1) in row 1 of *F*, cells (*c*_{1},2),…,(*c*_{2},2) in row 2 of *F*, cells (*c*_{2},3),…,(*c*_{3},3) in row 3 of *F*, etc.. Now if *F*_{1} is the \(\operatorname {PBF}\) with basement \(\bar{\epsilon}_{n}\) that results from evacuating *e*_{1}, then it follows that in *F*_{1}, each of the entries *F*(*c*_{i},*i*+1) will end up in row *i* of *F*_{1} and all the other entries will be in the same row in *F*_{1} as they were in *F*. We claim that in \(G^{\epsilon_{n}} = \rho^{-1}_{\epsilon_{n}}(F)\), the column containing *e*_{1} consists of *e*_{1},*F*(*c*_{1},2),*F*(*c*_{2},3),…,*F*(*c*_{s−1},*s*), reading from bottom to top. Once we prove the claim, it will follow that in our direct evacuation of *e*_{1} in \(G^{\epsilon_{n}}\) to produce \(G_{1}^{\epsilon_{n}}\), the entries in row *i* in *F*_{1} and \(G_{1}^{\epsilon_{n}}\) are the same. But then \(\rho(G_{1}^{\epsilon_{n}}) =F_{1}\) so that \(\rho^{-1}_{\epsilon_{n}}(F_{1}) = G^{\epsilon_{n}}_{1}\) since the row sets of *F*_{1} completely determine \(\rho^{-1}_{\epsilon_{n}}(F_{1})\). The theorem then easily follows by induction.

To prove the claim, note that the entries in the first row of *F* must all be distinct so that in constructing \(\rho^{-1}_{\epsilon_{n}}(F)\), each entry *i* in row 1 of *F* will be placed on column *i*. Now the fact that (2,1),…,(*c*_{1},1) are in the jeu de taquin path means that *F*(2,1)≥*F*(1,2),*F*(3,1)≥*F*(2,2),…,*F*(*c*_{1},1)≥*F*(*c*_{1}−1,2). The fact that *F*(*c*_{1},2) is in the jeu de taquin path means that *F*(*c*_{1},2)>*F*(*c*_{1}+1,1) or *F*(*c*_{1}+1,1) is not defined. It then follows that in constructing \(\rho^{-1}_{\epsilon_{n}}(F)\), the entries *F*(1,2),…*F*(*c*_{1}−1,2) can be placed on the columns occupied by *F*(2,1),…,*F*(*c*_{1},1) but not on top of any of the columns occupied by *F*(*c*_{1}+1,1),*F*(*c*_{1}+2,1),…. Thus the *F*(1,2),…*F*(*c*_{1}−1,2) will be placed somewhere in the columns occupied by *F*(2,1),…,*F*(*c*_{1},1). Thus when we go to place *F*(*c*_{1},2) in the left-most available column, it must go on top of *e*_{1} since it cannot go on top of any of the columns occupied by *F*(*c*_{1}+1,1),*F*(*c*_{1}+2,1),…. Finally any entries strictly right of (*c*_{1},2) in row 2 must be placed on top of columns occupied by entries strictly to the left of the column containing *e*_{1} in row 1 of *F*. Now consider the construction of the third row of \(\rho^{-1}_{\epsilon_{n}}(F)\). The entries *F*(1,3),…,*F*(*c*_{1}−1,3) can go on top of the entries *F*(1,2),…,*F*(*c*_{1}−1,2) since *F*(*i*,3)≤*F*(*i*,2) for all *i* for which both *F*(*i*,3) and *F*(*i*,2) are defined. Next the fact that (*c*_{1}+1,2),…,(*c*_{3},2) are in the jeu de taquin path means that *F*(*c*_{1}+1,2)≥*F*(*c*_{1},3),…,*F*(*c*_{2}−1,3)≥*F*(*c*_{2},2). Thus *F*(*c*_{1},3),*F*(*c*_{1}+1,3),…,*F*(*c*_{2},3) can go on top of the entries *F*(*c*_{1}+1,2),…,*F*(*c*_{2},2) in row two of \(\rho^{-1}_{\epsilon_{n}}(F)\). The fact that (*c*_{2},3) is in the jeu de taquin path of *e*_{1} in *F* means that *F*(*c*_{2},3)>*F*(*c*_{2}+1,2) so that none of *F*(*c*_{1}+1,3),…,*F*(*c*_{2},3) can go on top of entries *F*(*c*_{2}+1,2),*F*(*c*_{2}+2,2),… in row two of \(\rho^{-1}_{\epsilon_{n}}(F)\). Hence *F*(1,3),…,*F*(*c*_{2}−1,3) will be able go on top of entries *F*(1,2),…,*F*(*c*_{1}−1,2),*F*(*c*_{1}+1,2),…,*F*(*c*_{2},2) in row 2 of \(\rho^{-1}_{\epsilon_{n}}(F)\) but they cannot go on top of the entries *F*(*c*_{2}+1,2),*F*(*c*_{2}+2,2),… in row 2 of \(\rho^{-1}_{\epsilon_{n}}(F)\). Hence it must be the case that *F*(1,3),…,*F*(*c*_{2}−1,3) end up on top of entries *F*(1,2),…,*F*(*c*_{1}−1,2),*F*(*c*_{1}+1,2),…,*F*(*c*_{2},2) in row 2 of \(\rho^{-1}_{\epsilon_{n}}(F)\). Since *F*(*c*_{2},3) cannot go on top of the entries *F*(*c*_{2}+1,2),*F*(*c*_{2}+2,2),… in row 2 of \(\rho^{-1}_{\epsilon_{n}}(F)\), the only place left to place *F*(*c*_{2},3) is on top of the column that contains *e*_{1}. Continuing on in this way establishes the claim.^{1} □

## Footnotes

- 1.
We wish to thank an anonymous referee who made numerous helpful suggestions for improving the presentation of this paper.

### References

- 1.Haglund, J., Haiman, M., Loehr, N.: A combinatorial formula for non-symmetric Macdonald polynomials. Am. J. Math.
**103**, 359–383 (2008) MathSciNetCrossRefGoogle Scholar - 2.Haglund, J., LoBue, J., Remmel, J.: Generalized Demazure atoms and the weak Bruhat order. In preparation Google Scholar
- 3.Haglund, J., Luoto, K., Mason, S., van Willigenburg, S.: Quasisymmetric Schur functions. J. Comb. Theory, Ser. A
**118**, 463–490 (2011) MATHCrossRefGoogle Scholar - 4.Haglund, J., Luoto, K., Mason, S., van Willigenburg, S.: Refinements of the Littlewood–Richardson rule. Trans. Am. Math. Soc.
**363**, 1665–1686 (2011) MATHCrossRefGoogle Scholar - 5.Lascoux, A., Schützenberger, M.-P.: Keys and standard bases. In: Tableaux and Invariant Theory. IMA Volumes in Mathematics and Its Applications, vol. 19, pp. 125–144 (1990) Google Scholar
- 6.Macdonald, I.G.: Affine Hecke algebras and orthogonal polynomials. In: Séminaire Bourbaki, vol. 1994/95. Astérisque, vol. 237, pp. 189–207 (1996) Google Scholar
- 7.Macdonald, I.G.: Symmetric Functions and Hall Polynomials. With Contributions by A. Zelevinsky, 2nd edn. Oxford Mathematical Monographs. Oxford Science Publications. Clarendon/Oxford University Press, New York (1995) Google Scholar
- 8.Marshall, D.: Symmetric and nonsymmetric Macdonald polynomials. Ann. Comb.
**3**, 385–415 (1999) MathSciNetMATHCrossRefGoogle Scholar - 9.Mason, S.: A decomposition of Schur functions and an analogue of the Robinson–Schensted–Knuth algorithm. Sémin. Lothar. Comb.
**57**, B57e (2006) Google Scholar - 10.Mason, S.: An explicit construction of type
*A*Demazure atoms. J. Algebr. Comb.**29**(3), 295–313 (2009) MATHCrossRefGoogle Scholar - 11.Stanley, R.P.: Enumerative Combinatorics, vol. 2. Cambridge Studies in Advanced Mathematics, vol. 62. Cambridge University Press, Cambridge (1999). With forward by Gian-Carlo Rota and Appendix 1 by Sergey Fomin CrossRefGoogle Scholar