Combinatorial Generation via Permutation Languages. III. Rectangulations

Abstract

A generic rectangulation is a partition of a rectangle into finitely many interior-disjoint rectangles, such that no four rectangles meet in a point. In this work we present a versatile algorithmic framework for exhaustively generating a large variety of different classes of generic rectangulations. Our algorithms work under very mild assumptions, and apply to a large number of rectangulation classes known from the literature, such as generic rectangulations, diagonal rectangulations, 1-sided/area-universal, block-aligned rectangulations, and their guillotine variants, including aspect-ratio-universal rectangulations. They also apply to classes of rectangulations that are characterized by avoiding certain patterns, and in this work we initiate a systematic investigation of pattern avoidance in rectangulations. Our generation algorithms are efficient, in some cases even loopless or constant amortized time, i.e., each new rectangulation is generated in constant time in the worst case or on average, respectively. Moreover, the Gray codes we obtain are cyclic, and sometimes provably optimal, in the sense that they correspond to a Hamilton cycle on the skeleton of an underlying polytope. These results are obtained by encoding rectangulations as permutations, and by applying our recently developed permutation language framework.

1 Introduction

Partitioning a geometric shape into smaller shapes is a fundamental theme in discrete and combinatorial geometry. In this paper we consider rectangulations, i.e., partitions of a rectangle into finitely many interior-disjoint rectangles. Such partitions have an abundance of practical applications, which motivates their combinatorial and algorithmic study. For example, rectangulations are an appealing way to represent geographic information as a cartogram. This is a map where each country is represented as a rectangle, the adjacencies between rectangles correspond to those between countries, and the areas of the rectangles are determined by some geographic variable, such as population size [26]. If the rectangulation is area-universal [12] or aspect-ratio-universal [16], respectively, then such an adjacency-preserving cartogram can be drawn for any assignment of area values or aspect ratios to the rectangles. Another important use of rectangulations is as floorplans in VLSI design and architectural design. These problems often involve additional constraints on top of adjacency, such as extra space for wires [33] or proportion limits for the rooms [31]. An important notion in this context are slicing floorplans [33], also known as guillotine floorplans, i.e., floorplans that can be subdivided into their constituent rectangles by a sequence of straight vertical or horizontal cuts.

Rectangulations have rich combinatorial properties, and a task that has received a lot of attention is counting, i.e., determining the number of rectangulations of a particular type with n rectangles, either exactly as a function of n [43] or asymptotically as n grows [39]. This led to several beautiful bijections of rectangulations with pattern-avoiding permutations [1, 4, 36] or with twin binary trees [43]. The focus of this paper is on another fundamental algorithmic task, which is more fine-grained than counting, namely exhaustive generation, meaning that every rectangulation from a given class must be produced exactly once. While such generation algorithms are known for many other discrete objects such as permutations, combinations, subsets, trees etc. and covered in standard textbooks such as Knuth’s [25], much less is known about the generation of geometric objects such as rectangulations.

The ultimate goal for a generation algorithm is to produce each new object in time ${\mathcal O}(1)$, which requires that consecutively generated objects differ only by a ‘small local change’. Such a minimum change listing of combinatorial objects is often called a Gray code [38]. If the time bound ${\mathcal O}(1)$ for producing the next object holds in every step, then the algorithm is called loopless [11], and if it holds on average it is called constant amortized time (CAT) [37]. The Gray code problem entails the definition of a flip graph, which has as nodes all the combinatorial objects to be generated, and an edge between any two objects that differ in the specified small way. Clearly, computing a Gray code ordering of the objects is equivalent to traversing a Hamilton path or cycle in the corresponding flip graph. It turns out that some interesting flip graphs arising from rectangulations can be equipped with a natural lattice structure [29], analogous to the Tamari lattice on triangulations, and realized as polytopes in high-dimensional space [27], analogous to the associahedron (see [34] for generalizations). This ties in the Gray code problem with deep methods and results from lattice and polytope theory.

1.1 Our Results

The main contribution of this paper is a versatile algorithmic framework for generating a large variety of different classes of generic rectangulations, i.e., rectangulations with the property that no four rectangles meet in a point. In particular, we obtain efficient generation algorithms for several interesting classes known from the literature, in some cases loopless or CAT algorithms; see Table 1.

The initialization time and memory requirement for all these algorithms is linear in the number of rectangles. The classes of rectangulations shown in the table arise from generic rectangulations by imposing structural constraints, such as the guillotine property or forbidden configurations, or by equivalence relations, and they will be defined in Sect. 2.2. We implemented the algorithms generating the classes of rectangulations from the table in C++, and we made the code available for download and experimentation on the Combinatorial Object Server [45].

Table 1 Classes of rectangulations that can be generated by our algorithms

The classes of rectangulations that our algorithms can generate are not limited to the examples shown in Table 1, but can be described by the following closure property; see Fig. 1. Given an infinite class of rectangulations ${\mathcal C}$, we require that if a rectangulation R is contained in ${\mathcal C}$, then the rectangulation obtained from R by deleting the bottom-right rectangle is also in ${\mathcal C}$, and the two rectangulations obtained from R by inserting a new rectangle at the bottom or right, respectively, are also in ${\mathcal C}$ (formal definitions of deletion and insertion are given in Sect. 2). If ${\mathcal C}$ satisfies this property, then our algorithms allow generating the set ${\mathcal C}_n\subseteq {\mathcal C}$ of all rectangulations from ${\mathcal C}$ with exactly n rectangles, for every $n\ge 1$, by so-called jumps, a minimum change operation that generalizes simple flips, T-flips, and wall slides studied in [9, 36] (the formal definition of jumps is in Sect. 3.1). Moreover, if the class ${\mathcal C}$ is symmetric, i.e., if R is in ${\mathcal C}$ then the rectangulation obtained from R by reflection at the diagonal from top-left to bottom-right is also in ${\mathcal C}$, then the jump Gray code for ${\mathcal C}_n$ is cyclic, i.e., the last rectangulation differs from the first one only by a jump. In other words, we not only obtain a Hamilton path in the corresponding flip graph, but a Hamilton cycle. In fact, all the classes of rectangulations listed in Table 1 satisfy the aforementioned closure and symmetry properties, so in all those cases we obtain cyclic jump Gray codes.

Generic rectangulations and diagonal rectangulations, shown in the first two rows of Table 1, have an underlying lattice and polytope structure [27, 29], and in those two cases our Gray codes form a Hamilton cycle on the skeleton of this polytope, i.e., jumps are provably optimal minimum change operations. The Gray codes for these two rectangulation classes with $n=1,\ldots ,5$ rectangles are shown in Appendix D.

Fig. 1

Closure property and symmetry property

It turns out that many interesting classes of rectangulations can be characterized by pattern avoidance; see the second column in Table 1. Under very mild conditions on the patterns, these classes satisfy the aforementioned closure property, and can hence be generated by our framework. In this work we initiate a systematic investigation of pattern avoidance in rectangulations, and we obtain the first counting results for many known and new classes; see the third column in Table 1 and the more extensive table in Sect. 9.

Our generation framework for rectangulations consists of two main algorithms. The first is a simple greedy algorithm that generates a jump Gray code ordering for any set of rectangulations ${\mathcal C}_n\subseteq {\mathcal C}$ for which ${\mathcal C}$ satisfies the aforementioned closure property; see Algorithm J$^{\square }$ and Theorem 3.3 in Sect. 3. The second is a memoryless version of the first algorithm, which computes the same ordering of rectangulations; see Algorithm M$^\square$ and Theorem 5.1 in Sect. 5. This algorithm can be fine-tuned to derive efficient algorithms for several known rectangulation classes such as the ones listed in Table 1, by providing corresponding jump oracles for the class ${\mathcal C}$.

To prove Theorems 3.3 and 5.1, we encode rectangulations by permutations as described by Reading [36], and we then apply our framework for exhaustively generating permutation languages presented in [20,21,22]. The minimum change operations on permutations used in that framework translate to jumps on rectangulations. Generating different classes of rectangulations efficiently is thus another major new application of our permutation language framework, and in this paper we flesh out the details of this application.

1.2 Related Work

There has been some prior work on generating a few special classes of rectangulations, all based on Avis and Fukuda’s reverse search method [6]. Specifically, Nakano [32] described a CAT generation algorithm for generic rectangulations, which does not produce a Gray code, however. This algorithm has been adapted by Takagi and Nakano [40] to generate generic rectangulations with bounds on the number of rectangles that do not touch the outer face. Yoshii et al. [44] gave a Gray code for generic rectangulations based on a generating tree that is different from ours, resulting in a loopless algorithm. Their Gray code changes at most three edges of the rectangulation in each step, whereas our algorithm changes only 1 edge in each step for generic and for diagonal rectangulations. Consequently, none of the listings produced by these earlier algorithms corresponds to a walk along the skeleton of the underlying polytope.

There has been a lot of work on combinatorial properties of rectangulations. Yao et al. [43] showed that diagonal rectangulations are counted by the Baxter numbers and that guillotine diagonal rectangulations are counted by the Schröder numbers, using a bijection between diagonal rectangulations and twin binary trees. Ackerman et al. [1] presented another bijection between diagonal rectangulations and Baxter permutations, which also yields a bijection between guillotine diagonal rectangulations and separable permutations. Leifheit [28] showed that this bijection can be restricted to the 1-sided variants of these two rectangulation classes by adding two permutation patterns; see Table 1. Shen and Chu [39] provided asymptotic estimates for diagonal rectangulations and their guillotine variant. Moreover, He [18] presented an optimal encoding of diagonal rectangulations with n rectangles using only $3n-3$ bits, which is optimal.

The term ‘generic rectangulation’ was coined by Reading [36], who established a bijection between generic rectangulations and 2-clumped permutations, proving that these permutations are representatives of equivalence classes of a lattice congruence of the weak order on the symmetric group. Earlier, generic rectangulations had been studied under the name ‘rectangular drawings’ by Amano et al. [3] and by Inoue et al. [15, 23], who established recursion formulas and asymptotic bounds for their number. More general classes of rectangular partitions were analyzed by Conant and Michaels [10].

Ackerman et al. [2] considered the setting where we are given a set of n points in general position in a rectangle, and the goal is to partition the rectangle into smaller rectangles by n walls, such that each point from the set lies on a distinct wall. They showed that for every set of points that forms a separable permutation in the plane, the number of possible rectangulations is the $(n+1)$st Baxter number, and for every point set the number of possible guillotine rectangulations is the nth Schröder number. They also presented a counting and generation procedure based on simple flips and T-flips using reverse search, which was later improved by Yamanaka et al. [42].

1.3 Outline of This Paper

In Sect. 2 we provide basic definitions and concepts that will be used throughout the paper. In Sect. 3 we present a greedy algorithm for generating a set of rectangulations by jumps, and we provide a sufficient condition for the algorithm to succeed. In Sect. 4 we show that the algorithm applies to a large number of rectangulation classes that are characterized by pattern avoidance. In Sect. 5 we demonstrate how to make our generation algorithm memoryless and efficient. The data structures and basic functions used by our algorithms are provided in Sects. 6 and 7. The proofs of Theorems 3.3 and 5.1 are presented in Sect. 8, by establishing a connection between rectangulations and permutations and by applying our permutation language framework. In Sect. 9 we report on our computer experiments about counting pattern-avoiding rectangulations. We conclude the paper with some interesting open questions in Sect. 10. The proofs of two technical lemmas and for the results on rectangulations under S-equivalence mentioned in Table 1 are deferred to Appendices A and B, as well as several visualizations of Gray codes produced by our algorithms.

2 Preliminaries

2.1 Generic Rectangulations

A generic rectangulation, or rectangulation for short, is a partition of a rectangle into finitely many interior-disjoint axis-aligned rectangles, such that no four rectangles of the partition have a point in common; see Fig. 2. In other words, every point where three rectangles meet, or where two rectangles meet the outer face forms a T-joint with the incident rectangle boundaries. Given rectangles r and s, we say that r is left of s, and s is right of r, if the right side of r intersects the left side of s (necessarily in a line segment, rather than a single point). Similarly, we say that r is below s, and s is above r, if the top side of r intersects the bottom side of s. We consider generic rectangulations up to equivalence that preserves the left/right and below/above relations between rectangles, and we use ${\mathcal R}_n$, $n\ge 1$, to denote the set of all rectangulations with n rectangles. We write $\square$ for the unique rectangulation in ${\mathcal R}_1$, i.e., the rectangulation consisting of a single rectangle.

Fig. 2

Generic rectangulation R with 11 rectangles. The rectangle $r_1$ is below $r_2$, $r_3$, and $r_4$, above $r_5$ and $r_6$, right of $r_7$, and left of $r_8$

We refer to every rectangle corner in a rectangulation as a vertex, to every minimal line segment between two vertices as an edge, and to every maximal line segment between two vertices that are not corners of the rectangulation as a wall. The type of a vertex that is not a corner of the rectangulation describes the shape of the T-joint at this vertex, and it is one of ${\top }$, ${\vdash }$, ${\bot }$, or ${\dashv }$.

2.2 Flip Operations and Classes of Rectangulations

Our Gray codes use three types of local change operations on rectangulations; see Fig. 3.

Fig. 3

Local change operations on rectangulations

A wall slide swaps the order of two neighboring vertices of types ${\vdash }$ and ${\dashv }$ along a vertical wall, or of types ${\top }$ and ${\bot }$ along a horizontal wall. A simple flip swaps the orientation of a wall that separates two rectangles. Given a vertex v that belongs to three rectangles, we consider the wall w that goes through v and the wall t that ends at v, and we let $w'$ and $w''$ be the two halves of w meeting in v. If $w'$ or $w''$ is an edge, respectively, then a T-flip swaps the orientation of this edge so that it merges with t.

We now define various interesting subclasses of generic rectangulations that have been studied in the literature and that appear in Table 1. Examples illustrating these classes are in Fig. 4. A diagonal rectangulation is one in which every rectangle intersects the main diagonal that goes from the top-left to the bottom-right corner of the rectangulation. We write ${\mathcal D}_n\subseteq {\mathcal R}_n$ for the set of all diagonal rectangulations with n rectangles. Diagonal rectangulations are characterized by avoiding the wall patterns  and  [9]. Consider the equivalence relation on ${\mathcal R}_n$ obtained from wall slides, sometimes referred to as R-equivalence [4]. The equivalence classes are referred to as mosaic floorplans, and every equivalence class contains exactly one diagonal rectangulation, obtained by repeatedly destroying occurrences of  or  by wall slides [9]. Consequently, in a diagonal rectangulation, along every vertical wall, all ${\vdash }$-vertices are below all ${\dashv }$-vertices, and along every horizontal wall, all ${\bot }$-vertices are to the left of all ${\top }$-vertices.

In a 1-sided rectangulation, every wall is the side of at least one rectangle, i.e., these rectangulations are characterized by avoiding the four patterns , and . The notion of 1-sidedness was introduced by Eppstein et al. [12] to characterize area-universal rectangulations, i.e., for any assignment of areas to the rectangles, the rectangulation can be drawn so that each rectangle has the prescribed area.

Asinowski et al. [4] also considered the equivalence relation on ${\mathcal R}_n$ obtained from wall slides and simple flips, and they called it S-equivalence. By definition, S-equivalence is a coarser relation than R-equivalence, i.e., the equivalence classes are obtained by identifying mosaic floorplans that differ in simple flips. In Appendix B we introduce block-aligned rectangulations, which are a subset of diagonal rectangulations with the property that every equivalence class of S-equivalence contains exactly one block-aligned rectangulation.

A rectangulation is guillotine, if each of its rectangles can be cut out from the entire rectangulation by a sequence of straight vertical or horizontal cuts. Guillotine rectangulations are characterized by avoiding the windmill patterns  and , which is a folklore result. Various special classes of guillotine diagonal rectangulations, characterized by the avoidance of certain wall configurations, were introduced by Asinowski and Mansour [5] (see Sect. 4 for precise definitions of these configurations). Mosaic floorplans that are guillotine are also known as slicing floorplans.

Felsner, Nathenson, and Tóth [16] showed that 1-sided guillotine rectangulations are precisely the aspect-ratio-universal rectangulations, i.e., for any assignment of aspect ratios to the rectangles, the rectangulation can be drawn so that each rectangle has the prescribed aspect ratio.

Fig. 4

Examples of different classes of rectangulations: a diagonal, but neither 1-sided nor guillotine, b 1-sided, but not guillotine, c guillotine, but not diagonal, d guillotine and 1-sided. Occurrences of the corresponding forbidden patterns are highlighted

2.3 Deletion of Rectangles

We now describe two operations on a generic rectangulation R, namely deleting a rectangle and inserting a rectangle. The resulting rectangulations will be denoted by p(R) and $c_i(R)$, notations that refer to the parent and children of R, in a tree structure that will be discussed shortly. The deletion and insertion operations were introduced in [19] and heavily used e.g. in [1] and [32].

The idea of deletion is to contract the rectangle in the bottom-right corner of the rectangulation. Formally, given a rectangulation $R\in {\mathcal R}_n$, $n\ge 2$, we consider the rectangle r in the bottom-right corner, and we consider the top-left vertex of r. If this vertex has type ${\vdash }$, then we collapse r by sliding its top side, which forms a wall, downwards until it merges with the bottom side of r; see Fig. 5a. Similarly, if this vertex has type ${\top }$, then we collapse r by sliding its left side, which forms a wall, to the right until it merges with the right side of r; see Fig. 5b. We denote the resulting rectangulation with $n-1$ rectangles by $p(R)\in {\mathcal R}_{n-1}$, and we say that p(R) is obtained from R by deletion.

Fig. 5

Deletion operation

Fig. 6

A rectangulation and the indexing of its rectangles given by repeated deletion

Moreover, we denote the n rectangles of R by $r_n,r_{n-1},\ldots ,r_1$ in the order in which they are deleted when applying the deletion operation exhaustively; see Fig. 6. Clearly, if $r_i$ is deleted and its top-left vertex has type ${\vdash }$, then the rightmost rectangle above $r_i$ is $r_{i-1}$. Similarly, if the top-left vertex has type ${\top }$, then the lowest rectangle to the left of $r_i$ is $r_{i-1}$.

For any $R\in {\mathcal R}_n$ and $i=1,\ldots ,n$ we define $R^{[i]}:=p^{n-i}(R)$, i.e., this is the sub-rectangulation of R formed by the first i rectangles; see Fig. 6.

2.4 Insertion of Rectangles

The idea of insertion is to add a new rectangle into the bottom-right corner of the rectangulation. Given a rectangulation $R\in {\mathcal R}_{n-1}$, we first define a set of points in R that can become the top-left corner of the newly added rectangle; see Fig. 7.

For any rectangle r in $R\in {\mathcal R}_{n-1}$, $n\ge 2$, that touches the bottom boundary of R, we consider all edges forming the left side of r, and from every such edge we select one interior point, and we refer to it as a vertical insertion point.

Fig. 7

Linear ordering of insertion points. First and last insertion point are filled

Similarly, for any rectangle r in R that touches the right boundary of R, we consider the set of all edges forming the top side of r, and from every such edge we select one interior point, and we refer to it as a horizontal insertion point. Combinatorially it does not make a difference which interior point of each edge is selected.

We order the insertion points linearly, by sorting all vertical insertion points lexicographically by their (x, y)-coordinates, followed by all horizontal insertion points sorted lexicographically by their (y, x)-coordinates; see Fig. 7. We write $I(R)=(q_1,q_2,\ldots ,q_\nu )$ for the sequence of all insertion points ordered in this linear order. In particular, $\nu =\nu (R)$ denotes the number of insertion points.

Lemma 2.1

For any rectangulation $R\in {\mathcal R}_{n-1}$ we have $\nu (R)\le n$.

Proof

Each rectangle in R has at most one vertical insertion point on its right side, and at most one horizontal insertion point on its bottom side. Moreover, no rectangle has both, the bottom-right rectangle $r_{n-1}$ has neither of the two, and exactly two insertion points lie on the boundary of R. Combining these observations shows that $\nu (R)\le ((n-1)-1)+2=n$. $\square$

Clearly, the upper bound in Lemma 2.1 is attained if every rectangle touches the bottom or right boundary of R.

Given $R\in {\mathcal R}_n$ and the sequence of insertion points $I(R)=(q_1,\ldots ,q_\nu )$, for each $i=1,\ldots ,\nu$ we define a rectangulation $c_i(R)\in {\mathcal R}_n$ as follows: If $q_i$ is a vertical insertion point, then $c_i(R)$ is obtained from R by inserting a new rectangle $r_n$ in the bottom-right corner such that $r_n$ has above it exactly all rectangles which in R lie to the right of $q_i$ and touch the bottom boundary of R, and such that $r_n$ has to its left exactly all rectangles which in R touch the vertical wall through $q_i$ below $q_i$; see Fig. 8a. Similarly, if $q_i$ is a horizontal insertion point, then $c_i(R)$ is obtained from R by inserting a new rectangle $r_n$ in the bottom-right corner such that $r_n$ has to its left exactly all rectangles which in R lie below $q_i$ and touch the right boundary of R, and such that $r_n$ has above it exactly all rectangles which in R touch the horizontal wall through $q_i$ to the right of $q_i$; see Fig. 8b. We say that $c_i(R)$ is obtained from R by insertion.

Fig. 8

Insertion operation

By these definitions, the operations of deletion and insertion are inverse to each other, which we record in the following lemma.

Lemma 2.2

For any rectangulation $R\in {\mathcal R}_{n-1}$ and any two distinct insertion points $q_i$ and $q_j$ from I(R), the rectangulations $c_i(R)\in {\mathcal R}_n$ and $c_j(R)\in {\mathcal R}_n$ are distinct, and we have $R=p(c_i(R))=p(c_j(R))$. Moreover, for any $R'\in {\mathcal R}_n$ with $p(R')=R$ there is an insertion point $q_i$ in I(R) such that $c_i(R)=R'$.

The first and last insertion point play a special role in our arguments, which is why they are highlighted in Fig. 8. We say that R is bottom-based if R has a rectangle whose bottom side is the entire bottom boundary of R, and R is right-based if R has a rectangle whose right side is the entire right boundary of R. Note that the rectangulation $\square \in {\mathcal R}_1$ is both bottom-based and right-based, and if $n\ge 2$, then $R\in {\mathcal R}_n$ is bottom-based if and only if $R=c_1(p(R))$ and right-based if and only if $R=c_{\nu (p(R))}(p(R))$.

3 The Basic Algorithm

In this section we present the basic algorithm that we use to generate a set of rectangulations ${\mathcal C}_n\subseteq {\mathcal R}_n$.

3.1 Jumps in Rectangulations

To state the algorithm, we first introduce a local change operation that generalizes the three kinds of flips introduced in Sect. 2.2 (recall Fig. 3) and that will be applied when moving from one rectangulation in ${\mathcal C}_n$ to the next in the algorithm. A jump changes the insertion point for exactly one rectangle of the rectangulation. Formally, for a rectangulation $R\in {\mathcal R}_n$, we say that $R'\in {\mathcal R}_n$ differs from R by a right jump of rectangle $r_j$ by d steps, denoted , where $2\le j\le n$ and $d>0$, if one of the following conditions holds; see Fig. 10:

• $j=n$, and we have $p(R)=p(R')=:P\in {\mathcal R}_{n-1}$, $R=c_k(P)$, and $R'=c_{k+d}(P)$ for some $k>0$;

• $j<n$, and R and $R'$ are either both bottom-based or both right-based, and $p(R')$ differs from p(R) in a right jump of rectangle $r_j$ by d steps.

In words, the first condition asserts that the first $n-1$ rectangles in R and $R'$ form the same rectangulation $P\in {\mathcal R}_{n-1}$, and R and $R'$ are obtained by insertion from P using the kth and $(k+d)$th insertion point, respectively. The second condition asserts that R and $R'$ agree in the rectangle $r_n$, which either forms the bottom boundary or the right boundary of those rectangulations, and $p(R')$ differs from p(R) in a right jump with the same parameters.

Fig. 9

Jumps generalize wall slides, simple flips, and T-flips

A right jump as before is called minimal w.r.t. to a set of rectangulations ${\mathcal C}_n\subseteq {\mathcal R}_n$, if in the first condition above there is no index $\ell$ with $k<\ell <k+d$ such that $c_\ell (P)\in {\mathcal C}_n$.

A (minimal) left jump, denoted , is defined analogously by replacing $c_{k+d}$ by $c_{k-d}$ and $k<\ell <k+d$ by $k>\ell >k-d$ in the definitions above. Clearly, if $R'$ differs from R by a right jump of rectangle $r_j$ by d steps, then R differs from $R'$ by a left jump of rectangle $r_j$ by d steps, and vice versa, i.e., we have if and only if . We sometimes simply say that R and $R'$ differ in a jump, without specifying the direction left or right.

We state the following simple observations for further reference; see Fig. 9.

Lemma 3.1

Consider two rectangulations $R,R'\in {\mathcal R}_n$ that differ in a jump of rectangle $r_j$, define $P:=R^{[j-1]}=R'^{[j-1]}\in {\mathcal R}_{j-1}$, and let $q_k$ and $q_\ell$ be the insertion points in I(P) such that $R^{[j]}=c_k(P)$ and $R'^{[j]}=c_\ell (P)$.

• (a) If $q_k$ and $q_\ell$ are consecutive (w.r.t. I(P)) on a common wall of P, then R and $R'$ differ in a wall slide.

• (b) If $q_k$ lies on the last vertical wall and $q_\ell$ on the first horizontal wall of P (w.r.t. I(P)), then R and $R'$ differ in a simple flip.

• (c) If $q_k$ lies on a vertical wall and $q_\ell$ is the first insertion point on the next vertical wall of P (w.r.t. I(P)), or if $q_k$ lies on a horizontal wall and $q_\ell$ is the last insertion point on the previous horizontal wall, then R and $R'$ differ in a T-flip.

For any rectangulation $R\in {\mathcal R}_n$, we say that two insertion points from I(R) belong to the same vertical or horizontal group, if they lie on the same vertical or horizontal wall in R, respectively. In the sequence I(R), insertion points belonging to the same group appear consecutively.

Fig. 10

Illustration of jumps

3.2 Generating Rectangulations by Minimal Jumps

Consider the following algorithm that attempts to greedily generate a set of rectangulations ${\mathcal C}_n\subseteq {\mathcal R}_n$ using minimal jumps.

To illustrate how Algorithm J$^{\square }$ works, we consider the set of five rectangulations ${\mathcal C}_4=\{R_1,\ldots ,R_5\}\subseteq {\mathcal R}_4$ shown in Fig. 11. If initialized with $R_0:=R_1$, then the algorithm performs a left jump of rectangle 4 by one step (a right jump of rectangle 4 is impossible) to reach $R_2$, i.e., we have . In $R_2$, there are two options, either a right jump of rectangle 4 by one step, leading back to $R_1$, which has been visited before, or a left jump of rectangle 4 by two steps, leading to $R_3$, so we visit . In $R_3$, the jumps involving rectangle 4 lead to rectangulations that were visited before ($R_1$ and $R_2$). Moreover, a jump of rectangle 3 does not lead to a rectangulation in ${\mathcal C}_4$. However, a right jump of rectangle 2 by one step leads to $R_4$ (a left jump of rectangle 2 is impossible), so we visit . Finally, in $R_4$ a right jump of rectangle 4 by two steps leads to  (a left jump of rectangle 4 is impossible). In this example, Algorithm J$^{\square }$ successfully visits every rectangulation from ${\mathcal C}_4$ exactly once.

Fig. 11

Example execution of Algorithm J$^{\square }$

On the other hand, suppose we instead initialize the algorithm with $R_0:=R_3$. The algorithm will then visit followed by , and then terminates without success, as from $R_1$ no jump leads to an unvisited rectangulation from ${\mathcal C}_4$. Lastly, suppose we initialize Algorithm J$^{\square }$ with $R_0:=R_2$. As before, in $R_2$, there are two possibilities, either a right jump or a left jump of rectangle 4, both leading to an unvisited rectangulation from ${\mathcal C}_4$. Both are minimal jumps in opposite directions, and as the jump direction is ambiguous, the algorithm terminates immediately without success.

Remark 3.2

We do not recommend using Algorithm J$^{\square }$ in the stated form to generate a set of rectangulations efficiently! This is because the algorithm requires to maintain the list of all previously visited rectangulations (possibly exponentially many), and to look up this list in each step to check whether a rectangulation obtained by a jump from the current one has been visited before. For us, Algorithm J$^{\square }$ is merely a tool to define a Gray code ordering of the rectangulations in the given set ${\mathcal C}_n$ in way that is easy to remember (cf. [41]). In fact, in Sect. 5 we will present a modified algorithm that dispenses with the costly lookup operations, and that computes the very same sequence of rectangulations.

3.3 A Guarantee for Success

By definition, Algorithm J$^{\square }$ visits every rectangulation from a given set ${\mathcal C}_n\subseteq {\mathcal R}_n$ at most once, but it may terminate before having visited all. We now provide a sufficient condition guaranteeing that Algorithm J$^{\square }$ visits every rectangulation from ${\mathcal C}_n$ exactly once.

A set of generic rectangulations ${\mathcal C}_n\subseteq {\mathcal R}_n$ is called zigzag, if either $n=1$ and ${\mathcal C}_1=\{\square \}$, or if $n\ge 2$ and ${\mathcal C}_{n-1}:=\{p(R)\mid R\in {\mathcal C}_n\}$ is zigzag and for every $R\in {\mathcal C}_{n-1}$ we have $c_1(R)\in {\mathcal C}_n$ and $c_{\nu (R)}(R)\in {\mathcal C}_n$. In words, the set ${\mathcal C}_n$ must be closed under repeatedly deleting bottom-right rectangles and replacing them by rectangles inserted either below or to the right of the remaining ones; recall Fig. 1. The name ‘zigzag’ does not refer to the shape of a rectangulation, but to the order in which they are visited by Algorithm J$^{\square }$, which will become clear momentarily. We also say that ${\mathcal C}_n$ is symmetric, if reflection at the main diagonal is an involution of ${\mathcal C}_n$, i.e., if $R\in {\mathcal C}_n$, then the rectangulation obtained from R by reflection at the main diagonal is also in ${\mathcal C}_n$. We write  for the rectangulation that consists of n vertically stacked rectangles.

Fig. 12

Tree of generic rectangulations up to depth 3 with insertion points highlighted, where first and last insertion point are filled. The rectangulations in the dashed boxes at the bottom level ${\mathcal R}_4$ are stacked on top of each other due to space constraints, but they are children of a common parent node. Bottom- or right-based rectangulations, corresponding to insertion at the first or last insertion point, are marked by gray boxes

Theorem 3.3

Given any zigzag set of rectangulations ${\mathcal C}_n$ and initial rectangulation , Algorithm J$^{\square }$ visits every rectangulation from ${\mathcal C}_n$ exactly once. Moreover, if ${\mathcal C}_n$ is symmetric, then the ordering of rectangulations generated by Algorithm J$^{\square }$ is cyclic, i.e., the first and last rectangulation differ in a minimal jump.

The proof of Theorem 3.3 is provided in Sect. 8.

Note that the rectangulation is contained in every zigzag set by definition, so this is a valid initialization for Algorithm J$^{\square }$. We write $J^\square ({\mathcal C}_n)$ for the sequence of rectangulations generated by Algorithm J$^{\square }$ for a zigzag set ${\mathcal C}_n$ when initialized with .

It is easy to see that the number of distinct zigzag sets of generic rectangulations is at least $2^{|{\mathcal R}_n|(1-o(1))}\ge 2^{\Omega (11.56^n)}$ (the latter estimate uses the best known lower bound on $|{\mathcal R}_n|$ from [3]), i.e., at least double-exponential in n. In other words, Algorithm J$^{\square }$ exhaustively generates a given set of generic rectangulations in a vast number of cases. Moreover, many natural classes of rectangulations are in fact zigzag. In particular, all the different classes introduced in Sect. 2.2 and shown in Table 1 satisfy the aforementioned closure property. Moreover, all of these classes are symmetric, so for each of them we obtain cyclic jump orderings. Several such Gray codes are visualized in Appendix D.

3.4 Tree of Rectangulations

The notion of zigzag sets and the operation of Algorithm J$^{\square }$ can be interpreted combinatorially in the so-called tree of rectangulations, which is an infinite rooted tree, defined recursively as follows; see Fig. 12: The root of the tree is a single rectangle $\square \in {\mathcal R}_1$. For any node $R\in {\mathcal R}_{n-1}$, $n\ge 2$, of the tree we consider all insertion points of the rectangulation R, and the set of children of R in the tree is $\{c_i(R)\in {\mathcal R}_n\mid i=1,\ldots ,\nu (R)\}$. Conversely, the parent of each $R\in {\mathcal R}_n$, $n\ge 2$, is $p(R)\in {\mathcal R}_{n-1}$. In words, insertion leads to the children of a node, and deletion leads to the parent of a node. By Lemma 2.2, each generic rectangulation appears exactly once in the tree, and the set of nodes at distance n from the root of the tree is precisely the set ${\mathcal R}_{n+1}$ of generic rectangulations with $n+1$ rectangles. We emphasize that this tree is unordered, i.e., there is no specified ordering among the children of a node.

By Lemma 2.1, a node $R\in {\mathcal R}_n$ in the tree has at most $n+1$ children, i.e., we have $|{\mathcal R}_n|\le n!$. As we see from Fig. 12, this inequality is tight up to $n=4$, but starting from $n=4$, there are nodes $R\in {\mathcal R}_n$ with strictly less than $n+1$ children, i.e., we have $|{\mathcal R}_5|<5!$. In fact, it was shown in [3] that $|{\mathcal R}_n|={\mathcal O}(28.3^n)$.

A subset ${\mathcal C}_n\subseteq {\mathcal R}_n$ of nodes in depth $n-1$ of this tree is zigzag, if and only if it arises from the full tree of rectangulations by pruning some subtrees whose roots are neither bottom-based nor right-based rectangulations. In Fig. 12, all bottom-based or right-based rectangulations are highlighted by gray boxes, and can therefore not be pruned, while all other nodes can possibly be pruned. If no nodes are pruned, then we have ${\mathcal C}_n={\mathcal R}_n$, and if all possible nodes are pruned, then ${\mathcal C}_n$ is the set ${\mathcal B}_n$ of $2^{n-1}$ rectangulations obtained by repeatedly stacking a new rectangle either below or to the right of the previous ones, i.e., ${\mathcal B}_n=\{c_1(R),c_{\nu (R)}(R)\mid R\in {\mathcal B}_{n-1}\}$ for $n\ge 2$ and ${\mathcal B}_1=\{\square \}$. Moreover, we have ${\mathcal B}_n\subseteq {\mathcal C}_n\subseteq {\mathcal R}_n$ for any zigzag set ${\mathcal C}_n$.

The operation of Algorithm J$^{\square }$ for a zigzag set ${\mathcal C}_n$ as input can be interpreted as follows: Given the pruned tree corresponding to ${\mathcal C}_n$, we consider the set of nodes on all previous levels of the tree, i.e., the sets ${\mathcal C}_{i-1}:=\{p(R)\mid R\in {\mathcal C}_i\}$ for $i=n,n-1,\ldots ,2$, which are all zigzag sets by definition. Moreover, we consider the orderings $J^\square ({\mathcal C}_i)$, $i=1,\ldots ,n$, defined by Algorithm J$^{\square }$ for each of these sets. These sequences turn the unordered tree corresponding to ${\mathcal C}_n$ into an ordered tree, where the children $c_i(R)$ of each node R from left to right appear alternatingly in increasing order $i=1,\ldots ,\nu (R)$ or in decreasing order $i=\nu (R),\nu (R)-1,\ldots ,1$. Consequently, in the sequence $J^\square ({\mathcal C}_i)$, $i\ge 2$, which forms the left-to-right sequence of all nodes in depth $i-1$ of this ordered tree, the rectangle $r_i$ alternatingly jumps left and right between the first and last insertion point, which motivates the name ‘zigzag’ set; see also the animations provided in [45].

It is important to realize that these orderings are not consistent with respect to taking subsets, i.e., if we have two zigzag sets ${\mathcal C}_n'\subseteq {\mathcal C}_n$, then the entries of the sequence $J^\square ({\mathcal C}_n')$ do not necessarily appear in the same relative order in the sequence $J^\square ({\mathcal C}_n)$.

4 Pattern-Avoiding Rectangulations

In this section we show that Algorithm J$^{\square }$ applies to a large number of rectangulation classes that are defined by pattern avoidance, under some very mild conditions on the patterns; recall Table 1.

A rectangulation pattern is a configuration of walls with prescribed directions and incidences. For example, the windmill patterns  and  describe four walls such that when considering the walls in clockwise or counterclockwise order, respectively, the end vertex of one wall lies in the interior of the next wall. We can also think of a pattern as the rectangulation formed by the given walls and incidences. For example, we can think of the windmill patterns as rectangulations with five rectangles. We say that a rectangulation R contains the pattern P, if R contains a subset of walls with the directions and incidences specified by P. Otherwise we say that R avoids P. For any set of rectangulation patterns ${\mathcal P}$ and for any set of rectangulations ${\mathcal C}$, we write ${\mathcal C}({\mathcal P})$ for the rectangulations from ${\mathcal C}$ that avoid each pattern from ${\mathcal P}$. For example, diagonal rectangulations are given by . Examples of rectangulations containing and avoiding various patterns are shown in Fig. 4.

We say that a rectangulation pattern P is tame, if for any rectangulation R that avoids P, we also have that $c_1(R)$ and $c_{\nu (R)}(R)$ avoid P. In words, inserting a new rectangle below R or to the right of R must not create the forbidden pattern P. The next lemma follows directly from these definitions.

Lemma 4.1

If a rectangulation pattern is neither bottom-based nor right-based, then it is tame. In particular, each of the patterns is tame.

The following powerful theorem allows to obtain many new zigzag sets of rectangulations from a given zigzag set ${\mathcal C}_n\subseteq {\mathcal R}_n$ by forbidding one or more tame patterns. All of these zigzag sets can then be generated by our Algorithm J$^{\square }$.

Theorem 4.2

Let ${\mathcal C}_n\subseteq {\mathcal R}_n$ be a zigzag set of rectangulations, and let ${\mathcal P}$ be a set of tame rectangulation patterns. Then ${\mathcal C}_n({\mathcal P})$ is a zigzag set of rectangulations. Moreover, if ${\mathcal P}$ is symmetric, then ${\mathcal C}_n({\mathcal P})$ is symmetric.

Recall that ${\mathcal P}$ is symmetric if for each pattern $P\in {\mathcal P}$, we have that the pattern obtained from P by reflection at the main diagonal is also in ${\mathcal P}$. The significance of the second part of the theorem is that if ${\mathcal C}_n({\mathcal P})$ is symmetric, then the ordering of rectangulations of ${\mathcal C}_n({\mathcal P})$ generated by Algorithm J$^{\square }$ is cyclic by Theorem 3.3.

Proof

As ${\mathcal C}_n$ is a zigzag set of rectangulations, we know that ${\mathcal C}_{i-1}:=\{p(R)\mid R\in {\mathcal C}_i\}$ for $i=n,n-1,\ldots ,2$ are also zigzag sets. We argue by induction that ${\mathcal C}_i({\mathcal P})$ is also a zigzag set for all $i=1,\ldots ,n$. For the induction basis $i=1$ note that the rectangulation $\square$ that consists of a single rectangle has no walls, so it avoids any pattern, showing that ${\mathcal C}_1({\mathcal P})={\mathcal C}_1=\{\square \}$. For the induction step we assume that ${\mathcal C}_i({\mathcal P})$, $i\in \{1,\ldots ,n-1\}$, is a zigzag set, and we prove it for ${\mathcal C}_{i+1}({\mathcal P})$. Note that $\{p(R)\mid R\in {\mathcal C}_{i+1}({\mathcal P})\}={\mathcal C}_i({\mathcal P})$, and so we only need to check that $c_1(R)$ and $c_{\nu (R)}(R)$ are in ${\mathcal C}_{i+1}({\mathcal P})$ for all $R\in {\mathcal C}_i({\mathcal P})$, which is guaranteed by the assumption that each pattern $P\in {\mathcal P}$ is tame. This proves the first part of the theorem.

It remains to prove the second part. If $R\in {\mathcal C}_n({\mathcal P})$, then R avoids every pattern from ${\mathcal P}$. Let $R'$ be the rectangulation obtained from R by reflection at the main diagonal. $R'$ must also avoid every pattern from ${\mathcal P}$, because if it contained a pattern P from ${\mathcal P}$, then R would contain the corresponding reflected pattern $P'$, which is in ${\mathcal P}$ because of the assumption that ${\mathcal P}$ is symmetric. It follows that $R'\in {\mathcal C}_n({\mathcal P})$, completing the proof. $\square$

5 Efficient Computation

Recall from Remark 3.2 that Algorithm J$^{\square }$ in its stated form is unsuitable for efficient implementation. We now discuss how to make the algorithm efficient, so as to achieve the time bounds claimed in Table 1 for several interesting classes of rectangulations.

5.1 Memoryless Algorithm

Consider Algorithm M$^\square$ below, which takes as input a zigzag set of rectangulations ${\mathcal C}_n\subseteq {\mathcal R}_n$ and generates them exhaustively by minimal jumps in the same order as Algorithm J$^{\square }$, i.e., in the order $J^\square ({\mathcal C}_n)$. After initialization in line M1, the algorithm loops over lines M2–M5, visiting the current rectangulation R at the beginning of each iteration (line M2), until it terminates (line M3).

The key idea of the algorithm is to track explicitly which rectangle jumps in each step, and the direction of the jump. With this information, the jump is determined by the condition that it must be minimal w.r.t. ${\mathcal C}_n$, i.e., starting from the current insertion point of the given rectangle, we choose the first insertion point (w.r.t. their linear ordering) for that rectangle in the given direction that creates the next rectangulation from ${\mathcal C}_n$.

Table 2 Execution of Algorithm M$^\square$ for the set ${\mathcal C}_4={\mathcal D}_4$ of diagonal rectangulations with four rectangles. Empty entries in the o and s column are unchanged compared to the previous row

Specifically, the jump directions are maintained by an array $o=(o_1,\ldots ,o_n)$, where $o_j=\,\vartriangleleft \,$ means that rectangle $r_j$ performs a left jump in the next step, and $o_j=\,\vartriangleright \,$ means that rectangle $r_j$ performs a right jump in the next step (line M4). All sub-rectangulations of the initial rectangulation  are right-based, so the initial jump directions are $o_j=\,\vartriangleleft \,$ for $j=1,\ldots ,n$ (line M1). Whenever rectangle $r_j$ jumps left and reaches the first insertion point, which means that $R^{[j]}$ is bottom-based, or if it jumps right and reaches the last insertion point, which means that $R^{[j]}$ is right-based, then the jump direction $o_j$ is reversed (line M5).

The array $s=(s_1,\ldots ,s_n)$ is used to determine which rectangle jumps in each step. Specifically, the last entry $s_n$ determines the rectangle that jumps in the current iteration (line M3). This array simulates a stack in a loopless fashion, following an idea first used by Bitner et al. [7]. The stack is initialized by $(s_1,\ldots ,s_n)=(1,2,\ldots ,n)$ (line M1), with $s_n$ being the value on the top of the stack. The stack is popped (by the instruction $s_j\leftarrow s_{j-1}$ in line M5) when rectangle $r_j$ reaches its first or last insertion point in this step, meaning that this rectangle is not eligible to jump in the next step, but becomes eligible again after the next step, which is achieved by pushing the value j on the stack again (by the instructions $s_n\leftarrow n$ and $s_{j-1}\leftarrow j-1$ in line M5).

Table 2 shows the execution of Algorithm M$^\square$ with input ${\mathcal C}_4={\mathcal D}_4$ being the set of all diagonal rectangulations with four rectangles.

Theorem 5.1

For any zigzag set of rectangulations ${\mathcal C}_n\subseteq {\mathcal R}_n$, Algorithm M$^\square$ visits every rectangulation from ${\mathcal C}_n$ exactly once, in the order $J^\square ({\mathcal C}_n)$ defined by Algorithm J$^{\square }$.

The proof of Theorem 5.1 is provided in Sect. 8. To make meaningful statements about the running time of Algorithm M$^\square$, we need to specify the data structures used to represent the current rectangulation R, and the operations on this data structure to perform jumps in line M4 and to check the bottom-based and right-based property in line M5. Most importantly, we will develop oracles which efficiently compute the next minimal jump w.r.t. ${\mathcal C}_n$ for some interesting zigzag sets ${\mathcal C}_n$. One should think of ${\mathcal C}_n$ here as a class of rectangulations specified by some properties or forbidden patterns, such as ‘diagonal guillotine rectangulations’, and not as a large precomputed set of rectangulations. All of these details are described in the following sections, and they are part of our C++ implementation provided in [45].

6 Data Structures and Basic Functions

In the following we describe the data structures we use to represent and manipulate generic rectangulations, and the efficient implementation of jump operations using those data structures.

6.1 Data Structures

We represent a generic rectangulation with n rectangles as follows; see Fig. 13: Rectangles are stored in the variables $r_1,\ldots ,r_n$, indexed by the reverse deletion order described in Sect. 2.3 (recall Fig. 6). Vertices and edges are stored in variables $v_1,\dots ,v_{2n+2}$ and $e_1,\ldots ,e_{3n+1}$, respectively (indexed in no particular order).

Fig. 13

Data structures used to represent a generic rectangulation with $n=5$ rectangles. Edges are labeled black, and walls are labeled gray

Each vertex v points to the edges incident to it in the four directions by $v.\texttt {north}$, $v.\texttt {east}$, $v.\texttt {south}$, and $v.\texttt {west}$. Some of these can be 0, indicating that no edge is incident. This information determines the type $v.\texttt {type}$, which is one of ${\top }$, ${\vdash }$, ${\bot }$, ${\dashv }$, or 0 at the corner vertices of the rectangulation. We give all edges a default orientation from left to right, or from bottom to top. The $\texttt {dir}$ entry of each edge e specifies its direction, which is either $e.\texttt {dir}=\,\vartriangleright \,$ for a horizontal edge or $e.\texttt {dir}=\,\vartriangle \,$ for a vertical edge. Each edge e points to its two end vertices, specifically to its tail by $e.\texttt {tail}$ and to its head by $e.\texttt {head}$ (with respect to the default orientation). It also points to the previous and next edge, in the direction of its orientation, by $e.\texttt {prev}$ and $e.\texttt {next}$, respectively, which can be 0 if no such edge exists. The rectangle to the left and right side of an edge e, in the direction of its orientation, are stored in $e.\texttt {left}$ and $e.\texttt {right}$, which can be 0 at the boundary of the rectangulation. Each rectangle r points to its four corner vertices by $r.\texttt {neast}$, $r.\texttt {seast}$, $r.\texttt {swest}$, and $r.\texttt {nwest}$ in the corresponding directions.

For some rectangulation classes it is useful to store information about walls, i.e., maximal sequences of edges between two vertices that are not corners of the rectangulation. These are stored in $w_1,\ldots ,w_{n+3}$, where for simplicity we also keep track of the four maximal line segments between corners of the rectangulation (which are not walls in our definition). We also think of walls having a default orientation from left to right, or from bottom to top, and each wall w points to its first and last vertex by $w.\texttt {first}$ and $w.\texttt {last}$, respectively, in the direction of its orientation. Moreover, each edge e has an entry $e.\texttt {wall}$ pointing to the wall that contains it.

Remark 6.1

The aforementioned data structures are natural in the sense that they also capture the dual graph of the rectangulation, i.e., the graph obtained by replacing every rectangle by a vertex, and by joining any two vertices that correspond to rectangles sharing a common edge. This allows constructing the so-called transversal structure [17] (also known as regular edge labeling [24]), which is useful for computing a layout of the rectangulation; see Felsner’s survey [13]. Our data structures also allow to easily extract the twin binary tree representation of diagonal rectangulations described in [43].

We now use these data structures for implementing jumps efficiently. Recall the conditions stated in Lemma 3.1 when a jump is one of the three flip operations shown in Fig. 3. We refer to a jump as in (a), (b), or (c) in the lemma as a W-jump, S-jump, or T-jump, respectively. By these definitions, a W-jump is a special wall slide, an S-jump is a special simple flip, and a T-jump is a special T-flip. We refer to W-, S- and T-jump as local jumps collectively. Moreover, a W-jump or T-jump between two horizontal insertion points, or between two vertical insertion points, is referred to as a horizontal or vertical W- or T-jump, respectively.

Consider two rectangulations R and $R'$ that differ in a jump of rectangle $r_j$. If the jump is an S-jump or a horizontal T-jump, we let $h(R,R')$ denote the number of horizontal insertion points of $I(R^{[j-1]})=I(R'^{[j-1]})$ that lie in the interior of the top side of rectangle $r_j$ in R or $R'$; see Fig. 14. Similarly, if the jump is an S-jump or a vertical T-jump, we let $v(R,R')$ denote the number of vertical insertion points of $I(R^{[j-1]})=I(R'^{[j-1]})$ that lie in the interior of the left side of rectangle $r_j$ in R or $R'$.

Fig. 14

Illustration of Lemma 6.2

Lemma 6.2

Local jumps can be implemented with the following time guarantees:

• (a) A W-jump takes time ${\mathcal O}(1)$.

• (b) An S-jump between rectangulations R, $R'$ takes time ${\mathcal O}(h(R,R')+v(R,R')+1)$.

• (c) A horizontal T-jump between rectangulations R and $R'$ takes time ${\mathcal O}(h(R,R')+1)$ and a vertical T-jump takes time ${\mathcal O}(v(R,R')+1)$.

Clearly, every jump can be performed as a sequence of local jumps, and then the time bounds given by Lemma 6.2 can be added up.

Proof

The time bounds follow from the number of incidences that change during local jumps. The crucial point is that during a jump of rectangle $r_j$ between R and $R'$, all rectangles $r_k$ for $k=j+1,\ldots ,n$ are right-based or bottom-based in $R^{[k]}$ and $R'^{[k]}$, entailing that only constantly many incidences with such rectangles have to be modified. $\square$

6.2 Auxiliary Functions

In Sect. 6.3 we provide implementations of local jumps with the runtime guarantees stated in Lemma 6.2. Before doing so, we introduce some auxiliary functions to add and remove edges from a rectangulation. These auxiliary functions only update the incidences between edges, vertices and walls, but not the incidences between rectangles and any other objects and the type of vertices (this will be done separately later).

The following function $\texttt {remHead}(\beta )$ removes the edge $e_\beta$ together with its head vertex.

After defining some auxiliary variables in the first step, the function $\texttt {remHead}(\beta )$ updates the incidences between edges and vertices in the second step, and the incidences between walls and edges in the third step. We also define an analogous function $\texttt {remTail}(\beta )$ that removes $e_\beta$ and its tail instead of its head. For details, see our C++ implementation [45].

The following two functions $\texttt {insBefore}(\beta ,a,\gamma )$ and $\texttt {insAfter}(\alpha ,a,\beta )$ insert the edge $e_\beta$ with head $v_a$ or tail $v_a$, respectively, before or after the edge $e_\gamma$ or $e_\alpha$.

6.3 Local Jumps

Armed with these auxiliary functions, we now tackle the implementation of local jumps with the time guarantees stated in Lemma 6.2. Each of the functions $\texttt {Wjump}(R,j,d,\alpha )$, $\texttt {Sjump}(R,j,d,\alpha )$, and $\texttt {Tjump}(R,j,d,\alpha )$ below takes as input the current rectangulation R in which the jump is performed, the index j of the rectangle $r_j$ to be jumped, the direction $d\in \{\!\,\vartriangleleft \,,\!\,\vartriangleright \,\!\}$ of the jump, and the index $\alpha$ of the edge $e_\alpha$ which contains the insertion point of $R^{[j-1]}$ that will become the top-left vertex of $r_j$ after the jump. In the pseudocode of these algorithms, all references to rectangles $r_i$, edges $e_i$, vertices $v_i$ or walls $w_i$ are with respect to the current rectangulation R.

We first present the implementation of W-jumps. For simplicity, we only show the implementation of left horizontal W-jumps in the function $\texttt {Wjump}_{\texttt {h}}(R,j,\!\,\vartriangleleft \,\!,\alpha )$ below; see Fig. 15a. The implementation of right horizontal W-jumps, and of left and right vertical W-jumps in a function $\texttt {Wjump}_{\texttt {v}}(R,j,d,\alpha )$ is very similar; we omit the details here.

The running time of $\texttt {Wjump}_{\texttt {h}}(R,j,\!\,\vartriangleleft \,\!,\alpha )$ is clearly ${\mathcal O}(1)$, as claimed in part (a) of Lemma 6.2.

We proceed with the implementation of S-jumps. For simplicity, we only provide the implementation of left S-jumps in the function $\texttt {Sjump}(R,j,\!\,\vartriangleleft \,\!,\alpha )$ below; see Fig. 15b. The implementation of right S-jumps is very similar.

Fig. 15

Implementation of local jumps: a horizontal W-jumps; b S-jumps; c horizontal T-jumps. Vertices are drawn as squares, insertion points as circles

Let $R'$ be the rectangulation obtained from R by one call of $\texttt {Sjump}(R,j,\,\vartriangleleft \,,\alpha )$. The running time of this call is ${\mathcal O}(h(R,R')+v(R,R')+1)$, as claimed in part (b) of Lemma 6.2. This time is incurred by the while-loops in step 3. Specifically, the first while-loop is iterated exactly $h(R,R')$ times, and the second while-loop is iterated exactly $v(R,R')+1$ times.

We complete this section by presenting the implementation of T-jumps; see Fig. 15c. For simplicity, we only provide the implementation of left horizontal T-jumps in the function $\texttt {Tjump}_{\texttt {h}}(R,j,\,\vartriangleleft \,,\alpha )$ below. The implementation of right horizontal T-jumps, and of left and right vertical T-jumps in a function $\texttt {Tjump}_{\texttt {v}}(R,j,d,\alpha )$ is very similar.

Let $R'$ be the rectangulation obtained from R by one call of $\texttt {Tjump}_{\texttt {h}}(R,j,\,\vartriangleleft \,,\alpha )$. The running time of this call is ${\mathcal O}(h(R,R')+1)$, as claimed in part (c) of Lemma 6.2. This time is incurred by the while-loop in step 3, which is iterated exactly $h(R,R')+1$ times.

7 Minimal Jump Oracles

A minimal jump oracle is a function that is called in line M4 of Algorithm M$^\square$ to compute a jump in the current rectangulation R that is minimal respect to the given zigzag set of rectangulations ${\mathcal C}_n\subseteq {\mathcal R}_n$. In this section we specify such oracles for the zigzag sets ${\mathcal C}_n$ mentioned in Table 1, which allows us to establish the runtime bounds for Algorithm M$^\square$ stated in the last column of the table (except for block-aligned rectangulations, which are handled in Appendix B). A minimal jump oracle has the form $\texttt {next}_{{\mathcal C}_n}(R,j,d)$, and this function call performs in the current rectangulation R a jump of rectangle $r_j$ in direction d that is minimal w.r.t. ${\mathcal C}_n$, and the function will modify R accordingly. Depending on ${\mathcal C}_n$, our minimal jump oracles perform a suitable W-, S-, or T-jump, or a combination thereof, as implemented in the previous section.

7.1 Generic Rectangulations

We first consider the case ${\mathcal C}_n={\mathcal R}_n$ of generic rectangulations. Given the current rectangulation R, upon a jump of rectangle $r_j$ in direction d, every insertion from $I(R^{[j-1]})$ is used, so we simply need to detect the next one.

By Lemma 3.1, a W-jump occurs between any two consecutive (w.r.t. $I(R^{[j-1]})$) insertion points belonging to the same horizontal or vertical group, an S-jump occurs between the last vertical insertion point and the first horizontal insertion point, and a T-jump occurs between the last insertion point of a group and the first insertion point of the next group, if both groups are vertical or horizontal. Specifically, suppose there are $\lambda$ vertical groups and $\mu$ horizontal groups with cardinalities $g_k$, $k=1,\ldots ,\lambda$, and $h_k$, $k=1,\ldots ,\mu$, respectively in $I(R^{[j-1]})$ (note that $g_1=h_\mu =1$). Then the jump sequence consisting of letters $\{W,S,T\}$ that specifies the types of jumps performed with rectangle $r_j$ from the first to the last insertion point is

$$\begin{aligned} (TW^{g_2-1})(TW^{g_3-1})\cdots (TW^{g_\lambda -1})\,S\,(W^{h_1-1}T)(W^{h_2-1}T)\cdots (W^{h_{\mu -1}-1}T); \end{aligned}$$

(1)

see Fig. 16a. Of course, during Algorithm M$^\square$, these jump operations are not consecutive, but they are interleaved with the jump sequences of other rectangles $r_k$, $k>j$. The details are spelled out in the function $\texttt {next}_{{\mathcal R}_n}(R,j,d)$.

The four distinct cases treated in lines N2–N4 come from the directions $d\in \{\!\,\vartriangleleft \,\!,\!\,\vartriangleright \,\!\}$ and whether the jump is horizontal or vertical. The latter condition is determined in line N1 by querying the type of the top-left vertex of $r_j$, which is either ${\top }$ or ${\vdash }$. Note that the code in lines N2 and N4 is symmetric by reflecting all directions at the main diagonal. The same holds for the code in lines N3 and N5.

Lemma 7.1

Consider a rectangulation $P\in {\mathcal R}_{n-1}$ with $\nu =\nu (P)$ insertion points. Then calling $\texttt {next} _{{\mathcal R}_n}(R,n,\!\,\vartriangleright \,\!\!)$ exactly $\nu -1$ times with initial rectangulation $R=c_1(P)$, yields $c_i(P)$ for $i=1,\ldots ,\nu$, and the total time for these calls is ${\mathcal O}(\nu )$. An analogous statement holds for $\texttt {next} _{{\mathcal R}_n}(R,n,\!\,\vartriangleleft \,\!)$.

Proof

If the sequence of insertion points I(P) has $\lambda$ vertical groups and $\mu$ horizontal groups of cardinalities $g_k$, $k=1,\ldots ,\lambda$, and $h_k$, $k=1,\ldots ,\mu$, respectively, then the sequence of jumps performed by the calls to $\texttt {next}_{{\mathcal R}_n}$ has the form (1). We clearly have $\nu =\sum _{k=1}^\lambda g_k+\sum _{k=1}^\mu h_k$. We use Lemma 6.2 to bound the overall time to perform those operations; see Fig. 16a. The number of W-jumps in (1) is $w:=\sum _{k=1}^\lambda (g_k-1)$ $+\sum _{k=1}^\mu (h_k-1)\le \nu$. The sum of the terms $v(R,R')+1$ and $h(R,R')+1$ over any two consecutive rectangulations $R,R'$ in this sequence that differ in a T-jump is $t:=\sum _{k=1}^{\lambda -1}g_k$ and $t':=\sum _{k=2}^\mu h_k$, respectively. The sum $v(R,R')+h(R,R')+1$ for the two consecutive rectangulations $R,R'$ in this sequence that differ in an S-jump is $s:=g_\lambda +h_1-1$. Clearly, we have $s+t+t'\le \nu$. Consequently, the overall time for those operations is ${\mathcal O}(w+s+t+t')={\mathcal O}(\nu )$, as claimed. $\square$

Theorem 7.2

Algorithm M$^\square$ with the minimal jump oracle $\texttt {next} _{{\mathcal R}_n}$ takes time ${\mathcal O}(1)$ on average to visit each generic rectangulation.

Fig. 16

Illustration of the proofs of a Theorem 7.2 for generic rectangulations and b Theorem 7.5 for diagonal rectangulations

Proof

For some fixed $j\in \{2,\ldots ,n\}$, we consider all jumps of rectangle $r_j$. Whenever rectangle $r_j$ jumps in a rectangulation $R\in {\mathcal R}_n$, then $R^{[k]}$ is either bottom-based or right-based for all $k=j+1,\ldots ,n$. Moreover, none of the rectangles $k=j-1,$ $j-2,\ldots ,2$ jumps unless $R^{[j]}$ is bottom-based or right-based. Consequently, we can partition the jumps of $r_j$ in the entire jump sequence into subsequences, such that in each subsequence, for each rectangulation $R\in {\mathcal R}_n$ of the subsequence, $R^{[j-1]}\in {\mathcal R}_{j-1}$ is the same subrectangulation and in R the rectangle $r_j$ jumps to the next insertion point of $I(R^{[j-1]})$. By Lemma 7.1, the total time for visiting the $\nu =\nu (R^{[j-1]})$ many rectangulations of this subsequence is ${\mathcal O}(\nu )$, which is ${\mathcal O}(1)$ on average. $\square$

Remark 7.3

By slightly modifying our data structures, we could even obtain a loopless algorithm for generic rectangulations. The idea is to introduce an additional data structure called sides. Each rectangle is subdivided into four sides, and in the incidence relations, sides sit between edges and rectangles, i.e., edges do not point to the two touching rectangles directly, but to the relevant sides of those rectangles, and each side points to the rectangle it belongs to. During S-jumps and T-jumps, a rectangle can be broken up into its four sides and the sides of two rectangles can be interchanged in constant time, avoiding the while-loops in the functions $\texttt {Sjump}$ and $\texttt {Tjump}$ that need to update possibly linearly many incidences between edges and rectangles. To keep the presentation simple, we do not show these modifications. Also, the resulting improvement is not substantial, and sides are a somewhat artificial concept.

7.2 Diagonal Rectangulations

Recall that in a diagonal rectangulation $R\in {\mathcal D}_n$, every rectangle intersects the main diagonal, or equivalently, R avoids the patterns  and  . Consequently, during a jump of rectangle  $r_j$ in the current rectangulation R, we need to consider precisely the insertion points from $I(R^{[j-1]})$ that are the first insertion point of a vertical group, or the last insertion point of a horizontal group, as any other insertion point from each group would create one of the forbidden pattern; see Fig. 16b. Consequently, if the sequence $I(R^{[j-1]})$ has $\lambda$ vertical groups and $\mu$ horizontal groups, then the jump sequence that specifies the types of jumps with rectangle $r_j$ from the first to the last insertion point is

$$\begin{aligned} T^{\lambda -1}\,S\,T^{\mu -1}. \end{aligned}$$

In particular, we do not perform any wall slides. An implementation of this is provided in the function $\texttt {next}_{{\mathcal D}_n}(R,j,d)$.

Similarly to before, the code in lines N2 and N4, and in lines N3 and N5 is symmetric by reflecting all directions at the main diagonal. For diagonal rectangulations the runtime analysis is straightforward and gives a loopless algorithm.

Lemma 7.4

Each call $\texttt {next} _{{\mathcal D}_n}(R,j,d)$ takes time ${\mathcal O}(1)$.

Proof

Let $R'$ be the rectangulation after the call $\texttt {next}_{{\mathcal D}_n}(R,j,d)$, which differs from R in an S-jump or T-jump. As we only consider the first insertion point of each vertical group and the last insertion point of each horizontal group of $I(R^{[j-1]})$, we have $v(R,R')=0$ and $h(R,R')=0$. The claim now follows from Lemma 6.2, (b) and (c). $\square$

Lemma 7.4 immediately yields the following result.

Theorem 7.5

Algorithm M$^\square$ with the minimal jump oracle $\texttt {next} _{{\mathcal D}_n}$ takes time ${\mathcal O}(1)$ to visit each diagonal rectangulation.

Remark 7.6

Jumps as performed by the oracles $\texttt {next}_{{\mathcal R}_n}$ and $\texttt {next}_{{\mathcal D}_n}$ and shown in Fig. 16 correspond to cover relations in the lattice of generic rectangulations and the lattice of diagonal rectangulations described by Meehan [29] and Law and Reading [27], respectively, which both arise as lattice quotients of the weak order on the symmetric group. Consequently, our cyclic Gray codes correspond to Hamilton cycles in the cover graphs of those lattices. In [22] we showed that our permutation language framework can be used to generate a Hamilton path on every lattice quotient of the weak order, which also yields a Hamilton path on the skeleton of the corresponding polytope [35] (see also [34]).

7.3 Pattern-Avoiding Rectangulations

For any zigzag set of rectangulations ${\mathcal C}_n\subseteq {\mathcal R}_n$, and any set of tame patterns ${\mathcal P}$, Theorem 4.2 guarantees that the set ${\mathcal C}_n({\mathcal P})$ of rectangulations that avoid all patterns from ${\mathcal P}$ is also a zigzag set. We now describe how we can obtain a minimal jump oracle for ${\mathcal C}_n({\mathcal P})$ from a minimal jump oracle $\texttt {next}_{{\mathcal C}_n}(R,j,d)$ for ${\mathcal C}_n$. The idea is simply to perform a minimal jump of $r_j$ w.r.t. ${\mathcal C}_n$, and to test after each jump whether the resulting rectangulation contains any pattern from ${\mathcal P}$, repeating this process until we arrive at a rectangulation that avoids all patterns from ${\mathcal P}$. This is guaranteed to terminate after at most $j\le n$ iterations, as the first and last insertion point of $I(R^{[j-1]})$ will produce rectangulations that avoid all patterns from ${\mathcal P}$, due to the zigzag property.

We immediately obtain the following generic runtime bounds.

Theorem 7.7

Let ${\mathcal P}$ be a finite set of tame rectangulation patterns. If the zigzag set ${\mathcal C}_n$ has a minimal jump oracle $\texttt {next} _{{\mathcal C}_n}$ that runs in time $f_n$, and containment of any pattern from ${\mathcal P}$ in R can be tested in time $t_n$, then $\texttt {next} _{{\mathcal C}_n({\mathcal P})}$ is a minimal jump oracle for ${\mathcal C}_n({\mathcal P})$ that runs in time ${\mathcal O}(n(f_n+t_n))$.

In some cases the runtime bound for at most n consecutive calls of $\texttt {next}_{{\mathcal C}_n}(R,j,d)$ or several consecutive pattern containment tests can be improved upon the trivial bounds ${\mathcal O}(nf_n)$ and ${\mathcal O}(nt_n)$, respectively (see the proof of Theorem 7.9 below). Moreover, for some patterns further optimizations of the function $\texttt {next}_{{\mathcal C}_n({\mathcal P})}(R,j,d)$ are possible. For example, the property of R to be guillotine is invariant under W-jumps and S-jumps, so if R is found to contain one of the windmill patterns, then we only need to check containment after the next T-jump performed by the call $\texttt {next}_{{\mathcal C}_n}(R,j,d)$. Most importantly, when testing for containment of a pattern, we only need to check incidences of walls involving the rectangle $r_j$. In the following, we provide functions $\texttt {contains}(R,j,P)$ that test whether R contains one of the tame patterns P listed in Lemma 4.1 after a sequence of jumps of rectangle $r_j$ from a rectangulation that avoids P. We emphasize here that these functions only work under these assumptions, and are not suitable for general pattern containment testing of arbitrary rectangulations, but only for use within our algorithm M$^\square$.

We first present an implementation of such a containment testing function $\texttt {contains}(R,j,P)$ for the clockwise windmill ; see Fig. 17a. It uses the wall data structure $w_1,\ldots ,w_{n+3}$ to quickly move to the end vertex of a wall (without traversing the possibly many edges along the wall).

The function that tests for the counterclockwise windmill is symmetric, and is not shown here for simplicity.

Fig. 17

Testing containment of the patterns a , b , and c . The forbidden configuration of walls is highlighted

The next two functions test for containment of the patterns and , respectively; see Fig. 17, b and c.

Similarly to before, testing for the patterns  and  is symmetric to the previous two cases, so we omit those implementations.

It remains to provide containment testing for the patterns  and . We only show the first case, as the other is symmetric; see Fig. 18.

Lines C2–C4 are essentially a triply nested loop that moves along the edges of the vertical wall x that contains the left side of $r_j$ (line C2), the edges of each horizontal wall y whose right end vertex lies on x (line C3), and the edges of each vertical wall z whose bottom end vertex lies on y, searching for a vertex of type ${\dashv }$ on z (line C4); see Fig. 18. This is realized by repeatedly calling line C3 from within the while-loop in line C2, and upon completion returning to from where the call occurred. Similarly, line C4 is repeatedly called from within the while-loop in line C3, and upon completion it returns to from where it was called.

Fig. 18

Testing containment of the pattern . The forbidden configuration of walls is highlighted

The aforementioned functions have the following runtime guarantees.

Lemma 7.8

The function $\texttt {contains} (R,j,P)$ takes time ${\mathcal O}(1)$ for the patterns and time ${\mathcal O}(n)$ for the patterns .

Proof

For the first six patterns the statement is obvious, as the specified functions only make constantly many changes to our data structures. For the pattern , note that each edge index of the rectangulation R is assigned to one of the variables $\beta ,\gamma ,\delta$ at most once during the call $\texttt {contains}(R,j,P)$ in lines C2–C4, and the number of edges of R is $3n+1={\mathcal O}(n)$. For the pattern the argument is the same. $\square$

The next theorem combines all the observations from this section, thus establishing most of the runtime bounds stated in Table 1.

Theorem 7.9

Let and . Algorithm M$^\square$ with the minimal jump oracle $\texttt {next} _{{\mathcal R}_n({\mathcal P})}$ or $\texttt {next} _{{\mathcal D}_n({\mathcal P})}$ visits each rectangulation from ${\mathcal R}_n({\mathcal P})$ or ${\mathcal D}_n({\mathcal P})$, respectively, in time ${\mathcal O}(n)$ for any set of patterns ${\mathcal P}\subseteq {\mathcal P}_1$ and in time ${\mathcal O}(n^2)$ for any set of patterns ${\mathcal P}\subseteq {\mathcal P}_1\cup {\mathcal P}_2$.

All the bounds stated in Theorem 7.9 hold in the worst case (not just on average).

Proof

We first consider the minimal jump oracle $\texttt {next}_{{\mathcal D}_n({\mathcal P})}$, which repeatedly calls the function $\texttt {next}_{{\mathcal D}_n}$ described in Sect. 7.2. Applying Theorem 7.7 with the bound $f_n={\mathcal O}(1)$ from Lemma 7.4 and the bounds $t_n={\mathcal O}(1)$ for ${\mathcal P}\subseteq {\mathcal P}_1$ and $t_n={\mathcal O}(n)$ for ${\mathcal P}\subseteq {\mathcal P}_1\cup {\mathcal P}_2$ from Lemma 7.8, the term $n(f_n+t_n)$ evaluates to ${\mathcal O}(n)$ or ${\mathcal O}(n^2)$, respectively, as claimed.

We now consider the minimal jump oracle $\texttt {next}_{{\mathcal R}_n({\mathcal P})}$, which repeatedly calls $\texttt {next}_{{\mathcal R}_n}$ described in Sect. 7.1. In this case applying Theorem 7.7 directly would not give the desired bounds, so we have to refine the analysis of the while-loop in the algorithm $\texttt {next}_{{\mathcal R}_n({\mathcal P})}$. Specifically, we consider the sequence of calls to $\texttt {next}_{{\mathcal R}_n}(R,j,d)$ from one rectangulation $R\in {\mathcal R}_n$ avoiding all patterns from ${\mathcal P}$ until the next one. The length of this sequence is at most $\nu =\nu (R^{[j-1]})\le n$ (recall Lemma 2.1), and by Lemma 7.1 the total time of all calls to $\texttt {next}_{{\mathcal R}_n}(R,j,d)$ is ${\mathcal O}(\nu )={\mathcal O}(n)$. The total time of all pattern containment tests is at most $\nu t_n\le nt_n$, which is ${\mathcal O}(n)$ for ${\mathcal P}\subseteq {\mathcal P}_1$ and ${\mathcal O}(n^2)$ for ${\mathcal P}\subseteq {\mathcal P}_1\cup {\mathcal P}_2$ by Lemma 7.8. This proves the claimed bounds, completing the proof of the theorem. $\square$

Remark 7.10

For we have ${\mathcal R}_n({\mathcal P})={\mathcal D}_n$, so we could use, or rather ‘misuse’, the minimal jump oracle $\texttt {next}_{{\mathcal R}_n({\mathcal P})}$ to generate ${\mathcal D}_n$. However, this would give a worse guarantee of ${\mathcal O}(n)$ time per visited diagonal rectangulation, rather than ${\mathcal O}(1)$ for $\texttt {next}_{{\mathcal D}_n}$ as guaranteed by Theorem 7.5.

Remark 7.11

We write $c({\mathcal C}_{n-1}({\mathcal P}))$ for the set of all rectangulations from ${\mathcal C}_n$ that are obtained by inserting a rectangle into a rectangulation from ${\mathcal C}_{n-1}({\mathcal P})$. To assess the runtime bounds stated in Theorem 7.9, one may try to investigate the quantity $|c({\mathcal C}_{n-1}({\mathcal P})|/|{\mathcal C}_n({\mathcal P})|$. This is a lower bound for the average number of iterations of the while-loop of the algorithm $\texttt {next}_{{\mathcal C}_n({\mathcal P})}$ before it returns a rectangulation from ${\mathcal C}_n({\mathcal P})$. Experimentally, we found that this ratio grows with n in many cases, though maybe not linearly with n, hinting at the possibility that the time bounds stated in Theorem 7.9 are too pessimistic and can be improved in an average case analysis.

8 Proofs of Theorems 3.3 and 5.1

In this section we present the proofs of Theorems 3.3 and 5.1. For this purpose we first recap the exhaustive generation framework for permutation languages developed in [20, 21]. Definitions and terminology intentionally parallel the corresponding definitions given for rectangulations before, and the connection between rectangulations and permutations will be made precise in Lemma 8.7 below.

8.1 Permutation Basics

For any two integers $a\le b$ we define $[a,b]:=\{a,a+1,\ldots ,b\}$, and we refer to a set of this form as an interval. We also define $[n]:=[1,n]=\{1,\ldots ,n\}$. We write $S_n$ for the set of permutations on [n], and we write $\pi \in S_n$ in one-line notation as $\pi =\pi (1)\pi (2)\cdots \pi (n)=a_1a_2\cdots a_n$. Moreover, we use $\varepsilon \in S_0$ to denote the empty permutation, and ${{\,\textrm{id}\,}}_n=12\cdots n\in S_n$ to denote the identity permutation.

Given two permutations $\pi$ and $\tau$, we say that $\pi$ contains the pattern $\tau$, if there is a subsequence of $\pi$ whose elements have the same relative order as in $\tau$. Otherwise we say that $\pi$ avoids $\tau$. For example, contains the pattern $\tau =231$, as the highlighted entries show, whereas $\pi =654123$ avoids $\tau =231$. In a vincular pattern $\tau$, there is exactly one underlined pair of consecutive entries, with the interpretation that the underlined entries must match adjacent positions in $\pi$. For instance, the permutation contains the pattern $\tau =231$, but it avoids the vincular pattern $\tau =\underline{23}1$.

8.2 Deletion, Insertion, and Jumps in Permutations

For $\pi \in S_n$, $n\ge 1$, we write $p(\pi )\in S_{n-1}$ for the permutation obtained from $\pi$ by deleting the largest entry n. We also define $\pi ^{[i]}:=p^{n-i}(\pi )$ for $i=1,\ldots ,n$. Moreover, for any $\pi \in S_{n-1}$ and any $1\le i\le n$, we write $c_i(\pi )\in S_n$ for the permutation obtained from $\pi$ by inserting the new largest value n at position i of $\pi$, i.e., if $\pi =a_1\cdots a_{n-1}$ then $c_i(\pi )=a_1\cdots a_{i-1}\, n\, a_i \cdots a_{n-1}$. For example, for $\pi =412563$ we have $p(\pi )=41253$ and $c_1(\pi )=7412563$, $c_5(\pi )=4125763$, and $c_7(\pi )=4125637$. Given a permutation $\pi =a_1\cdots a_n$ with a substring $a_i\cdots a_{i+d}$ with $d>0$ and $a_i>a_{i+1},\ldots ,a_{i+d}$, a right jump of the value $a_i$ by d steps is a cyclic left rotation of this substring by one position to $a_{i+1}\cdots a_{i+d} a_i$. Similarly, given a substring $a_{i-d}\cdots a_i$ with $d>0$ and $a_i>a_{i-d},\ldots ,a_{i-1}$, a left jump of the value $a_i$ by d steps is a cyclic right rotation of this substring to $a_i a_{i-d}\cdots a_{i-1}$. For example, a right jump of the value 5 in the permutation 265134 by two steps yields 261354.

We say that a jump is minimal w.r.t. a set of permutations $L_n\subseteq S_n$, if every jump of the same value in the same direction by fewer steps creates a permutation that is not in $L_n$.

8.3 Generating Permutations by Minimal Jumps

Consider the following analogue of Algorithm J$^{\square }$ for greedily generating a set of permutations $L_n\subseteq S_n$ using minimal jumps.

The following results were proved in [21]. A set of permutations $L_n\subseteq S_n$ is called a zigzag language, if either $n=0$ and $L_0=\{\varepsilon \}$, or if $n\ge 1$ and $L_{n-1}\!:=\!\{p(\pi )\!\mid \!\pi \!\in \!L_n\}$ is a zigzag language and for every $\pi \in L_{n-1}$ we have $c_1(\pi )\in L_n$ and $c_n(\pi )\in L_n$.

We now define a sequence $J(L_n)$ of all permutations from a zigzag language $L_n\subseteq S_n$. For any $\pi \in L_{n-1}$ we let be the sequence of all $c_i(\pi )\in L_n$ for $i=1,2,\ldots ,n$, starting with $c_1(\pi )$ and ending with $c_n(\pi )$, and we let denote the reverse sequence, i.e., it starts with $c_n(\pi )$ and ends with $c_1(\pi )$. In words, those sequences are obtained by inserting into $\pi$ the new largest value n in all possible positions from left to right, or from right to left, respectively, in all possible positions that yield a permutation from $L_n$, skipping the positions that yield a permutation that is not in $L_n$. If $n=0$ then we define $J(L_0):=\varepsilon$, and if $n\ge 1$ then we consider the finite sequence $J(L_{n-1})=:\pi _1,\pi _2,\ldots$ and define

(2)

i.e., this sequence is obtained from the previous sequence by inserting the new largest value n in all possible positions alternatingly from right to left, or from left to right.

Theorem 8.1

[21, Thm. 1 + Lem. 4] Given any zigzag language of permutations $L_n$ and initial permutation $\pi _0={{\,\textrm{id}\,}}_n$, Algorithm J visits every permutation from $L_n$ exactly once, in the order $J(L_n)$ defined by (2). Moreover, if $|L_i|$ is even for all $2\le i\le n-1$, then the sequence $J(L_n)$ is cyclic, i.e., the first and last permutation differ in a minimal jump.

A permutation $\pi$ is called 2-clumped if it avoids each of the vincular patterns $3\underline{51}24$, $3\underline{51}42$, $24\underline{51}3$, and $42\underline{51}3$. We write $S_n'\subseteq S_n$ for the set of 2-clumped permutations.

Lemma 8.2

[21, Thm. 8 + Lem. 10] We have $S_0'=\{\varepsilon \}$, and for every $n\ge 1$ we have $S_{n-1}'=\{p(\pi )\mid \pi \in S_n'\}$ and $S_n'\supseteq \{c_1(\pi ),c_n(\pi )\mid \pi \in S_{n-1}'\}$. In particular, $S_n'$ is a zigzag language for all $n\ge 0$.

We also state the following observations for further reference.

Lemma 8.3

The sequence of permutations $J(L_n)$ defined in (2) has the following properties:

• (a) The first permutation in $J(L_n)$ is the identity permutation ${{\,\textrm{id}\,}}_n$.

• (b) For $j=2,\ldots ,n$, the first jump of the value j in $J(L_n)$ is a left jump.

• (c) Every jump in the sequence $J(L_n)$ is minimal w.r.t. $L_n$.

• (d) Given two consecutive permutations $\pi ,\rho$ in $J(L_n)$ that differ in a jump of some value j, then we have $\pi ^{[k]}=c_1(\pi ^{[k-1]})$ and $\rho ^{[k]}=c_1(\rho ^{[k-1]})$, or $\pi ^{[k]}=c_k(\pi ^{[k-1]})$ and $\rho ^{[k]}=c_k(\rho ^{[k-1]})$ for all $k=j+1,\ldots ,n$.

• (e) Let $\pi ,\rho$ be two consecutive permutations in $J(L_n)$ such that $\rho$ is obtained from $\pi$ by a left jump of some value j, and let $\pi ',\rho '$ be the next two consecutive permutations in $J(L_n)$ that differ in a jump of j. If j is not at the first position in $\rho$ and the value left of it is smaller than j, then $\rho '$ is obtained from $\pi '$ by a left jump. Conversely, if j is at the first position in $\rho$ or the value left of it is bigger than j, then $\rho '$ is obtained from $\pi '$ by a right jump. An analogous statement holds with left and right interchanged.

The proof of Lemma 8.3 is deferred to Appendix A.

8.4 A Surjection from Permutations to Generic Rectangulations

Observe that a diagonal rectangulation with n rectangles can be laid out canonically so that each rectangle intersects the main diagonal in a 1/n-fraction. Specifically, rectangle $r_i$ intersects the main diagonal in the ith such line segment counted from top-left to bottom-right, for $i=1,\ldots ,n$. We say that $r_i$ is left-fixed or left-extended, if its left side touches or does not touch the diagonal, respectively. These notions are defined analogously for all the other three sides right, bottom, and top.

Fig. 19

Illustration of the mapping $\rho :S_n\rightarrow {\mathcal D}_n$ for the permutation $\pi =(8,13,7,5,11,2,14,6,15,9, 10,3,1,4,12)$ (example from [36])

We begin by reviewing a mapping $\rho :S_n\rightarrow {\mathcal D}_n$, $n\ge 1$, from permutations to diagonal rectangulations, first described by Law and Reading [27]. Maps closely related to $\rho$ have appeared previously in the literature, see e.g. [2, 14]. We consider the outer rectangle and divide its main diagonal into n equally sized line segments numbered $1,\ldots ,n$ from top-left to bottom-right. Given a permutation $\pi =a_1\cdots a_n \in S_n$, the diagonal rectangulation $\rho (\pi )$ is obtained as follows; see Fig. 19: For $i=1,\ldots ,n$, in step i we add the rectangle $r_{a_i}$ such that it intersects the main diagonal precisely in the $a_i$th line segment, and such that the rectangle is maximal w.r.t. the property that the rectangles $r_{a_1}\cup \cdots \cup r_{a_i}$ form a staircase, which means that the bottom-left boundary of $r_{a_1}\cup \cdots \cup r_{a_i}$ is an L-shape, and the top-right boundary is a non-increasing polygonal line.

With any wall w of a generic rectangulation $R\in {\mathcal R}_n$ we associate a wall shuffle $\sigma (w)$, which is a permutation of a subset of the rectangles that share a side with w, defined as follows; see Fig. 20. If the wall w is horizontal, we move from the left endpoint of w to the right endpoint, and whenever we encounter a vertical wall $w'$ that is incident to w from the bottom, we record the rectangle whose top side lies on w and left side lies on $w'$, and if we encounter a vertical wall $w'$ that is incident to w from the top, we record the rectangle whose bottom side lies on w and right side lies on $w'$. Clearly, we record all rectangles whose bottom or top side lies on w, except the first rectangle below w and the last rectangle above w. On the other hand, if the wall w is vertical, we move from the bottom endpoint of w to the top endpoint, and whenever we encounter a horizontal wall $w'$ that is incident to w from the left, we record the rectangle whose right side lies on w and bottom side lies on $w'$, and if we encounter a horizontal wall $w'$ that is incident to w from the right, we record the rectangle whose left side lies on w and top side lies on $w'$. In this case we record all rectangles whose left or right side lies on w, except the first rectangle to the left of w and the last rectangle to the right of w. Observe that wall slides do not affect the rectangles that appear in a wall shuffle, but only their relative order in the shuffle.

Fig. 20

Illustration of wall shuffles

Fig. 21

Illustration of the surjection $\gamma :S_n\rightarrow {\mathcal R}_n$. This example continues Fig. 19

We are now in position to define the mapping $\gamma :S_n\rightarrow {\mathcal R}_n$, $n\ge 1$, from permutations to generic rectangulations; see Fig. 21. From $\pi \in S_n$ we first construct the diagonal rectangulation $R:=\rho (\pi )\in {\mathcal D}_n$. Let w be a horizontal wall in R, and consider the rectangles in R whose bottom side lies on w from left to right. By construction of $\rho$, these rectangles form an increasing subsequence of $\pi$. Similarly, the rectangles in R whose top side lies on w from left to right form an increasing subsequence of $\pi$. Thus, we can specify a wall shuffle $\sigma (w)$ by taking the subsequence of $\pi$ that contains the appropriate rectangle numbers. On the other hand, for a vertical wall w in R, the rectangles in R whose left side lies on w from bottom to top form a decreasing subsequence of $\pi$, and the rectangles whose right side lies on w form a decreasing subsequence of $\pi$, so we can specify a wall shuffle $\sigma (w)$ by taking the subsequence of $\pi$ containing the appropriate rectangle numbers. The rectangulation $\gamma (\pi )\in {\mathcal R}_n$ is obtained from $R\in {\mathcal D}_n$ by applying wall slides to it, so as to obtain the wall shuffles specified by $\pi$.

Lemma 8.4

[36, Prop. 4.2] The map $\gamma :S_n \rightarrow {\mathcal R}_n$ is surjective.

Even though $\gamma$ is not a bijection, Reading [36] showed that it becomes a bijection when restricting the domain to 2-clumped permutations.

Theorem 8.5

[36, Thm. 4.1] The map $\gamma$ is a bijection when restricted to the set $S_n'$ of 2-clumped permutations.

8.5 The Connection Between Permutations and Rectangulations

The key lemma of this section, Lemma 8.7 below, asserts that deletion, insertion and jumps in permutations as defined in Sect. 8.2 are in bijective correspondence under $\gamma$ to deletion, insertion and jumps in generic rectangulations as defined in Sects. 2.3, 2.4, and 3.1. In order to prove it, we first establish a coarser version of this statement for the mapping $\rho$.

Lemma 8.6

Let $\pi =a_1\cdots a_n\in S_n$, $n\ge 1$, and define $P:=\rho (\pi )\in {\mathcal D}_n$. In the sequence $I(P)=(q_1,\ldots ,q_\nu )$, consider the subsequence $I'(P)=(q_{j_1},\ldots ,q_{j_\mu })$ consisting of the first insertion point of every vertical group, and the last insertion point of every horizontal group. Then we have $p(\rho (c_i(\pi )))=P$ for all $i=1,\ldots ,n+1$, and the sequence $1,\ldots ,n+1$ can be partitioned into consecutive nonempty intervals $I_1,\ldots ,I_\mu$ with the following properties:

• (a) for every $k=1,\ldots ,\mu$ and every $i\in I_k$ we have $\rho (c_i(\pi ))=c_{j_k}(P)$;

• (b) for any interval $I_k=[{\hat{\imath }},{\check{\imath }}]$, $1<k<\mu$, such that the top-left vertex of $r_{n+1}$ in $\rho (c_i(\pi ))$, $i\in I_k$, has type ${\vdash }$, we have that the rectangle $r_{a_{{\hat{\imath }}-1}}$ is the unique rectangle left of $r_{n+1}$, and the rectangle $r_{a_{\check{\imath }}}$ is the leftmost rectangle above $r_{n+1}$;

• (c) for any interval $I_k=[{\hat{\imath }},{\check{\imath }}]$, $1<k<\mu$, such that the top-left vertex of $r_{n+1}$ in $\rho (c_i(\pi ))$, $i\in I_k$, has type ${\top }$, we have that the rectangle $r_{a_{{\hat{\imath }}-1}}$ is the topmost rectangle left of $r_{n+1}$, and the rectangle $r_{a_{\check{\imath }}}$ is the unique rectangle above $r_{n+1}$;

• (d) we have $I_1=\{1\}$ and $I_\mu =\{n+1\}$.

Proof

For the reader’s convenience, the proof is illustrated in Fig. 22. Recall the definition of the mapping $\rho$ via the process described in Sect. 8.4 and illustrated in Fig. 19. Using the definition of the rectangle insertion from Sect. 2.4, we first observe that $\rho (c_1(\pi ))=c_1(P)$ and $\rho (c_{n+1}(\pi ))=c_\nu (P)$. We consider the sequence of permutations $c_i(\pi )$ for $i=1,\ldots ,n+1$ and their images under $\rho$. Observe that $c_{i+1}(\pi )$ is obtained from $c_i(\pi )$ by the adjacent transposition $(n+1)a_i\rightarrow a_i(n+1)$. We consider the construction of $R:=\rho (c_i(\pi ))$, specifically the two steps in which the rectangles $r_{n+1}$ and $r_{a_i}$ are added, and we analyze how the swapped insertion order of these two rectangles changes the resulting rectangulation $R':=\rho (c_{i+1}(\pi ))$. Clearly, in both R and $R'$, the rectangle $r_{n+1}$ is either left-extended and top-fixed, or left-fixed and top-extended, and we treat both cases separately. In particular, these two cases are not symmetric.

Fig. 22

Illustration of the proof of Lemma 8.6

Case (i):  $r_{n+1}$ is left-extended and top-fixed in R.    This case is shown in the top and middle part of Fig. 22. We distinguish two subcases, namely $a_i=n$ and $a_i<n$.

Case (ia):   $a_i=n$.    In this case the top side of $r_{n+1}$ coincides with the bottom side of $r_{a_i}=r_n$ in R, as both rectangles left-extend to the same vertical line by the staircase property and the fact that every rectangle intersects the main diagonal; see Fig. 22a2. Consequently, inserting them in the swapped order, first $r_n$ and then $r_{n+1}$, produces a sub-rectangulation that differs in a simple flip of the wall between these two rectangles; see Fig. 22a3. As the height of the top side of $r_n$ is not determined by $r_{n+1}$, but only by previous rectangles $r_{a_1},\ldots ,r_{a_{i-1}}$, both insertion orders produce the same staircase $r_{a_1}\cup \cdots \cup r_n\cup r_{n+1}$, which by the definition of $\rho$ is all that matters for the next construction steps. Consequently, R and $R'$ differ in a simple flip of the rectangles $r_{n+1}$ and $r_n$. Moreover, by the definition of rectangle deletion given in Sect. 2.3 we have $p(R)=p(R')=P$, and by the definition of rectangle insertion and the definition of $I'(P)$ we have $R=c_{j_k}(P)$ and $R'=c_{j_{k+1}}(P)$ for some index k. Note that in R, the top-left vertex of $r_{n+1}$ has type ${\vdash }$ and $r_{a_{\check{\imath }}}$, ${\check{\imath }}:=i$, is the leftmost rectangle above $r_{n+1}$. Furthermore, in $R'$, the top-left vertex of $r_{n+1}$ has type ${\top }$ and $r_{a_{{\hat{\imath }}-1}}=r_{a_i}$, ${\hat{\imath }}:=i+1$, is the topmost rectangle left of $r_{n+1}$.

Case (ib): $a_i{<}\,n$.   If $r_{a_i}$ is bottom-fixed in R, or if $r_{n+1}$ does not left-extend beyond the vertical line $\ell$ through the right endpoint of the $a_i$th interval on the main diagonal, then the rectangles $r_{n+1}$ and $r_{a_i}$ do not touch; see Fig. 22a1. Consequently, inserting them in the swapped order, first $r_{a_i}$ and then $r_{n+1}$, produces the same sub-rectangulation. It follows that $R=R'$, i.e., $\rho (c_i(\pi ))=\rho (c_{i+1}(\pi ))$, and therefore trivially $p(R)=p(R')$.

On the other hand, if $r_{a_i}$ is bottom-extended and $r_{n+1}$ left-extends beyond $\ell$, then $r_{a_i}$ must be right-fixed because of $a_i<n$ (otherwise $r_{a_i}$ would reach into the $(a_i+1)$st line segment on the main diagonal). Moreover, because of the staircase property the top side of $r_{n+1}$ contains the bottom side of $r_{a_i}$, and both rectangles left-extend to the same vertical line; see Fig. 22b2. Consequently, inserting them in the swapped order, first $r_{a_i}$ and then $r_{n+1}$ produces a sub-rectangulation that differs in a T-flip from ${\bot }$ to ${\vdash }$ around the top-right common vertex of these two rectangles; see Fig. 22b3. As the height of the top side of $r_{a_i}$ is not determined by $r_{n+1}$, and as $r_{a_i}$ is right-fixed and $r_{n+1}$ is top-fixed in both insertion orders, we obtain that R and $R'$ differ in a T-flip around the top-right common vertex of the rectangles $r_{n+1}$ and $r_{a_i}$. Moreover, by the definition of rectangle insertion and deletion and the definition of $I'(P)$, we have $p(R)=p(R')=P$, and $R=c_{j_k}(P)$ and $R'=c_{j_{k+1}}(P)$ for some index k. Note that in R, the top-left vertex of $r_{n+1}$ has type ${\vdash }$ and $r_{a_{\check{\imath }}}$, ${\check{\imath }}:=i$, is the leftmost rectangle above $r_{n+1}$. Furthermore, in $R'$, the top-left vertex of $r_{n+1}$ has type ${\vdash }$ and $r_{a_{{\hat{\imath }}-1}}=r_{a_i}$, ${\hat{\imath }}:=i+1$, is the unique rectangle left of $r_{n+1}$.

Case (ii):   $r_{n+1}$ is left-fixed and top-extended in R.    This case is shown in the bottom part of Fig. 22. As $r_{n+1}$ is top-extended, the staircase property implies that $a_i<n$, as n must be among the first entries $a_1,\ldots ,a_{i-1}$ of $\pi$.

If $r_{a_i}$ is right-fixed in R, or if $r_{n+1}$ does not top-extend beyond the horizontal line $\ell$ through the right endpoint of the $(a_i+1)$st interval on the main diagonal, then the rectangles $r_{n+1}$ and $r_{a_i}$ do not touch; see Fig. 22c1. Consequently, inserting them in the swapped order, first $r_{a_i}$ and then $r_{n+1}$, produces the same sub-rectangulation. It follows that $R=R'$, i.e., $\rho (c_i(\pi ))=\rho (c_{i+1}(\pi ))$, and therefore trivially $p(R)=p(R')$.

On the other hand, if $r_{a_i}$ is right-extended and $r_{n+1}$ top-extends beyond $\ell$, then $r_{a_i}$ must be bottom-fixed because of $a_i<n$ (otherwise $r_{a_i}$ would reach into the $(a_i+1)$st line segment on the main diagonal). Moreover, because of the staircase property the top side of $r_{n+1}$ must lie on the horizontal line through the bottom side of $r_{a_i}$, and therefore $r_{a_i}$ right-extends to the right outer boundary, i.e., the bottom side of $r_{a_i}$ contains the top side of $r_{n+1}$; see Fig. 22c2. Consequently, inserting them in the swapped order, first $r_{a_i}$ and then $r_{n+1}$ produces a sub-rectangulation that differs in a T-flip from ${\top }$ to ${\dashv }$ around the bottom-left common vertex of these two rectangles; see Fig. 22c3. As the height of the top side of $r_{a_i}$ is not determined by $r_{n+1}$ in the two insertion orders, we obtain that R and $R'$ differ in a T-flip around the bottom-left common vertex of the rectangles $r_{n+1}$ and $r_{a_i}$. Moreover, by the definition of rectangle insertion and deletion and the definition of $I'(P)$, we have $p(R)=p(R')=P$, and $R=c_{j_k}(P)$ and $R'=c_{j_{k+1}}(P)$ for some index k. Note that in R, the top-left vertex of $r_{n+1}$ has type ${\top }$ and $r_{a_{\check{\imath }}}$, ${\check{\imath }}:=i$, is the unique rectangle above $r_{n+1}$. Furthermore, in $R'$, the top-left vertex of $r_{n+1}$ has type ${\top }$ and $r_{a_{{\hat{\imath }}-1}}=r_{a_i}$, ${\hat{\imath }}:=i+1$, is the topmost rectangle left of $r_{n+1}$.

This proves (a), (b), and (c). For (d) observe that the rectangle in the bottom-left corner of $\rho (\pi )$ is $r_{a_1}$, whereas the rectangle in the top-right corner is $r_{a_n}$, so $\rho (c_1(\pi ))\ne \rho (c_2(\pi ))$ and $\rho (c_n(\pi ))\ne \rho (c_{n+1}(\pi ))$. This completes the proof of the lemma. $\square$

Lemma 8.7

Let $\pi =a_1\cdots a_n\in S_n$, $n\ge 1$, and consider the rectangulation $P:=\gamma (\pi )\in {\mathcal R}_n$ with $\nu =\nu (P)$ insertion points. Then we have $p(\gamma (c_i(\pi )))=P$ for all $i=1,\ldots ,n+1$, and the sequence $i=1,\ldots ,n+1$ can be partitioned into consecutive nonempty intervals $I_1,\ldots ,I_\nu$, such that for every $k=1,\ldots ,\nu$ and every $i\in I_k$ we have $\gamma (c_i(\pi ))=c_k(P)$. Furthermore, if $\pi$ is 2-clumped, then each interval $I_k$ contains exactly one 2-clumped permutation.

The proof of Lemma 8.7 shows that $I_1=\{1\}$ and $I_\nu =\{n+1\}$.

Proof

For any $i=1,\ldots ,n+1$, consider the permutation $c_i(\pi )$, and the two diagonal rectangulations $P':=\rho (\pi )\in {\mathcal D}_n$ and $R':=\rho (c_i(\pi ))\in {\mathcal D}_n$. From Lemma 8.6 we know that $P'=p(R')$ and $R'=c_j(P')$ for some index j such that the jth insertion point in $I(P')$ is the first of a vertical group, or the last of a horizontal group. By the definitions from Sect. 8.4, $\gamma$ consists of applying $\rho$ plus wall slides that are determined by the wall shuffles of $P'$ and $R'$ and the relative order of the rectangle indices in those shuffles in $\pi$ and $c_i(\pi )$, i.e., $P=\gamma (\pi )$ and $R:=\gamma (c_i(\pi ))$ are obtained from $P'$ and $R'$ by wall slides. Observe that in $R'$ and R, the bottom-right rectangle $r_{n+1}$ is contained in at most one wall shuffle. Specifically, if the top-left corner of $r_{n+1}$ has type ${\vdash }$ and does not lie on the left boundary of the rectangulation, then $r_{n+1}$ participates in a single wall shuffle of the vertical wall that contains the left side of $r_{n+1}$, whereas if the top-left corner of $r_{n+1}$ has type ${\top }$ and does not lie on the upper boundary of the rectangulation, then $r_{n+1}$ participates in a single wall shuffle of the horizontal wall that contains the top side of $r_{n+1}$. In particular, in the cases $j=1$ and $j=\nu (P')$ the rectangle $r_{n+1}$ is not contained in any wall shuffle. As the elements of all subsequences of $\pi$ and $c_i(\pi )$ that do not contain $n+1$ appear in the same relative order in both permutations, all wall shuffles of P and R are the same, except the wall shuffle of R containing $n+1$, which is obtained from a wall shuffle of P by inserting the value $n+1$. We conclude that $p(R)=P$ and $R=c_k(P)$ for some index k.

The desired interval partition $I_1,\ldots ,I_\nu$ of the indices $1,\ldots ,n+1$ is obtained by refining the partition $I_1',\ldots ,I_\mu '$ guaranteed by Lemma 8.6, where $\mu$ is the number of vertical and horizontal groups of $P'$, which is the same for P, as wall slides do not affect it. As $I_1'=\{1\}$ and $I_\mu '=\{n+1\}$ by Lemma 8.6(d), these two intervals are not refined, so we have $I_1:=I_1'=\{1\}$ and $I_\nu :=I_\mu =\{n+1\}$. It remains to consider the remaining intervals $I_k'$, $1<k<\mu$.

First consider an interval $I_k'=:[{\hat{\imath }},{\check{\imath }}]$, $1<k<\mu$, such that in $R':=\rho (c_i(\pi ))\in {\mathcal D}_n$, $i\in I_k$, the top-left vertex has type ${\vdash }$ (recall Lemma 8.6(a)). By Lemma 8.6(b), in $R'$ the rectangle $r_{a_{{\hat{\imath }}-1}}$ is the unique rectangle left of $r_{n+1}$, and the rectangle $r_{a_{\check{\imath }}}$ is the leftmost rectangle above $r_{n+1}$. Consider the wall shuffle $\sigma (w)$ of the vertical wall w between $r_{a_{{\hat{\imath }}-1}}$ and $r_{a_{\check{\imath }}}$ in P. It has the form $\sigma (w)=(b_1,\ldots ,b_\lambda ,a_{{\check{\imath }}},\ldots )$, for some $\lambda \ge 0$, i.e., the rectangles $r_{b_1},\ldots ,r_{b_\lambda }$ are stacked on top of $r_{a_{{\hat{\imath }}-1}}$ and their bottom sides are incident with w below the incidence of the top side of $r_{\check{\imath }}$ with w. It follows that $\pi$ contains the subsequence $a_{{\hat{\imath }}-1},b_1,\ldots ,b_\lambda ,a_{\check{\imath }}$, and so the permutations $c_i(\pi )$ for $i={\hat{\imath }},\ldots ,{\check{\imath }}$ have the value $n+1$ appear at every possible position within the subsequence $b_1,\ldots ,b_\lambda$. Consequently, the interval $I_k'$ is refined into $\lambda$ subintervals such that $\gamma (c_i(\pi ))=\gamma (c_{i+1}(\pi ))$ if $i,i+1$ are in the same subinterval and $\gamma (c_i(\pi ))=c_\ell (P)$ and $\gamma (c_{i+1}(\pi ))=c_{\ell +1}(P)$ for some index $\ell$ if $i,i+1$ are in consecutive subintervals.

Now consider an interval $I_k'=:[{\hat{\imath }},{\check{\imath }}]$, $1<k<\mu$, such that in $R':=\rho (c_i(\pi ))\in {\mathcal D}_n$, $i\in I_k$, the top-left vertex has type ${\top }$ (recall Lemma 8.6(a)). By Lemma 8.6(c), in $R'$ the rectangle $r_{a_{{\hat{\imath }}-1}}$ is the topmost rectangle left of $r_{n+1}$, and the rectangle $r_{a_{\check{\imath }}}$ is the unique rectangle above $r_{n+1}$. Consider the wall shuffle $\sigma (w)$ of the horizontal wall w between $r_{a_{{\hat{\imath }}-1}}$ and $r_{a_{\check{\imath }}}$ in P. It has the form $\sigma (w)=(\ldots ,a_{{\hat{\imath }}-1},b_1,\ldots ,b_\lambda )$, for some $\lambda \ge 0$, i.e., the rectangles $r_{b_\lambda },\ldots ,r_{b_1}$ are stacked to the left of $r_{a_{\check{\imath }}}$ and their right sides are incident with w to the right of the incidence of the left side of $r_{a_{{\hat{\imath }}-1}}$ with w. It follows that $\pi$ contains the subsequence $a_{{\hat{\imath }}-1},b_1,\ldots ,b_\lambda ,a_{\check{\imath }}$, and so the permutations $c_i(\pi )$ for $i={\hat{\imath }},\ldots ,{\check{\imath }}$ have the value $n+1$ appear at every possible position within the subsequence $b_1,\ldots ,b_\lambda$. Consequently, the interval $I_k'$ is refined into $\lambda$ subintervals such that $\gamma (c_i(\pi ))=\gamma (c_{i+1}(\pi ))$ if $i,i+1$ are in the same subinterval and $\gamma (c_i(\pi ))=c_\ell (P)$ and $\gamma (c_{i+1}(\pi ))=c_{\ell +1}(P)$ for some index $\ell$ if $i,i+1$ are in consecutive subintervals.

It remains to argue that each set of permutations $C_k:=\{c_i(\pi )\mid i\in I_k\}$, $k=1,\ldots ,\nu$, contains exactly one 2-clumped permutation. Indeed, $C_k$ can contain at most one 2-clumped permutation by Theorem 8.5, as all $\pi \in C_k$ have the same image under $\gamma$. Suppose for the sake of contradiction that $C_k$ contains no 2-clumped permutation. Then by Theorem 8.5 there is another 2-clumped permutation $\rho \in S_{n+1}$ with $\rho \notin C:=\{c_i(\pi )\mid i=1,\ldots ,n+1\}$ and $\gamma (\rho )=\gamma (c_i(\pi ))$ for all $i\in C_k$. However, by Lemma 8.2, the permutation $\rho ':=p(\rho )\in S_n$ is also 2-clumped, i.e., we have $\rho '\in S_n'$. Moreover, we have $\rho '\ne \pi$ as $\rho \notin C$, and by Lemma 8.7 we have $\gamma (\rho ')=\gamma (\pi )=R$, a contradiction to the fact that $\gamma$ is a bijection between $S_n'$ and ${\mathcal R}_n$ by Theorem 8.5. $\square$

8.6 Proof of Theorem 3.3

With Lemma 8.7 in hand, we are now in position to present the proof of Theorem 3.3.

Proof of Theorem 3.3

Consider a zigzag set of rectangulations ${\mathcal C}_n\subseteq {\mathcal R}_n$, and consider the zigzag sets ${\mathcal C}_{i-1}:=\{p(R)\mid R\in {\mathcal C}_i\}$ for $i=n,n-1,\ldots ,2$. By Lemma 8.7, for all $i=1,\ldots ,n$ there is a set of 2-clumped permutations $L_i\subseteq S_i'$ such that $\gamma$ restricted to $L_i$ is a bijection between $L_i$ and ${\mathcal C}_i$, and such that $L_{i-1}=\{p(\pi )\mid \pi \in L_i\}$ for all $i=2,\ldots ,n$. Moreover, as ${\mathcal C}_i$ is a zigzag set, we know that for all $R\in {\mathcal C}_{i-1}$ we have $c_1(R)\in {\mathcal C}_i$ and $c_{\nu (R)}(R)\in {\mathcal C}_i$, for all $i=2,\ldots ,n$. By Lemma 8.7, this implies that for all $\pi \in L_{i-1}$ we have $c_1(\pi )\in L_i$ and $c_i(R)\in L_i$, for all $i=2,\ldots ,n$, i.e., $L_n$ is a zigzag language of 2-clumped permutations (using that $L_1=\{1\}$ and $L_0:=\{p(\pi )\mid \pi \in L_1\}=\{\varepsilon \}$ are zigzag languages).

By Theorem 8.1, Algorithm J visits every permutation from $L_n$ exactly once, in the order $J(L_n)$ defined by (2). From Lemma 8.3(d) we obtain that if Algorithm J performs a jump of some value j in the current permutation $\pi \in L_n$, then the corresponding rectangulations $R^{[k]}:=\gamma (\pi ^{[k]})$ for $k=j+1,\ldots ,n$ are either bottom-based or right-based. Using the definition of jumps from Sect. 3.1 and Lemma 8.7, a minimal left/right jump of the value j in $\pi \in L_n$, as performed by Algorithm J, corresponds to a minimal left/right jump of the rectangle $r_j$ in $\gamma (\pi )\in \gamma (L_n)={\mathcal C}_n$, as performed by Algorithm J$^{\square }$. This together with the observation that proves the first part of the theorem. Specifically, the ordering of rectangulations generated by Algorithm J$^{\square }$ is

$$\begin{aligned} J^\square ({\mathcal C}_n)=\gamma (J(L_n)). \end{aligned}$$

(3)

To prove the second part of the theorem, by Theorem 8.1 it suffices to show that $|{\mathcal C}_i|=|\gamma (L_i)|$ is even for all $2\le i\le n-1$. For any rectangulation $R\in {\mathcal R}_n$, we write $\lambda (R)\in {\mathcal R}_n$ for the rectangulation obtained by reflection at the main diagonal. First observe that if $R,R'\in {\mathcal R}_n$, $n\ge 2$, satisfy $R'=\lambda (R)$, then we also have $p(R')=\lambda (p(R))$. Consequently, the assumption that ${\mathcal C}_n$ is symmetric implies that ${\mathcal C}_i$ is symmetric for all $i=1,\ldots ,n$. Consider a rectangulation $R\in {\mathcal R}_n$, $n\ge 2$, and observe that if the top-left vertex of the bottom-right rectangle $r_n$ of R has type ${\vdash }$, then it has type ${\top }$ in $\lambda (R)$, and vice versa. It follows that $\lambda$ is an involution without fixed points on ${\mathcal C}_i$ for all $i=2,\ldots ,n$, proving that $|{\mathcal C}_i|$ is even. This completes the proof. $\square$

8.7 Memoryless Generation of Permutations

Consider Algorithm M below, which takes as input a zigzag language of permutations $L_n\subseteq S_n$ and generates them exhaustively by minimal jumps in the same order as Algorithm J, i.e., in the order $J(L_n)$.

Theorem 8.8

For any zigzag language of permutations $L_n\subseteq S_n$, $n\ge 1$, Algorithm M visits every rectangulation from $L_n$ exactly once, in the order $J(L_n)$ defined by (2).

The rest of this section is devoted to proving Theorem 8.8. For any $\pi$ in the sequence $J(L_n)$ we define a sequence $s_n^\pi =(s_{n,1}^\pi ,\ldots ,s_{n,n}^\pi )$ as follows: If $n=1$ we have $J(L_1)=\pi$ with $\pi :=1$ and we define $s_1^\pi =(1)$. If $n\ge 2$, we consider the permutation $\pi ':=p(\pi )\in S_{n-1}$ in the sequence $J(L_{n-1})$, and we define if $\pi '$ appears at an odd position in $J(L_{n-1})$, or if $\pi '$ appears at an even position. If $\pi$ is not the last permutation in $\overline{c}(\pi ')$ we define

for $i=1,\ldots ,n$, and if $\pi$ is the last permutation in $\overline{c}(\pi ')$ we define, for $i=1,\ldots ,n$,

The following lemma captures important properties of the sequences defined in this way.

Lemma 8.9

The sequences defined in (4) and (5) have the following properties:

• (a) For the first permutation $\pi ={{\,\textrm{id}\,}}_n$ in the sequence $J(L_n)$, we have $s_n^\pi =(1,2,\ldots ,n)$.

• (b) For any two consecutive permutations $\pi ,\rho$ in the sequence $J(L_n)$, $\rho$ is obtained from $\pi$ by a jump of the value $s_{n,n}^\pi$.

• (c) For the last permutation $\pi$ in $J(L_n)$ we have $s_{n,n}^\pi =1$.

Moreover, for any three consecutive permutations $\pi ,\rho ,\sigma$ in $J(L_n)$ we have:

• (d) If $\pi$ and $\rho$ differ in a jump of n, and $\rho$ and $\sigma$ differ in a jump of n, then we have $s_{n,i}^\rho =s_{n,i}^\pi$ for $i=1,\ldots ,n-1$.

• (e) If $\pi$ and $\rho$ differ in a jump of n, and $\rho$ and $\sigma$ differ in a jump of $j<n$, then we have $s_{n,i}^\rho =s_{n,i}^\pi$ for $i\in \{1,\ldots ,n-2\}$ and $s_{n,n-1}^\rho =n-1$.

• (f) If $\pi$ and $\rho$ differ in a jump of $j<n$, $\rho$ and $\sigma$ differ in a jump of n, and j is not at a boundary position in $p^{n-j}(\rho )$, then we have $s_{n,i}^\rho =s_{n,i}^\pi$ for $i=1,\ldots ,n-1$.

• (g) If $\pi$ and $\rho$ differ in a jump of $j<n$, $\rho$ and $\sigma$ differ in a jump of n, and j is at a boundary position in $p^{n-j}(\rho )$, then we have $s_{n,i}^\rho =s_{n,i}^\pi$ for $i\in \{1,\ldots ,n-1\}\setminus \{j-1,j\}$, $s_{n,j-1}^\rho =j-1$, and $s_{n,j}^\rho =s_{n,j-1}^\pi$.

The proof of Lemma 8.9 is deferred to Appendix A.

Proof of Theorem 8.8

We establish the following invariants about the permutation $\pi$ visited in line M2 of the algorithm:

• (A) For all $j=2,\ldots ,n$, the direction of the next jump of the value j after the permutation $\pi$ in $J(L_n)$ is left if $o_j\!=\,\vartriangleleft \,$ and right if $o_j\!=\,\vartriangleright \,$.

• (B) The values in the array $s=(s_1,\ldots ,s_n)$ satisfy $s=s_n^\pi$ with $s_n^\pi$ as defined in (4) and (5).

By Lemma 8.3(a), the identity permutation $\pi :={{\,\textrm{id}\,}}_n$ is the first permutation in the sequence $J(L_n)$. Moreover, by the initialization of $\pi$ in line M1, $\pi ={{\,\textrm{id}\,}}_n$ is also the first permutation visited in line M2. Combining this with the above invariants, we obtain by induction on the length of $J(L_n)$ that after visiting a permutation $\pi \in L_n$, the next permutation visited by Algorithm M is the permutation that succeeds $\pi$ in $J(L_n)$. Indeed, by the instructions in line M3 and M4, the next permutation $\rho$ visited by the algorithm is obtained from $\pi$ by a jump of the value $j:=s_n$ that is minimal w.r.t. $L_n$, and the jump direction is left if $o_j\!=\,\vartriangleleft \,$ and right if $o_j\!=\,\vartriangleright \,$. Applying (B) and Lemma 8.9(a), and (A) and Lemma 8.3(c), we obtain that $\rho$ is indeed the permutation that succeeds $\pi$ in $J(L_n)$. Also, the algorithm terminates correctly after visiting the last permutation in the sequence $J(L_n)$ by the condition in line M3 and Lemma 8.9(c).

We prove (A) + (B) by double induction on n and the number of iterations of Algorithm M. The induction basis $n=1$ is trivial. For the induction step, let $n\ge 2$, and assume that the invariants hold for the zigzag language $L_{n-1}=\{p(\pi )\mid \pi \in L_n\}$. We first verify that (A) + (B) hold in line M2 during the first iteration of the algorithm when $\pi ={{\,\textrm{id}\,}}_n$. By line M1 we have $o_j=\,\vartriangleleft \,$ for $j=2,\ldots ,n$, so (A) is satisfied by Lemma 8.3(b). By line M1 we also have $s=(s_1,\ldots ,s_n)=(1,\ldots ,n)$, which equals $s_n^\pi =s_n^{{{\,\textrm{id}\,}}_n}$ by Lemma 8.9(a).

For the induction step, consider three consecutive permutations ${\hat{\pi }},{\hat{\rho }},{\hat{\sigma }}$ in the sequence $J(L_n)$, and suppose that (A) + (B) are satisfied when Algorithm M visits $\pi ={\hat{\pi }}$. We need to verify that (A) + (B) still hold after one iteration through lines M2—M5, after which the algorithm visits $\pi ={\hat{\rho }}$ by the instructions in lines M3 and M4, as argued before.

Case (i):   We first consider the case that ${\hat{\pi }},{\hat{\rho }}$ satisfy $p({\hat{\pi }})=p({\hat{\rho }})=:{\hat{\pi }}'\in L_{n-1}$ and therefore both are contained in the subsequence $\overline{c}({\hat{\pi }}')$ of $J(L_n)$. We only treat the case , as the other case is symmetric. In this case ${\hat{\rho }}$ is obtained from ${\hat{\pi }}$ by a left jump of the value n. In particular, the variable j has the value $j=s_n=n$ and $o_n\!=\,\vartriangleleft \,$ in this iteration of the algorithm.

Case (ia):   The value n is not at the first position in ${\hat{\rho }}$. Then by (2) the permutation ${\hat{\sigma }}$ is obtained from ${\hat{\rho }}$ by another left jump of the value n. In line M5, the value of $s_n$ is set to n, which was the previous value, but none of the conditions in line M5 holds for $\pi ={\hat{\rho }}$, so overall none of the arrays s and o is modified. We conclude that (A) holds after this iteration for $\pi ={\hat{\rho }}$. Moreover, (B) holds by Lemma 8.9(c) and (4b).

Case (ib):  The value n is at the first position in ${\hat{\rho }}$, i.e., we have ${\hat{\rho }}=c_1({\hat{\pi }}')=n{\hat{\pi }}'$. Then by (2) we have ${\hat{\sigma }}=c_1({\hat{\rho }}')=n\,{\hat{\rho }}'$ where ${\hat{\rho }}'\in L_{n-1}$ succeeds ${\hat{\pi }}'$ in the sequence $J(L_{n-1})$. In line M5, the value of $s_n$ is set to n, which is the same as the previous value, but since the first if-condition is satisfied, $s_n$ is then overwritten by $s_{n-1}$, and $s_{n-1}$ is set to $n-1$. Consequently, the new values are $s_{n-1}=n-1$ and $s_n=s_{n,n-1}^{\hat{\pi }}=s_{n-1,n-1}^{{\hat{\pi }}'}$ by induction and (B) and (4a). Moreover, the value of $o_n$ is flipped to $o_n\!=\,\vartriangleright \,$. Using Lemma 8.3(e), we conclude that (A) holds after this iteration for $\pi ={\hat{\rho }}$. Applying Lemma 8.9(d) and using that $s_{n,n}^\pi =s_{n-1,n-1}^{{\hat{\pi }}'}$ by (5c), we obtain that (B) holds as well.

Case (ii):  It remains to consider the case that $p({\hat{\pi }})\ne p({\hat{\rho }})$, i.e., both permutations have n at the first or last position. By symmetry, it suffices to consider the case that n is at the first position, i.e., ${\hat{\pi }}=c_1({\hat{\pi }}')=n\,{\hat{\pi }}'$ and ${\hat{\rho }}=c_1({\hat{\rho }}')=n{\hat{\rho }}'$, where ${\hat{\pi }}'$ and ${\hat{\rho }}'$ are consecutive permutations in $J(L_{n-1})$. They differ in a jump of the value $j:=s_n<n$ by Lemma 8.9(b) and (B). Then by (2), the permutation ${\hat{\sigma }}$ is obtained from ${\hat{\rho }}$ by a right jump of n. We proceed to show that (A) and (B) hold for $\pi ={\hat{\rho }}$, and for this we distinguish subcases.

Case (iia):  The value j is not at a boundary position in $p^{n-j}({\hat{\rho }})$. Then by Lemma 8.3(d) in ${\hat{\rho }}$ the value j is surrounded by the same values as in $p^{n-j}({\hat{\rho }})$, both smaller than j. In line M5, the value of $s_n$ is set to n, and no other entries of s and o are modified. Using Lemma 8.3(e), we conclude that (A) holds after this iteration for $\pi ={\hat{\rho }}$. Moreover, (B) holds by Lemma 8.9(e), also using that $s_{n,n}^{\hat{\rho }}=n$ by (4b).

Case (iib):  The value j is at a boundary position in $p^{n-j}({\hat{\rho }})$, and the value k next to it is smaller than j. Then by Lemma 8.3(d) in ${\hat{\rho }}$ the value k is also next to j, and either j is at a boundary position (the right boundary, as n is at the first position in ${\hat{\rho }}$) or the other value next to in the direction $o_j$ of the jump is bigger than j. In line M5, the value of $s_n$ is set to n, the value of $s_j$ is set to $s_{j-1}$, and $s_{j-1}$ is set to $j-1$. Moreover, the value of $o_j$ is flipped. Using Lemma 8.3(e), we conclude that (A) holds after this iteration for $\pi ={\hat{\rho }}$. Moreover, (B) holds by Lemma 8.9(f), also using that $s_{n,n}^{\hat{\rho }}=n$ by (4b). This completes the proof of the theorem. $\square$

8.8 Proof of Theorem 5.1

Proof of Theorem 5.1

In the proof of Theorem 3.3 we showed that the ordering of rectangulations generated by Algorithm J$^{\square }$ is given by (3) for some zigzag language $L_n$ of 2-clumped permutations. The theorem hence follows by applying Theorem 8.8.

$\square$

9 Counting Pattern-Avoiding Rectangulations

In this section we report on computer experiments that count pattern-avoiding rectangulations ${\mathcal C}_n({\mathcal P})$ for all interesting subsets of patterns ${\mathcal P}\subseteq \{P_1,\ldots ,P_8\}$ where , , , , , , , . Clearly, we can omit sets of patterns that are equivalent to another set of patterns under $D_4$ actions (rotations and mirroring vertically or horizontally). Table 3 shows the results for generic rectangulations ${\mathcal C}_n={\mathcal R}_n$ as a base class. For counting results with block-aligned rectangulations as a base class, see Appendix C. The set of patterns used in each row of the table is denoted by the pattern indices, omitting curly brackets and commas. For example, the row 1478 refers to the set ${\mathcal P}=\{P_1,P_4,P_7,P_8\}$.

Table 3 Counts for pattern-avoiding rectangulations with generic rectangulations as a base class

Several of these counting sequences appear in the OEIS [46], and are related to pattern-avoiding permutations (see e.g. [8]). The matching OEIS entries marked with ? are observed through numerical experiments, but no formal bijective proof has been obtained yet, even though finding one should be straightforward in some cases. The last two rows in Table 3 with ? are interesting, as the correspondence to the objects mentioned in those OEIS entries is not obvious. This is true in particular for OEIS sequence A000984, which are the central binomial coefficients $\left( {\begin{array}{c}2n\\ n\end{array}}\right)$.

10 Open Questions

The subject of pattern-avoiding rectangulations deserves further systematic investigation, and may still hold many undiscovered gems; recall Table 3. Understanding the number of pattern-avoiding rectangulations that are obtained by rectangle insertion may also help to improve the runtime bounds for our generation algorithms (recall Remark 7.11). Moreover, does the avoidance of a rectangulation pattern always correspond to the avoidance of a particular permutation pattern, and what is this correspondence?

In our paper we considered R-equivalence and S-equivalence of generic rectangulations ${\mathcal R}_n$, and these equivalence relations are induced by wall slides, or by wall slides and simple flips, respectively. Considering all three basic flip operations $F=\{W,S,T\}$, namely wall slides, simple flips, and T-flips, there are $2^3=8$ possible subsets of F to induce an equivalence relation on ${\mathcal R}_n$. Which of these equivalence relations are interesting (apart from $\emptyset$, $\{W\}$, and $\{W,S\}$ considered here), and what are suitable representatives that can be generated efficiently?

Another interesting question to investigate would be Gray codes for rectangulations of point sets as introduced by Ackerman et al. [2]. Some first results in this direction have been obtained by Yamanaka et al. [42]. In particular, can we apply our permutation-based generation framework for this task?

Appendices

Appendix A: Proofs of Lemmas 8.3 and 8.9

Proof of Lemma 8.3

Properties (a) and (b) follow easily from the definition (e). Property (c) follows from Theorem 8.1 and line J2 of Algorithm J. We prove (d) and (e) by induction on n. Both statements are trivial for $n=0$ and $n=1$, which settles the induction basis. We now assume that $n\ge 2$, and suppose that $J(L_{n-1})=:\pi _1,\pi _2,\ldots$ satisfies (a) and (b) for jumps of all values $j\in \{2,\ldots ,n-1\}$.

We start with the induction step for (d). If $\pi ,\rho$ in $J(L_n)$ differ in a jump of the value $j=n$, then (d) is satisfied trivially, so it suffices to consider jumps of values $j\in \{2,\ldots ,n-1\}$ in $J(L_n)$. However, by (2), such jumps only occur at the transitions between and  or between  and  for some k. In the first case, $\pi =\pi ^{[n]}:=n \pi _k=c_1(\pi _k)=c_1(\pi ^{[n-1]})$ is followed by $\rho =\rho ^{[n]}:=n\pi _{k+1}=c_1(\pi _{k+1})=c_1(\rho ^{[n-1]})$, and in the second case $\pi =\pi ^{[n]}:=\pi _k n=c_n(\pi _k)=c_n(\pi ^{[n-1]})$ is followed by $\rho =\rho ^{[n]}:=\pi _{k+1}n=c_n(\pi _{k+1})=c_n(\rho ^{[n-1]})$, completing the proof.

We proceed with the induction step for (e). If $\pi ,\rho$ in $J(L_n)$ differ in a jump of the value $j=n$, then (e) follows from the definition (e) and of the sequences  and . On the other hand, consider $\pi ,\rho$ in $J(L_{n-1})$ that differ in a jump of some value $j\in \{2,\ldots ,n-1\}$, and let $\pi ',\rho '$ be the next two permutations in $J(L_{n-1})$ that differ in a jump of j, satisfying (e) by induction. Then in $J(L_n)$, after the jump of j from $c_1(\pi )$ to $c_1(\rho )$ or from $c_n(\pi )$ to $c_n(\rho )$, the next jump of the value j is from $c_1(\pi ')$ to $c_1(\rho ')$ or from $c_n(\pi ')$ to $c_n(\rho ')$. If j is not at the first position in $\rho$ and the value left of it is smaller than j, then j is not at the first position and the value left of it is smaller in both $c_1(\rho )$ and $c_n(\rho )$, and $c_1(\rho ')$ and $c_n(\rho ')$ are obtained from $c_1(\pi ')$ and $c_n(\pi ')$, respectively, by a left jump. On the other hand, if j is at the first position in $\rho$ or the value left of it is bigger than j, then j is at the first position or the value left of it is bigger than j in both $c_1(\rho )$ and $c_n(\rho )$, and $c_1(\rho ')$ and $c_n(\rho ')$ are obtained from $c_1(\pi ')$ and $c_n(\pi ')$, respectively, by a right jump. This completes the proof. $\square$

Proof of Lemma 8.9

We prove these properties by induction on n. The induction basis $n=1$ is trivial. For the induction step let $n\ge 2$ and assume that all properties hold for the sequence $J(L_{n-1})$. We first show the induction step for (a), (b), and (c). Consider a permutation $\pi$ in $J(L_n)$ and define $\pi ':=p(\pi )\in S_{n-1}$.

To prove (a), let $\pi ={{\,\textrm{id}\,}}_n$ be the first permutation in $J(L_n)$ (recall Lemma 8.3(a)). We have $\pi '={{\,\textrm{id}\,}}_{n-1}$ and by induction and (a) we hence have $s_n^{\pi '}=(1,2,\ldots ,n-1)$. Using (4) we obtain $s_n^\pi =(1,2,\ldots ,n)$, as claimed.

To prove (b), let $\pi$ be a permutation that is not the last in the sequence $J(L_n)$, and let $\rho$ be the permutation succeeding $\pi$ in $J(L_n)$. If $\pi$ is not the last permutation in the subsequence $\overline{c}(\pi ')$ of $J(L_n)$, then by (2) the permutation $\rho$ is obtained from $\pi$ by a jump of the value n, and then (b) follows directly from (4b). On the other hand, if $\pi$ is the last permutation in the subsequence $\overline{c}(\pi ')$, then $\rho$ is obtained from $\pi$ by a jump of the value $s_{n-1,n-1}^{\pi '}$ by induction and (b), and by (5c) we have $s_{n,n}^\pi =s_{n-1,n-1}^{\pi '}$, as desired.

To prove (c), let $\pi$ be the last permutation in $J(L_n)$. Then the permutation $\pi '$ is also the last permutation in $J(L_{n-1})$, so by induction we have $s_{n-1,n-1}^{\pi '}=1$. Using (5c) we see that $s_{n,n}^\pi =s_{n-1,n-1}^{\pi '}=1$, as desired.

To prove (d), note that $p(\pi )=p(\rho )=p(\sigma )$ and therefore $\rho$ is not the last permutation in the subsequence $\overline{c}(p(\rho ))$, so the claim follows directly from (4a).

To prove (e), note that $p(\pi )=p(\rho )\ne p(\sigma )$ and therefore $\rho$ is the last permutation in the subsequence $\overline{c}(p(\rho ))$, so the claim follows directly from (4a), (5a), and (5b).

To prove (f) and (g), let $\pi ':=p(\pi )$, $\rho ':=p(\rho )$ and $\sigma ':=p(\sigma )$. Note that $\pi '\ne \rho '=\sigma '$ and therefore $\pi$ is the last permutation in the subsequence $\overline{c}(\pi ')$, whereas $\rho$ is the first permutation in the subsequence $\overline{c}(\rho ')$. Consequently, $s_n^\pi$ is defined by (5) and $s_n^\rho$ is defined by (4). In particular, we have $s_{n,n-1}^\pi =n-1$ by (5b) and $s_{n,n-1}^\rho =s_{n-1,n-1}^{\rho '}$ by (4a).

We first prove (f), and we distinguish whether $j=n-1$ or $j<n-1$. If $j=n-1$, then $\pi '$ and $\rho '$ differ in a jump of $j=n-1$, and as j is not at a boundary position in $\rho '$, $\rho '$ and $\sigma '$ also differ in a jump of $n-1$ by (2). Consequently, we have $s_{n-1,i}^{\rho '}=s_{n-1,i}^{\pi '}$ for $i=1,\ldots ,n-2$ by induction and (d), and $s_{n-1,n-1}^{\rho '}=n-1$ by (4b), so (f) indeed holds in this case.

If $j<n-1$, then $\pi '$ and $\rho '$ differ in a jump of $j<n-1$, and $\rho '$ and $\sigma '$ differ in a jump of $n-1$ by (2). As j is not at a boundary position in $p^{n-j}(\rho )=p^{n-1-j}(\rho ')$, we have $s_{n-1,i}^{\rho '}=s_{n-1,i}^{\pi '}$ for $i=1,\ldots ,n-2$ by induction and (f), and $s_{n-1,n-1}^{\rho '}=n-1$ by induction and (b), so (f) indeed holds in this case.

We now prove (g), and again we distinguish whether $j=n-1$ or $j<n-1$. If $j=n-1$, then $\pi '$ and $\rho '$ differ in a jump of $j=n-1$, in particular $p(\pi ')=p(\rho ')$, and as $j=n-1$ is at a boundary position in $p^{n-j}(\rho )=p(\rho )=\rho '$, $\rho '$ and $\sigma '$ differ in a jump of some value smaller than $n-1$ by (2). Consequently, we have $s_{n-1,i}^{\rho '}=s_{n-1,i}^{\pi '}$ for $i=1,\ldots ,n-3$ and $s_{n-1,n-2}^{\rho '}=n-2$ by induction and (e). Moreover, we have $s_{n-1,n-1}^{\rho '}=s_{n-2,n-2}^{p(\rho ')}=s_{n-2,n-2}^{p(\pi ')}=s_{n-1,n-2}^{\pi '}$ by (4a). Combining these observations shows that indeed (g) holds in this case.

If $j<n-1$, then $\pi '$ and $\rho '$ differ in a jump of $j<n-1$, and $\rho '$ and $\sigma '$ differ in a jump of $n-1$ by (2). Clearly, j is at a boundary position in $p^{n-j}(\rho )=p^{n-1-j}(\rho ')$, and by induction and (g) we have $s_{n-1,i}^{\rho '}=s_{n-1,i}^{\pi '}$ for $i\in \{1,\ldots ,n-2\}\setminus \{j-1,j\}$, $s_{n-1,j-1}^{\rho '}=j-1$ and $s_{n-1,j}^{\rho '}=s_{n-1,j-1}^{\pi '}$. Moreover, we have $s_{n-1,n-1}^{\rho '}=n-1$ by (4a). Combining these observations proves (g) in this last case. $\square$

Appendix B: S-Equivalence of Rectangulations

Recall that R-equivalence is the equivalence relation on ${\mathcal R}_n$ obtained from wall slides, i.e., any two generic rectangulations that differ in a sequence of wall slides are equivalent. It is well known that every equivalence class contains exactly one diagonal rectangulation, i.e., ${\mathcal D}_n$ is a set of representatives for R-equivalence (see e.g. [9]).

Fig. 23

Equivalence classes of generic rectangulations under S-equivalence for $n=2,3,4$. Block-aligned rectangulations as equivalence class representatives are highlighted

We aim to do something analogous for S-equivalence, and to pick a suitable set of representatives for our generation algorithm. Recall that S-equivalence is the equivalence relation on ${\mathcal R}_n$ obtained from wall slides and simple flips, i.e., any two generic rectangulations that differ in a sequence of wall slides or simple flips are equivalent. Figure 23 shows the equivalence classes of generic rectangulations under S-equivalence for $n=2,3,4$. Unfortunately, one can check that there is no choice of representatives ${\mathcal P}_2\subseteq {\mathcal R}_2$, ${\mathcal P}_3\subseteq {\mathcal R}_3$, ${\mathcal P}_4\subseteq {\mathcal R}_4$ for those equivalence classes (i.e., $|{\mathcal P}_2|=1$, $|{\mathcal P}_3|=2$, $|{\mathcal P}_3|=6$) that is consistent with the operations of rectangle deletion and insertion, i.e., such that ${\mathcal P}_2=\{p(R)\mid R\in {\mathcal P}_3\}$ and ${\mathcal P}_3=\{p(R)\mid R\in {\mathcal P}_4\}$. Consequently, for S-equivalence there is no set of unique representatives, one for each equivalence class, that would form a zigzag set, so our generation algorithms cannot be applied directly. However, we will show how to choose representatives for each equivalence class (highlighted in Fig. 23), such that those representatives and the rectangulations obtained from them by a simple flip of the rectangle $r_n$ admit a generation tree approach with our algorithms.

1.1 B.1  Representatives for S-Equivalence

By definition, S-equivalence is a coarsening of R-equivalence, and we will therefore choose a subset of diagonal rectangulations as representatives. We start with some definitions; see Fig. 24. A rectangulation is horizontally aligned, or H-aligned for short, if all of its walls are horizontal. Moreover, a rectangulation is almost horizontally aligned, or AH-aligned for short, if all of its walls except one at the bottom are horizontal. Equivalently, it is obtained by gluing copies of $\square$ on top of .

Similarly, a rectangulation is vertically aligned, or V-aligned for short, if all of its walls are vertical. Moreover, a rectangulation is almost vertically aligned, or AV-aligned for short, if all of its walls except one at the right are vertical. Equivalently, it is obtained by gluing copies of $\square$ on the left of $\boxminus$.

Fig. 24

Illustration of aligned rectangulations. Rectangulation a is V-aligned, b is AV-aligned, c is V-alignable but neither V-aligned nor AV-aligned, d is V-aligned and AH-aligned

The rectangulation $\square$ is H-aligned and V-aligned. The rectangulation $\boxminus$ is H-aligned and AV-aligned, and the rectangulation  is V-aligned and AH-aligned. A rectangulation is H- or V-alignable, if we can apply a sequence of simple flips to make it H- or V-aligned, respectively. Clearly, a rectangulation is H-alignable if it is obtained by vertically gluing together copies of $\square$ and , and it is V-alignable if it is obtained by horizontally gluing together copies of $\square$ and $\boxminus$. A block in a rectangulation is a subset of rectangles whose union is a rectangle. The size of a block is the number of rectangles of the block.

Lemma B.1

Every diagonal rectangulation can be partitioned uniquely into maximal alignable blocks.

Proof

Suppose for the sake of contradiction that for some rectangulation $R\in {\mathcal D}_n$ there were two distinct block partitions $P,P'$ of R. Consider a block B in P that is not a block in $P'$. Consider one of the rectangles in B, and consider the block $B'$ of $P'$ containing this rectangle. As R does not have any points where four rectangles meet and $B'\ne B$, the block $B'$ must be a proper subset or superset of B, contradicting the maximal choice of the blocks. $\square$

Lemma B.1 holds more generally for generic rectangulations and for maximal blocks with any additional property (such as alignable), but this is not needed here.

Lemma B.2

For any diagonal rectangulation, the partition into maximal alignable blocks is invariant under simple flips.

Proof

Consider a wall that can be simple-flipped, and observe that the two rectangles to both sides of the wall must belong to the same alignable block due to the maximal choice of the blocks. $\square$

From now on, whenever we refer to a block in a rectangulation, we mean a maximal alignable block. A block is a base block, if it contains the bottom boundary of the rectangulation.

Based on the partition of a diagonal rectangulation $R\in {\mathcal D}_n$ into blocks, which is unique by Lemma B.1, we introduce the following definitions; see Fig. 25. We refer to each block of R as an H-block or V-block, if it is H-alignable or V-alignable, respectively. We consider an H-block B of size at least 2 with rectangle $r_i$ at the bottom-right. If $i=n$ or if rectangle $r_{i+1}$ of R is below $r_i$ we say that B is free, whereas if $r_{i+1}$ is right of $r_i$ we say that B is locked. Similarly, we consider a V-block B of size at least 2 with rectangle $r_i$ at the bottom-right. If $i=n$ or if rectangle $r_{i+1}$ of R is right of $r_i$ we say that B is free, whereas if $r_{i+1}$ is below $r_i$ we say that B is locked.

Fig. 25

Illustration of blocks and block-aligned rectangulations. Blocks are highlighted by the same shading. H-blocks are $B_1$, $B_2$, $B_3$, $B_4$, $B_6$, and $B_7$, with $B_7$ free and $B_3$, $B_4$, and $B_6$ locked. V-blocks are $B_1$, $B_2$, $B_4$, $B_5$, $B_6$, and $B_7$, with $B_4$, $B_6$, and $B_7$ free and $B_5$ locked. Rectangulation a is not block-aligned, whereas b is block-aligned

We say that $R\in {\mathcal D}_n$ is block-aligned if for every block B of size at least 2 in R the following conditions hold: if B is a free H-block then B is H-aligned, if B is a free V-block then B is V-aligned, if B is a locked H-block then B is AH-aligned, and if B is a locked V-block then B is AV-aligned. A special rule applies if the rectangle $r_n$ is contained in a block of size 2 (which is free and both H-alignable and V-alignable), and then we require this block to be V-aligned, unless it is a base block, in which case it must be H-aligned. Note that a block B of size exactly 2 that does not contain $r_n$ is both an H-block and a V-block, however, if B is a locked/free H-block then B is a free/locked V-block, respectively, so this definition is consistent (as AH-aligned equals V-aligned and H-aligned equals AV-aligned for a block of size 2).

We write ${\mathcal B}_n\subseteq {\mathcal D}_n$ for the set of diagonal rectangulations that are block-aligned. We partition this set into ${\mathcal B}_n^\square$ and ${\mathcal B}_n^\boxplus$, respectively, according to whether the rectangle $r_n$ is contained in a block of size 1 or at least 2, respectively. Note that if $R\in {\mathcal B}_n^\square$, then the wall between $r_n$ and $r_{n-1}$ does not admit a simple flip, whereas if $R\in {\mathcal B}_n^\boxplus$, then this wall admits a simple flip. For any $R\in {\mathcal B}_n^\boxplus$ we write $s(R)\in {\mathcal B}_n^\boxplus$ for the rectangulation obtained from $r_n$ by a simple flip of this wall. The set ${\mathcal B}_n^\boxplus$ is partitioned into ${\mathcal B}_n^\boxminus$ and  according to whether this wall is horizontal or vertical, respectively.

As a consequence of Lemma B.2, every equivalence class of generic rectangulations under S-equivalence contains exactly one block-aligned diagonal rectangulation; see Fig. 23. Consequently, we will use the block-aligned rectangulations ${\mathcal B}_n\subseteq {\mathcal D}_n$ as representatives for S-equivalence.

1.2 B.2  Insertion in Block-Aligned Rectangulations

The next two lemmas describe how to construct block-aligned rectangulations by rectangle insertion; see Fig. 27. For any diagonal rectangulation $R\in {\mathcal D}_n$, we let $I_v(R)$ denote the subsequence of I(R) of the first insertion point of each vertical group. Similarly, we let $I_h(R)$ denote the subsequence of I(R) of the last insertion point of each horizontal group.

Lemma B.3

Let $P\in {\mathcal B}_{n-1}^\square$, $I_v(P)=:(q_{i_1},\ldots ,q_{i_\lambda })$, and $I_h(P)=:(q_{j_1},\ldots ,q_{j_\mu })$. Then we have the following:

• For any $1\le k<\lambda$ we have $c_{i_k}(P)\in {\mathcal B}_n^\square$, and every $R\in {\mathcal B}_n^\square$ for which the top-left vertex of $r_n$ has type ${\vdash }$ and $r_{n-1}$ forms its own block is obtained by insertion from some $P\in {\mathcal B}_{n-1}^\square$ in this way.

• For any $1<k\le \mu$ we have $c_{j_k}(P)\in {\mathcal B}_n^\square$, and every $R\in {\mathcal B}_n^\square$ for which the top-left vertex of $r_n$ has type ${\top }$ and $r_{n-1}$ forms its own block is obtained by insertion from some $P\in {\mathcal B}_{n-1}^\square$ in this way.

• If $\lambda >1$ we have , and every for which $r_{n-1}$ and $r_n$ form a V-aligned block of size 2 is obtained by insertion from some $P\in {\mathcal B}_{n-1}^\square$ in this way.

• If $\lambda =1$ we have $c_{i_1}(P)\in {\mathcal B}_n^\boxminus$, and every $R\in {\mathcal B}_n^\boxminus$ for which $r_{n-1}$ and $r_n$ form an H-aligned base block of size 2 is obtained by insertion from some $P\in {\mathcal B}_{n-1}^\square$ in this way.

Proof

The first and second part of the lemma are symmetric, so it suffices to prove the first one. For this we analyze how the blocks of $R:=c_{i_k}(P)$ differ from the blocks of P, and prove that they are all aligned as required.

The rectangle $r_{n-1}$ forms a block of size 1 in P, and as $k<\lambda$ this is also true in R. Similarly, as $k<\lambda$ the rectangle $r_n$ forms a block of size 1 in R. Consequently, we only need to verify whether blocks of R not containing $r_{n-1}$ or $r_n$ in P are aligned as required. If $k=1$, then the blocks of R are the same as those of P, plus the block containing $r_n$, so we are done; see Fig. 26a. If $k>1$, we let $r_a$ and $r_b$ be the rectangles in P to the left and right of the edge that contains the insertion point $q_{i_k}$. If $r_a$ and $r_b$ belong to two distinct blocks in P, then the blocks of R are the same as those of P, plus the block containing $r_n$, so we are done; see Fig. 26b. On the other hand, if $r_a$ and $r_b$ belong to the same block B in P, then it must be a free V-block that is V-aligned or a locked H-block that AH-aligned, and we have $b=a+1$. If B is a free V-block in P, then in R this block is split into two free V-blocks $B'$ and $B''$, one containing $r_a$ and the other one containing $r_b=r_{a+1}$, and both $B'$ and $B''$ are V-aligned; see Fig. 26c. If B is a locked H-block in P, then in R this block is split into the H-block $B\setminus \{r_a,r_{a+1}\}$, and two blocks of size 1 containing $r_a$ or $r_{a+1}$, respectively; see Fig. 26d. Moreover, if $|B|=3$ then $|B\setminus \{r_a,r_{a+1}\}|=1$, and otherwise $B\setminus \{r_a,r_{a+1}\}$ is a free H-block that is H-aligned in R. In all cases we obtain $R\in {\mathcal B}_n^\square$, as claimed.

We continue to prove the third part of the lemma about the rectangulation $R:=c_{j_1}(P)$. The rectangle $r_{n-1}$ forms a block of size 1 in P, and together with $r_n$ it forms a block of size 2 in R; see Fig. 26e. This block is V-aligned in R, so we have , as claimed.

It remains to prove the fourth part of the lemma about the rectangulation $R:=c_{i_1}(P)$.The rectangle $r_{n-1}$ forms a block of size 1 in P, and together with $r_n$ it forms a block of size 2 in R; see Fig. 26f. This block is a base block and H-aligned in R, so we have $R\in {\mathcal B}_n^\boxminus$, as claimed. $\square$

Fig. 26

Illustration of the proofs of Lemmas B.3 and B.4

Lemma B.4

Let $P\in {\mathcal B}_{n-1}^\boxminus$ and $P':=s(P)$, or let and $P:=s(P')$, and define $I_v(P)=:(q_{i_1},\ldots ,q_{i_\lambda })$ and $I_h(P')=:(q_{j_1},\ldots ,q_{j_\mu })$. Then we have the following:

• For any $1\le k<\lambda$ we have $c_{i_k}(P)\in {\mathcal B}_n^\square$, and every $R\in {\mathcal B}_n^\square$ for which the top-left vertex of $r_n$ has type ${\vdash }$ and $r_{n-1}$ is contained in a block of size at least 2 is obtained by insertion from some $P\in {\mathcal B}_{n-1}^\boxminus$ in this way.

• For any $1<k\le \mu$ we have $c_{j_k}(P')\in {\mathcal B}_n^\square$, and every $R\in {\mathcal B}_n^\square$ for which the top-left vertex of $r_n$ has type ${\top }$ and $r_{n-1}$ is contained in a block of size at least 2 is obtained by insertion from some in this way.

• We have $c_{i_\lambda }(P)\in {\mathcal B}_n^\boxminus$, and every $R\in {\mathcal B}_n^\boxminus$ for which $r_{n-1}$ and $r_n$ are contained in an H-aligned block of size at least 3 is obtained by insertion from some $P\in {\mathcal B}_{n-1}^\boxminus$ in this way.

• We have , and every for which $r_{n-1}$ and $r_n$ are contained in a V-aligned block of size at least 3 is obtained by insertion from some in this way.

Proof

The proof for the first part in the case $P\in {\mathcal B}_{n-1}^\boxminus$ and for the second part in the case is analogous to the proof of Lemma B.3. Therefore, by symmetry, to complete the proof of the first two parts, it suffices to argue about the case , $P:=s(P')$ and the rectangulation $R:=c_{i_k}(P)$ for $1\le k<\lambda$; see Fig. 26g. The V-block B in $P'$ containing $r_{n-1}$, which is free and V-aligned in $P'$, is AV-aligned and free in P. Consequently, in R the block B is either split into two blocks, a free V-block $B'$ that is V-aligned to the left of a locked V-block $B''$ (Fig. 26g1) that is AV-aligned, or B remains a single locked V-block that is AV-aligned in R (Fig. 26g2), where the locking is due to the insertion of $r_n$. The remaining blocks of $P'$ are treated as in the proof of Lemma B.3. In all cases we obtain that $R\in {\mathcal B}_n^\square$, as claimed.

The third and fourth part of the lemma are symmetric, so it suffices to prove the third one about the rectangulation $R:=c_{i_\lambda }(P)$. In this case $\{r_{n-2},r_{n-1},r_n\}$ is an H-block that is free and H-aligned in R, and either $|B\setminus \{r_{n-2},r_{n-1}\}|=1$ or $B\setminus \{r_{n-2},r_{n-1}\}$ is a V-block that is free and V-aligned in R; see Fig. 26h. It follows that $R\in {\mathcal B}_n^\boxminus$, as claimed. $\square$

Fig. 27

Tree of block-aligned rectangulations. Rectangulations from ${\mathcal B}_n^\square$ are drawn gray, and those from ${\mathcal B}_n^\boxplus$ are drawn white. For any $R\in {\mathcal B}_n^\boxplus$, the wall between rectangles $r_n$ and $r_{n-1}$ is drawn dashed, whereas the corresponding wall in s(R) is drawn dotted. Each insertion point marked by a disk corresponds to one child of the current node as in the first two parts of Lemmas B.3 and B.4. Each insertion point marked by an empty square corresponds to one child as in the third or fourth part of Lemma B.3, whereas crossed insertion points are not used. Each insertion point marked by a solid square corresponds to one child as in the third or fourth part of Lemma B.4. The numbers at the bottom indicate the number of nodes in the next level of the tree, of which there are $2(17+18+9)=88$ overall (i.e., we have $|{\mathcal B}_6|=88$)

1.3 B.3  Tree of Block-Aligned Rectangulations

By Lemmas B.3 and B.4, all block-aligned rectangulations ${\mathcal B}_n$ can be obtained by suitable rectangle insertions into all block-aligned rectangulations ${\mathcal B}_{n-1}$ and $s({\mathcal B}_{n-1}^\boxplus )$. We consider the subtree of the tree of rectangulations discussed in Sect. 3.4 induced by the rectangulations ${\mathcal B}_n$ and $s({\mathcal B}_n^\boxplus )$ for all $n\ge 1$. By gluing together pairs of nodes (R, s(R)) for all $R\in {\mathcal B}_n^\boxplus$, we obtain the tree shown in Fig. 27.

For any $P\in {\mathcal B}_{n-1}^\square$, using the notation from Lemma B.3 we define

$$\begin{aligned} c(P):={\left\{ \begin{array}{ll} (c_{i_1}(P),\ldots ,c_{i_{\lambda -1}}(P),c_{j_1}(P),c_{j_2}(P),\ldots ,c_{j_\mu }(P)) &{} \text { if } \lambda >1, \\ (c_{i_1}(P),c_{j_2}(P),\ldots ,c_{j_\mu }(P)) &{} \text { if } \lambda =1. \end{array}\right. } \end{aligned}$$

(6a)

For any and $P':=s(P)$, using the notation from Lemma B.4 we define

$$\begin{aligned} c(P):=(c_{i_1}(P),\ldots ,c_{i_{\lambda -1}}(P),c_{i_\lambda }(P), c_{j_1}(P'),c_{j_2}(P'),\ldots ,c_{j_\mu }(P')). \end{aligned}$$

(6b)

These sequences define an ordering among the children of each node in the aforementioned (unordered) tree of block-aligned rectangulations.

Note that any two consecutive rectangulations in the sequence (6a) differ in a T-flip, except $c_{i_{\lambda -1}}(P)$ and $c_{j_1}(P)$, and $c_{i_1}(P)$ and $c_{j_2}(P)$, which differ in a T-flip plus a simple flip. Similarly, any two consecutive rectangulations in the sequence (6b) differ in a T-flip, except $c_{i_\lambda }(P)$ and $c_{j_1}(P')$, which differ in a simultaneous flip of the two walls between $r_n$, $r_{n-1}$, and $r_{n-2}$. We refer to this operation as a D-flip (D like ‘double’).

1.4 B.4  Next Oracle for Block-Aligned Rectangulations

Using (6), we may modify the minimal jump oracle $\texttt {next}_{{\mathcal D}_n}$ for diagonal rectangulations described in Sect. 7.2 for the generation of block-aligned rectangulations within Algorithm M$^\square$ as follows. Some Gray code orderings produced by this algorithm are shown in Sect. 1.

Lines N2–N4 are symmetric to lines N5–N7, so we only consider N2–N4; see the illustrations in Fig. 28. Lines N2 and N4 perform a T-flip, possibly followed by a simple flip. Line N3 performs a simple flip followed by a T-flip, possibly followed by a simple flip, and in this case the combination of three flips, simple flip plus T-flip plus simple flip, yields a D-flip overall. The function $\texttt {lock}(R,j,\texttt {dir})$, $\texttt {dir}\in \{\,\vartriangleright \,,\,\vartriangle \,\}$, called at the end of each of the lines N2–N7 checks whether rectangle $r_j$ participates in an H-aligned block (if $\texttt {dir}\!=\,\vartriangleright \,$) or V-aligned block (if $\texttt {dir}\!=\,\vartriangle \,$) that is locked and must be transformed to an AH-aligned block or AV-aligned block by a simple flip. The function $\texttt {unlock}(R,j,d)$, $d\in \{ {\,\vartriangleleft \,}, {\,\vartriangleright \,} \}$, called at the beginning in line N1 does the converse, namely checking whether $r_j$ participates in an AH-aligned block or AV-aligned block that must be made H-aligned or V-aligned, respectively, before performing a jump of rectangle $r_j$ in direction d. The implementation of these functions is shown below for the cases $\texttt {dir}\!=\,\vartriangleright \,$ and $d\!=\,\vartriangleright \,$. The other variants $\texttt {dir}=\,\vartriangle \,$ and $d=\,\vartriangleleft \,$ are omitted for simplicity.

Fig. 28

Flip operations in lines N2–N4 of the oracle $\texttt {next}_{{\mathcal B}_n}$

To use Algorithm M$^\square$ with this oracle, in line M5 we also need to check whether $R^{[j-1]}$ is bottom-based or right-based (in addition to $R^{[j]}$), and whether $R^{[j-1]}$ or $R^{[j]}$ are one simple flip away from such a configuration. Similarly, to use this oracle in conjunction with the oracle $\texttt {next}_{{\mathcal B}_n({\mathcal P})}$ defined in Sect. 7.3, we need to test containment of a pattern P in the rectangulation R after a jump of rectangle $r_j$ not only via $\texttt {contains}(R,j,P)$, but also using $\texttt {contains}(R,j-1,P)$ and $\texttt {contains}(R,j+1,P)$, as all three rectangles $r_{j-1}$, $r_j$, and $r_{j+1}$ may be modified through one call of $\texttt {next}_{{\mathcal B}_n}(R,j,d)$. For details see our C++ implementation [45].

We obtain the following analogue of Theorems 4.2 and 5.1.

Theorem B.5

Let $n\ge 3$. For any set of patterns ${\mathcal P}$ that are neither bottom-based nor right-based nor simple-flippable to a bottom-based or right-based pattern, Algorithm M$^\square$ with the oracle $\texttt {next} _{{\mathcal B}_n({\mathcal P})}$ defined in Sect. 7.3, which calls $\texttt {next} _{{\mathcal B}_n}$ as defined above, visits every rectangulation from ${\mathcal B}_n({\mathcal P})$ exactly once, performing a sequence of one T- or D-flip plus at most three simple flips in each step.

It remains to analyze the running time of this algorithm.

Lemma B.6

Each call $\texttt {next} _{{\mathcal B}_n}(R,j,d)$ takes time ${\mathcal O}(1)$.

As we are dealing with a subset of diagonal rectangulations, the proof is very similar to the proof of Lemma 7.4.

Proof

Consider any of the calls Sjump(R, j, d), Tjump $_{\texttt {h}} (R,j,d)$, Tjump $_{\texttt {v}} (R,j,d)$ in lines N2–N7 and let $R'$ be the rectangulation after the call. As we only consider the first insertion point of each vertical group and the last insertion point of each horizontal group of $I(R^{[j-1]})$, we have $v(R,R')=0$ and $h(R,R')=0$, so the claim follows from Lemma 6.2, (b) and (c). $\square$

Lemma B.6 immediately yields the following result.

Theorem B.7

Algorithm M$^\square$ with the oracle $\texttt {next} _{{\mathcal B}_n}$ takes time ${\mathcal O}(1)$ to visit each block-aligned rectangulation.

For the pattern avoidance version of this algorithm, we obtain the following runtime bounds.

Theorem B.8

For any set of patterns , Algorithm M$^\square$ with the oracle $\texttt {next} _{{\mathcal B}_n({\mathcal P})}$ visits each rectangulation from ${\mathcal B}_n({\mathcal P})$ in time ${\mathcal O}(n)$.

Proof

The oracle $\texttt {next}_{{\mathcal B}_n({\mathcal P})}$ repeatedly calls the function $\texttt {next}_{{\mathcal B}_n}$. Applying Theorem 7.7 with the bound $f_n={\mathcal O}(1)$ from Lemma B.6 and the bound $t_n={\mathcal O}(1)$ from Lemma 7.8, the term $n(f_n+t_n)$ evaluates to ${\mathcal O}(n)$, as claimed. $\square$

Appendix C: Counting Results for Block-Aligned Rectangulations

Table 4 is the counterpart of Table 3 shown in Sect. 9 for block-aligned rectangulations ${\mathcal C}_n={\mathcal B}_n$ as a base class. Note that the patterns $P_3,\ldots ,P_8$ cannot be used, as they do not satisfy the conditions of Theorem B.5.

Table 4 Counts for pattern-avoiding rectangulations with block-aligned rectangulations as a base class

Appendix D: Visualization of Gray Codes

In this section we visualize the Gray codes obtained from our algorithms for generic rectangulations, diagonal rectangulations and block-aligned rectangulations (without any forbidden patterns). The corresponding 2-clumped permutations are shown below each rectangulation.

1.1 D.1  Generic Rectangulations

See Figs. 29, 30, 31, 32, and 33.

Fig. 29

$n=1$

Fig. 30

$n=2$

Fig. 31

$n = 5$. The two non-guillotine rectangulations are marked by *

Fig. 32

$n=3$

Fig. 33

$n=4$

1.2 D.2  Diagonal Rectangulations

See Figs. 34, 35, 36, 37, and 38.

Fig. 34

$n=1$

Fig. 35

$n=2$

Fig. 36

$n = 5$. The two non-guillotine rectangulations are marked by *

Fig. 37

$n=3$

Fig. 38

$n=4$

1.3 D.3  Block-Aligned Rectangulations

See Figs. 39, 40, 41, 42, and 43.

Fig. 39

$n=1$

Fig. 40

$n=2$

Fig. 41

$n=3$

Fig. 42

$n=4$

Fig. 43

$n=5$. The two non-guillotine rectangulations are marked by *