Classification of Triples of Lattice Polytopes with a Given Mixed Volume

Abstract

We present an algorithm for the classification of triples of lattice polytopes with a given mixed volume m in dimension 3. It is known that the classification can be reduced to the enumeration of so-called irreducible triples, the number of which is finite for fixed m. Following this algorithm, we enumerate all irreducible triples of normalized mixed volume up to 4 that are inclusion-maximal. This produces a classification of generic trivariate sparse polynomial systems with up to 4 solutions in the complex torus, up to monomial changes of variables. By a recent result of Esterov, this leads to a description of all generic trivariate sparse polynomial systems that are solvable by radicals.

1 Introduction

1.1 Formulation of the Problem and Previous Results

In the last decade, there has been an increased interest in the algorithmic theory of lattice polytopes, which is motivated by applications in algebra, algebraic geometry, combinatorics, and optimization (see, for example, [3, 4, 6, 8,9,10,11,12, 19,20,21, 23]). So far, a special emphasis has been put on computer-assisted enumeration results, which are important from different perspectives. On the one hand, one can carry out enumeration to gather concrete data and then make and verify hypotheses based on these data. On the other hand, proving results on lattice polytopes frequently requires to handle special cases, which are only finitely many but are hard to determine without algorithmic assistance. Most notably, some structural classification results for lattice polytopes hold up to finitely many exceptional situations and, in this case, one is interested in describing the exceptional situations by means of enumeration in order to accomplish a classification. Thus, the structural and algorithmic theory are intertwined and constantly influence each other. Evaluation of new data established using a computer-assisted search leads to new theoretical questions, while new theoretical results (in particular, finiteness results) suggest new enumeration tasks.

Our point of departure is the classical Bernstein–Khovanskii–Kouchnirenko theorem which determines the number of solutions of a generic system of Laurent polynomial equations:

Theorem 1.1

([7])  Let $A_1,\ldots , A_d$ be non-empty finite subsets of ${\mathbb {Z}}^d$ and consider the system of d equations $f_1=\cdots = f_d =0$ where each $f_i$ is a Laurent polynomial of the form

$$\begin{aligned} f_i(x_1,\ldots ,x_d)\,=\!\sum _{(a_1,\ldots ,a_d) \in A_i}\!\!c_{i,(a_1,\ldots ,a_d)} x_1^{a_1} \cdots x_d^{a_d}. \end{aligned}$$

Then, for a generic choice of the coefficients $c_{i,a} \in \mathbb {C}$, $1 \le i \le d$, $a \in A_i$, the number of solutions of the system $f_1= \cdots = f_d = 0$ in the complex torus $(\mathbb {C}\setminus \{0\})^d$ is equal to the normalized mixed volume $V(P_1,\ldots ,P_d)$ of the lattice polytopes $P_1 ={{\,\mathrm{conv}\,}}A_1,\ldots , P_d = {{\,\mathrm{conv}\,}}A_d$.

The subset $A_i$ is called the support of $f_i$ and the polytope $P_i={{\,\mathrm{conv}\,}}A_i$ is the Newton polytope of $f_i$. Systems with generic coefficients and prescribed supports are often called sparse as they may have much fewer monomials than generic systems with prescribed degrees of the polynomials. By generic choice of the coefficients we mean that the vector of all coefficients of the system is chosen outside an appropriate proper algebraic subset of $\mathbb {C}^{|A_1| + \cdots + |A_d|}$. There is extensive literature on sparse systems covering both computational and theoretical points of view. For example, the reader may consult [13, Chap. 7] for a list of references.

Example 1.2

Consider general equations $f_1=0$ and $f_2=0$ of vertically and horizontally aligned parabolas given by polynomials

$$\begin{aligned} f_1(x,y)&= c_{1,(0,0)}+ c_{1,(1,0)} x +c_{1,(2,0)} x^2 + c_{1,(0,1)} y,\\ f_2(x,y)&= c_{2,(0,0)}+c_{2,(1,0)} x + c_{2,(0,1)} y +c_{2,(0,2)} y^2. \end{aligned}$$

In this case the supports $A_1$ and $A_2$ are

$$\begin{aligned} A_1 = \{(0,0), (1,0), (2,0), (0,1)\},\qquad A_2 = \{(0,0), (1,0), (0,1),(0,2)\} \end{aligned}$$

and the Newton polytopes $P_1$ and $P_2$ are triangles

$$\begin{aligned} P_1= & {} {{\,\mathrm{conv}\,}}A_1 = {{\,\mathrm{conv}\,}}{((0,0),(2,0),(1,0))},\\ P_2= & {} {{\,\mathrm{conv}\,}}A_2 = {{\,\mathrm{conv}\,}}{((0,0),(1,0),(0,2))}. \end{aligned}$$

By Theorem 1.1, if the vector

$$\begin{aligned} (c_{1,(0,0)},c_{1,(1,0)}, c_{1,(2,0)},c_{1,(0,1)}, c_{2,(0,0)}, c_{2,(1,0)}, c_{2,(0,1)},c_{2,(0,2)}) \in \mathbb {C}^8 \end{aligned}$$

of all coefficients of the polynomials $f_1$ and $f_2$ is generic, then the system $f_1=f_2=0$ has exactly four solutions in $(\mathbb {C}\setminus \{0\})^2$, because the normalized mixed volume $V(P_1,P_2)$ of $P_1$ and $P_2$ equals 4. The value $V(P_1,P_2)$ can be determined from the inclusion–exclusion type formula

$$\begin{aligned} V(P_1,P_2) = \frac{\mathrm{Vol}\,{(P_1 + P_2)} - \mathrm{Vol}\,P_1 - \mathrm{Vol}\,P_2}{2}. \end{aligned}$$

Here $\mathrm{Vol}\,P$ denotes the normalized 2-dimensional volume of a polytope P, which is twice the Euclidean area of P, and $P_1+ P_2$ is the Minkowski sum of the two triangles. See also Fig. 1.

Fig. 1

The system $f_1=f_2=0$, where $f_1=0$ is a generic vertical parabola and $f_2=0$ is a generic horizontal parabola, has four solutions in $(\mathbb {C}\setminus \{0\})^2$, because the normalized mixed volume of $P_1$ and $P_2$ is 4

In the notation of Theorem 1.1, we call $(A_1,\ldots ,A_d)$ the (total) support of the system $f_1=\cdots = f_d=0$. Theorem 1.1 determines the number of solutions of a generic sparse system with a given support. Recently, a reverse research direction was suggested by Esterov and Gusev [15, 16]: One can fix the number of solutions m and try to classify all supports $(A_1,\ldots ,A_d)$ of generic systems with exactly m solutions. By Theorem 1.1, it suffices to use the convex hulls of the sets $A_1,\ldots ,A_d$ in such a classification. This leads to the following problem.

Classification Problem 1.3

Given $d, m \in$ ${\mathbb {N}}$, describe all d-tuples $(P_1,\ldots ,P_d)$ of lattice polytopes whose normalized mixed volume equals m.

The solution of this problem has been known only in the following cases:

• The trivial case $d=1$.

• The case $m=1$, for which a solution is provided by a result of Esterov and Gusev [15].

• The case $d=2$, $m \le 4$, for which a solution is described by Esterov and Gusev in [16].

Classification Problem 1.3 is of particular interest for $m \le 4$, since by a result of Esterov [14], it allows to describe all generic systems $f_1=\cdots = f_d=0$ as in Theorem 1.1 that are solvable by radicals.

The family of d-tuples of lattice polytopes with mixed volume m is invariant under application of a common unimodular transformation to all polytopes of the tuple, independent translations of the polytopes by lattice vectors, and permutations of the polytopes of the d-tuple. We call d-tuples equivalent if they coincide up to the above transformations.

Note that, even modulo the above equivalence relation, the set of such d-tuples is not finite. Nevertheless, the classification can be reduced to the enumeration of so-called irreducible tuples. Recall that a tuple $(P_1,\ldots ,P_d)$ of polytopes in ${\mathbb {R}}^d$ is called irreducible if the Minkowski sum of any k polytopes in the tuple is at least $(k+1)$-dimensional for every $1\le k\le d-1$. In [14] Esterov showed that the number of irreducible d-tuples of lattice polytopes is finite up to equivalence for a fixed mixed volume and dimension (see Theorem 2.4 below).

There is a natural partial ordering on the set of equivalence classes of d-tuples of lattice polytopes with a given mixed volume m, induced by inclusion. Maximal elements with respect to this order are called maximal d-tuples with a given mixed volume. Theorem 2.4 implies that every irreducible d-tuple of lattice polytopes is contained in a maximal irreducible d-tuple of lattice polytopes. As a consequence, enumeration of irreducible d-tuples amounts to enumeration of maximal irreducible d-tuples. In Example 1.2, the irreducible pair $(P_1,P_2)$ with the normalized mixed volume 4 can be embedded into a maximal irreducible pair $(P_1, 2 P_1)$ with the same normalized mixed volume (see Fig. 2). We refer to Sect. 3 for more details.

Fig. 2

Embedding the pair $(P_1,P_2)$ from Example 1.2 into a maximal pair by enlarging $P_2$ to $2P_1$. Since $V(P_1,2 P_1) = 2V(P_1,P_1) = 2 \mathrm{Vol}\,P_1=4$, the normalized mixed volume remains unchanged

Example 1.4

Consider the triple $(P_1,P_2,P_3)$ of lattice polytopes in ${\mathbb {R}}^3$, where $P_1, P_2$ are 2-dimensional polytopes $P_1={{\,\mathrm{conv}\,}}{(0, 2 e_1, e_2)}$, $P_2 = {{\,\mathrm{conv}\,}}{(0, e_1, 2 e_2)}$ in ${\mathbb {R}}^2\times \{0\}$, and $P_3$ is a 3-dimensional lattice polytope that has width 1 in the direction $e_3$ and is contained in the slab ${\mathbb {R}}^2 \times [0,1]$ of width 1. The triple $(P_1,P_2,P_3)$ is not irreducible since the sum $P_1+ P_2$ is 2-dimensional. The normalized mixed volume of $(P_1,P_2,P_3)$ is the product of the normalized mixed volume of the pair $(P_1,P_2)$, which is equal to 4 (see Example 1.2), multiplied by the width 1 of the polytope $P_3$. See also Fig. 3.

This calculation can also be interpreted in light of Theorem 1.1. The triple $(P_1,P_2,P_3)$ corresponds to a generic system $f_1(x,y)=f_2(x,y)=f_3(x,y,z)=0$ in three unknowns x, y, z, where $f_1(x,y)=f_2(x,y)=0$ is a sub-system depending only on x and y, with $f_1$ and $f_2$ chosen as in Example 1.2. Since the equation $f_3(x,y,z)=0$ is linear in z, it can written as

$$\begin{aligned} f_3(x,y,z) :=a(x,y)z+ b(x,y) = 0 \end{aligned}$$

for some $a,b\in \mathbb {C}[x,x^{-1},y,y^{-1}]$. As explained in Example 1.2, the generic system $f_1(x,y) = f_2(x,y)=0$ has four solutions in $(\mathbb {C}\setminus \{0\})^2$. If a(x, y) and b(x, y) are generic, then by plugging these four solutions into $f_3(x,y,z)=0$ we arrive at linear equations in z, each having exactly one solution $z \in \mathbb {C}\setminus \{0\}$. Thus, the generic system $f_1(x,y) = f_2(x,y) = f_3(x,y,z) = 0$ has four solutions in total.

Fig. 3

An example of a non-irreducible triple in dimension three

1.2 Our Contribution

In this paper we present an algorithmic approach to Classification Problem 1.3 when m and d are given and $d \le 3$. Our algorithm has produced all maximal irreducible pairs of lattice polytopes in ${\mathbb {R}}^2$ of mixed volume up to 10 and all maximal irreducible triples of lattice polytopes in ${\mathbb {R}}^3$ of mixed volume up to 4.

Esterov and Gusev’s approach for showing finiteness of irreducible d-tuples $(P_1,\ldots ,P_d)$ with mixed volume m is to provide a bound on the volume of the Minkowski sum $P_1 + \cdots + P_d$ in terms of m,

$$\begin{aligned} \mathrm{Vol}\,{(P_1 + \cdots + P_d) }\le b_d(m). \end{aligned}$$

This yields a naive algorithm as, by a well-known result of Lagarias and Ziegler [22, Thm. 2], there exists a bounding box containing, up to lattice equivalence, all lattice polytopes of volume at most $b_d(m)$ in dimension d. Consequently, one may assume that $P_1 + \cdots + P_d$ and, hence, each $P_i$ for $i \in [d]$ is contained in such a bounding box and carry out an exhaustive search for d-tuples that actually have normalized mixed volume m. While a sharp upper bound $b_d(m)$ is not known in general, it has to be at least $(m+d-1)^d$, as this value is realized by the d-tuple $(\Delta _d,\ldots ,\Delta _d,m \Delta _d)$ of normalized mixed volume m, where $\Delta _d$ denotes the standard d-dimensional simplex. In particular, $b_3(4)\ge 6^3=216$. Clearly, the above approach of showing the finiteness does not lead to a computationally feasible algorithm. Already the task of enumeration of all 3-dimensional lattice polytopes with normalized volume at most $b_3(4)$ is hopelessly hard, as the number of such polytopes is tremendous.

In order to obtain a feasible algorithm, we make an extensive use of the theory of mixed volumes and mixed surface area measures. In particular we use Aleksandrov–Fenchel inequality to produce upper bounds for the normalized mixed volumes $V(P_i,P_j,P_k)$ for all choices $i,j,k \in [3]$ and the normalized volumes of the $P_i$. Our algorithm is recursive in m. It has been crucial to find an appropriate case distinction that would allow to keep the enumeration procedure computationally tractable. At the highest level, we distinguish between the full-dimensional case, in which all $P_i$ are three-dimensional, and the non-full-dimensional case, in which some of the $P_i$ are two-dimensional.

We remark that we have been able to prove that the sharp upper bound $b_d(m)$ is, in fact, $(m+d-1)^d$ for $d=2,3$ in the case of full-dimensional polytopes. In addition, our enumeration verifies that $b_3(m)=(m+2)^3$ for $1\le m\le 4$ for all irreducible triples. (For $d=2$ the notions of irreducible and full-dimensional coincide.) Based on this evidence we conjecture that $b_d(m)=(m+d-1)^d$ for all irreducible d-tuples in an arbitrary dimension. More results on this conjecture can be found in [2].

The following result produced by our algorithm provides the answer to Classification Problem 1.3 for $d=3$ and $1\le m\le 4$. Parts (i) and (ii) of Theorem 1.5 and Corollary 1.6 reduce the problem to the case of $d=2$ and have already been obtained in [16]. There we use projections $\pi _{1,2}:{\mathbb {R}}^3 \rightarrow {\mathbb {R}}^2$ and $\pi _3:{\mathbb {R}}^3 \rightarrow {\mathbb {R}}^1$ given by $\pi _{1,2}(x_1,x_2,x_3) = (x_1,x_2)$ and $\pi _{3}(x_1,x_2,x_3) = x_3$, respectively.

Theorem 1.5

(Classification of triples of normalized mixed volume at most 4) Let $m \in \{1,2,3,4\}$. Then $(P_1,P_2,P_3)$ is a triple of lattice polytopes in ${\mathbb {R}}^3$ of normalized mixed volume m if and only if, up to equivalence of triples, it satisfies one of the following conditions.

• (i) For some $m_1, m_2 \in \{1,2,3,4\}$ satisfying $m=m_1 m_2$, $P_3$ is a lattice segment $\{0\}^2 \times [0,m_1]$, while $\pi _{1,2}(P_1)\subseteq Q_1$ and $\pi _{1,2}(P_2)\subseteq Q_2$ for some pair $(Q_1,Q_2)$ appearing in the list of pairs of lattice polytopes of normalized mixed volume $m_2$ given in [1].

• (ii) For some $m_1, m_2 \in \{1,2,3,4\}$ satisfying $m=m_1 m_2$, $\pi _3(P_3) = [0,m_1]$, while $P_1\subseteq Q_1 \times \{0\}$, $P_2\subseteq Q_2\times \{0\}$ for some pair $(Q_1,Q_2)$ appearing in the list of pairs of lattice polytopes of normalized mixed volume $m_2$ given in [1].

• (iii) $P_1 \subseteq Q_1$, $P_2 \subseteq Q_2$, and $P_3 \subseteq Q_3$ for some triple $(Q_1,Q_2,Q_3)$ appearing in the list of triples of lattice polytopes of normalized mixed volume m given in [1].

Recall that a monomial change of variables for Laurent polynomials is given by $(x_1,\ldots , x_d)\mapsto (x^{u_1},\ldots , x^{u_d})$, where $x^{u_i}=x_1^{u_{1i}}\cdots x_d^{u_{di}}$ for some unimodular matrix $U=(u_{ij})\in {\text {GL}}(d,{\mathbb {Z}})$. We call two systems $f_1=\cdots =f_d=0$ and $f_1'=\cdots =f_d'=0$ monomially equivalent if after a possible permutation of the $f_i$ there is a monomial change of variables which transforms $f_i$ to $x^{a_i}f'_i$ for some monomial $x^{a_i}$ for every $1\le i\le d$. Note that monomially equivalent systems have the same number of solutions in $(\mathbb {C}\setminus \{0\})^d$. As an immediate consequence of Theorems 1.5 and 1.1 we obtain the following result about generic Laurent polynomial systems in three variables.

Corollary 1.6

(Classification of trivariate Laurent polynomial systems with at most 4 solutions) Let $m \in \{1,2,3,4\}$. Then $f_1=f_2=f_3=0$ is a generic system of Laurent polynomials with support $(A_1,A_2,A_3)\subset ({\mathbb {Z}}^3)^3$ and which has m solutions in $(\mathbb {C}\setminus \{0\})^3$ if and only if, up to monomial equivalence, it satisfies one of the following conditions.

• (i) For some $m_1, m_2 \in \{1,2,3,4\}$ satisfying $m=m_1 m_2$, $A_3\subseteq \{0\}^2 \times \{0,\ldots ,m_1\}$, while $\pi _{1,2}(A_1)\subseteq Q_1$ and $\pi _{1,2}(A_2)\subseteq Q_2$ for some pair $(Q_1,Q_2)$ appearing in the list of pairs of lattice polytopes of normalized mixed volume $m_2$ given in [1].

• (ii) For some $m_1, m_2 \in \{1,2,3,4\}$ satisfying $m=m_1 m_2$, $\pi _3(A_3)\subseteq \{0,\ldots ,m_1\}$, while $A_1\subseteq Q_1 \times \{0\}$, $A_2\subseteq Q_2\times \{0\}$ for some pair $(Q_1,Q_2)$ appearing in the list of pairs of lattice polytopes of normalized mixed volume $m_2$ given in [1].

• (iii) $A_1 \subseteq Q_1$, $A_2 \subseteq Q_2$, and $A_3 \subseteq Q_3$ for some triple $(Q_1,Q_2,Q_3)$ appearing in the list of triples of lattice polytopes of normalized mixed volume m given in [1].

Examples 1.2 and 1.4 provide an illustration to Theorem 1.5 and Corollary 1.6. The triple $(P_1,P_2,P_3)$ from Example 1.4 is classified by case (i) of our result with $m_1=1$, $m_2=4$, and the pair $(Q_1,Q_2)$ coinciding with the pair $({{\,\mathrm{conv}\,}}{(0,2 e_1, e_2)}, 2 {{\,\mathrm{conv}\,}}{(0, 2 e_1, e_2)})$ up to equivalence. One of the outcomes of our algorithm is the following quantitative result.

Theorem 1.7

Let $N_{3}(m)$ (resp. $N_{3}'(m)$) be the number of equivalence classes of triples of all (resp. 3-dimensional) lattice polytopes in ${\mathbb {R}}^3$ of mixed volume m that are irreducible and maximal. We have the following table of values for $1\le m\le 4$:

m	$N_{3}(m)$	$N_{3}'(m)$
1	1	1
2	7	4
3	21	10
4	92	30

In particular, our enumeration verifies the list of maximal irreducible triples of lattice polytopes with mixed volume 2 proposed in [15].

For triples of 3-dimensional lattice polytopes our enumeration produces the following structural result. We have four general constructions that produce maximal triples for an arbitrary value of the mixed volume m (see also Propositions 3.7 and 3.8). Our enumeration shows that for $m\le 4$ all, but three exceptional triples, are covered by these constructions. Recall that a polytope P is a combinatorial pyramid if P has a facet containing all but one vertex of P.

Theorem 1.8

Let $(P_1,P_2,P_3)$ be a maximal triple of 3-dimensional lattice polytopes in ${\mathbb {R}}^3$ with $V(P_1,P_2,P_3) \le 4$. Then, up to equivalence of triples, either

• (0) $P_1, P_2, P_3$ are all equal to a lattice polytope P;

• (1) there exists a lattice polytope P such that $P_1=\alpha P$, $P_2=\beta P$, $P_3=\gamma P$ for some integers $\alpha , \beta , \gamma \ge 1$, not all the same;

• (2) there exists a lattice polytope P and a lattice segment I in ${\mathbb {R}}^3$, as well as integers $\alpha ,\beta \ge 1$ and $\gamma \ge 0$ such that

  $$\begin{aligned} P_1 = P + \alpha I,\quad P_2 = P+ \beta I,\quad \text {and}\quad P_3 = P + \gamma I; \end{aligned}$$

• (3) there exists a lattice segment I and a lattice polytope P, which is a combinatorial pyramid with base having two edges parallel to I, such that

  $$\begin{aligned} P_1=P_2=P\quad \text { and }\quad P_3=P+\alpha I, \end{aligned}$$

  for some integer $\alpha \ge 1$;

• (4) $(P_1,P_2,P_3)$ is one of the following exceptional triples given by

  • (i) $P_1=P_2={{\,\mathrm{conv}\,}}{(0,2e_1,e_2,e_3)}$, $P_3 = P_1 + [0,e_1]$,

  • (ii) $P_1=P_2={{\,\mathrm{conv}\,}}{(0,3e_1,e_2,e_3)}$, $P_3 = P_1 + [0,e_1]$,

  • (iii) $P_1=P_2={{\,\mathrm{conv}\,}}{(0,e_1,e_2,e_1+e_2+2e_3)}$, $P_3 = P_1 + [0,e_1+e_3]$.

The numbers of equivalence classes corresponding to the above five types are presented in the following table:

m	type (0)	type (1)	type (2)	type (3)	type (4)
1	1	0	0	0	0
2	3	1	0	0	0
3	6	1	1	1	1
4	17	5	3	3	2

Examples of triples of polytopes of types (1)–(3) are presented in Fig. 4. We refer to [1] for a complete list with both pictures and coordinate representation. Note that we can compute the mixed volume m of the triples in each of the types (0)–(3) by the following general formulas. For the triples of types (0) and (1) we have $m=\mathrm{Vol}\,P$ and $m=\alpha \beta \gamma \mathrm{Vol}\,P$, respectively. For the triples of type (2) we have $m=\mathrm{Vol}\,P+(\alpha + \beta + \gamma )\mathrm{Vol}\,_{\pi _I}\!P$, where $\mathrm{Vol}\,_{\pi _I}$ denotes the 2-dimensional volume of the projection along the direction of I. Finally, for type (3) triples, we have $m=\mathrm{Vol}\,P+ \alpha \mathrm{Vol}\,_{\pi _I}\!P$.

Fig. 4

Examples of triples of each of the types (1)–(3) of mixed volume 4

Parts of our algorithm also provide a direct way to carry out enumeration of pairs of polygons of given mixed volume. We have carried out this enumeration for mixed volume up to 10 and obtained the following result.

Theorem 1.9

Let $N_2(m)$ be the number of equivalence classes of pairs of 2-dimensional lattice polytopes in ${\mathbb {R}}^2$ of mixed volume m that are maximal. We have the following table of values for $1\le m\le 10$.

m	$N_2(m)$
1	1
2	3
3	6
4	13
5	18
6	38
7	46
8	87
9	118
10	202

Our computations have been carried out using Sagemath [28] and an implementation of our enumeration algorithm as well as data files containing the enumeration results can be found at https://github.com/christopherborger/mixed_volume_classification.

The approach presented in this paper also admits a natural extension to higher dimensions. However, such an extension will be computationally much more expensive already in dimension 4.

As auxiliary tools, we have also developed equivalence tests for configurations of lattice polytopes in Sect. 4 as well as an algorithm for enumeration of d-dimensional lattice polytopes with a given volume in Sect. 7.3. The algorithmic problem of enumeration of lattice polytopes with a given volume is also interesting in its own right. Independently, another algorithm for solving this problem has recently been developed by Gabriele Balletti [5]. One of the nice features of our approach to enumeration by volume is its simplicity. Furthermore, the general template of our volume enumeration algorithm can be modified to solve further enumeration problems in the theory of lattice polytopes that are similar in nature. All in all, we hope that the technical parts will also be useful in other contexts and will help advance the algorithmic theory of lattice polytopes, which has been emerging in the last decade.

2 Basic Notions and Background Results

2.1 Basic Notation

Let ${\mathbb {N}}$ be the set of positive integers. For $k \in {\mathbb {Z}}_{\ge 0}$ we use [k] to denote the set $\{1,\ldots ,k\}$ with [0] being the empty set. Throughout, $d \in {\mathbb {N}}$ denotes the dimension of the ambient space ${\mathbb {R}}^d$. We use $\subseteq$ and $\subset$ for inclusion and strict inclusion, respectively.

The problems that we consider involve a choice of a real d-dimensional Euclidean space E together with a lattice $\Lambda \subset E$ of full rank. This choice is not important for our results, so most of the time we work with the integer lattice ${\mathbb {Z}}^d$ in ${\mathbb {R}}^d$. We let $e_1,\ldots ,e_d$ denote the standard basis vectors in ${\mathbb {R}}^d$ which also form a basis of the lattice ${\mathbb {Z}}^d$. The notation ${{\,\mathrm{Id}\,}}_k$ stands for the $k \times k$ identity matrix. If X, Y are subsets of a vector space E, then their Minkowski sum $X+Y$ is given by $X+Y=\{x+y:x \in X, \, y \in Y\}$. Furthermore, for $\lambda \in {\mathbb {R}}$ and $X \subseteq E$, we use the notation $\lambda X = \{\lambda x:x \in X\}$.

2.2 Groups of Unimodular Transformations and Their Action

Let $\Lambda \subset E$ be as above. We denote by ${\text {GL}}(\Lambda )$ the group of all linear transformations $\phi$ on E that satisfy $\phi (\Lambda ) = \Lambda$. We call elements of ${\text {GL}}(\Lambda )$ linear unimodular transformations. By choosing a basis in $\Lambda$ we can identify ${\text {GL}}(\Lambda )$ with the group of $d \times d$ unimodular matrices, which are the matrices $U \in {\mathbb {Z}}^{d \times d}$ with $|{\det U}| = 1$.

On several occasions, we use the group of linear unimodular transformations $\phi \in {\text {GL}}(\Lambda )$ that act identically on a $(d-1)$-dimensional linear subspace H of E spanned by $d-1$ linearly independent lattice vectors. We call such transformations unimodular shearings along H. In particular, for $\Lambda ={\mathbb {Z}}^d$ unimodular shearings $\phi \in {\text {GL}}(\Lambda )$ along the coordinate hyperplane $H={\mathbb {R}}^{d-1} \times \{0\}$ have the form $\phi (x) = U x$ with

$$\begin{aligned} U = \begin{pmatrix} {{\,\mathrm{Id}\,}}_{d-1} &{} t \\ 0 &{} 1 \end{pmatrix} \end{aligned}$$

and $t \in {\mathbb {Z}}^{d-1}$.

By ${\text {Aff}}(\Lambda )$ we denote the group of affine transformations $\phi$ on E satisfying $\phi (\Lambda ) = \Lambda$. These are transformations $\phi$ of the form $\phi (x)=\psi (x)+v$, where $\psi$ is a linear unimodular transformation and $v\in \Lambda$ is a lattice vector. We call elements of ${\text {Aff}}(\Lambda )$ affine unimodular transformations. Clearly, ${\text {GL}}(\Lambda )$ is a subgroup of ${\text {Aff}}(\Lambda )$ and ${\text {Aff}}(\Lambda )$ acts on subsets of E so that $\phi \in {\text {Aff}}(\Lambda )$ sends $X \subseteq E$ to its image $\phi (X)$. With this in mind, we introduce equivalences of subsets of E modulo subgroups of ${\text {Aff}}(\Lambda )$. If X and Y are subsets of E and G is a subgroup of ${\text {Aff}}(\Lambda )$, we say that X and Y coincide modulo G if $\phi (X)=Y$ for some $\phi \in G$. In this case, we write $X \equiv Y$ mod G. We also let ${\text {Aff}}(\Lambda )$ act on k-tuples $(X_1,\ldots ,X_k)$ of subsets of E by $\phi (X_1,\ldots , X_k) = (\phi (X_1),\ldots ,\phi (X_k))$. Thus, analogously, we say that two such k-tuples $(X_1,\ldots ,X_k)$ and $(Y_1,\ldots ,Y_k)$ coincide modulo G if $(\phi (X_1),\ldots ,\phi (X_k)) = (Y_1,\ldots ,Y_k)$ holds for some $\phi \in G$.

There is yet another group, which we denote by $\mathbb {G}_{d,k}$, that we let act on k-tuples $(X_1,\ldots ,X_k)$. An element of $\mathbb {G}_{d,k}$ is determined by a linear unimodular transformation $\phi \in {\text {GL}}(\Lambda )$, k lattice vectors $v_1,\ldots ,v_k \in \Lambda$, and a permutation $\sigma$ on [k]. Elements of $\mathbb {G}_{d,k}$ are transformations of $({\mathbb {R}}^d)^k$ of the form $(x_1,\ldots ,x_k)\mapsto (\phi (x_{\sigma (1)})+v_{\sigma (1)},\ldots ,\phi (x_{\sigma (k)})+v_{\sigma (k)})$. We say that two k-tuples $(X_1,\ldots ,X_k)$ and $(Y_1,\ldots ,Y_k)$ of subsets of ${\mathbb {R}}^d$ coincide up to $\mathbb {G}_{d,k}$-equivalence, if $\psi (X_1,\ldots ,X_k) = (Y_1,\ldots ,Y_k)$ holds for some $\psi \in \mathbb {G}_{d,k}$. We write $(X_1,\ldots ,X_k) \equiv (Y_1,\ldots ,Y_k)$ mod $\mathbb {G}_{d,k}$ to denote this equivalence relation.

2.3 Lattice Polytopes

Let E be a d-dimensional Euclidean space with a full-rank lattice $\Lambda$. By ${{\,\mathrm{conv}\,}}$ we denote the convex hull operation on E. A polytope in E is the convex hull of finitely many points in E. For a polytope $P\subset E$ we use ${{\,\mathrm{aff}\,}}(P)$ to denote the affine span of P, that is the smallest affine subspace of E containing P. By definition, $\dim P=\dim {{\,\mathrm{aff}\,}}(P)$. Given a polytope $P\subset E$, its support function $h_P:{\mathbb {R}}^d\rightarrow {\mathbb {R}}$ is defined by

$$\begin{aligned} h_P(\xi )=\max {\{\langle \xi , x\rangle : x \in K\}}. \end{aligned}$$

Here $\langle \xi , x\rangle$ denotes the inner product in E which in the case of $E={\mathbb {R}}^d$ is the standard inner product. Given $\xi \in {\mathbb {R}}^d$ we use $P^\xi$ to denote the face of P corresponding to $\xi$ defined by

$$\begin{aligned} P^\xi =\{ x \in P: \langle \xi , x\rangle = h_P(\xi )\}. \end{aligned}$$

Clearly, when $\xi =0$ we get $P^\xi =P$. For $\xi$ not equal to zero, $P^\xi$ depends only on the direction of $\xi$. Faces of dimension $\dim P-1$ are called facets and faces of dimension 0 are called vertices of P. We let ${\text {vert}}(P)$ denote the vertex set of P. When $\xi$ has Euclidean length 1 and $P^{\xi }$ is a facet, $\xi$ is uniquely determined by $P^{\xi }$ and is called the unit outer facet normal of $P^\xi$.

Given $X \subseteq E$, we denote by $\mathcal {P}(X)$ the family of all non-empty polytopes P with ${\text {vert}}(P) \subseteq X$. For $k \in [d]$, we use $\mathcal {P}_k(X)$ to denote the family of k-dimensional polytopes belonging to $\mathcal {P}(X)$. In this paper, we address enumeration and classification problems within the family $\mathcal {P}(\Lambda )$ of lattice polytopes.

Let $\{v_1,\ldots , v_d\}$ be a lattice basis for $\Lambda$. We call the simplex ${{\,\mathrm{conv}\,}}{(0,v_1,\ldots ,v_d)}$ and any of its lattice translates a unimodular simplex. In particular, when $\Lambda ={\mathbb {Z}}^d$ and $\{e_1,\ldots ,e_d\}$ is the standard basis for ${\mathbb {Z}}^d$, we call $\Delta _d : = {{\,\mathrm{conv}\,}}{(0,e_1,\ldots ,e_d)}$ the standard simplex. Note that a simplex in ${\mathbb {R}}^d$ is unimodular (with respect to the lattice ${\mathbb {Z}}^d$) if and only if it is ${\text {Aff}}({\mathbb {Z}}^d)$-equivalent to $\Delta _d$.

For lattice polytopes $P\in \mathcal {P}({\mathbb {Z}}^d)$ it is more convenient to consider the support function on the set $\mathcal {S}^{d-1}\subset {\mathbb {Z}}^d$ of primitive vectors. Recall that an integer vector $u\in {\mathbb {Z}}^d$ is primitive if its entries do not have a common divisor greater than 1. Then the support function $h_P(u)=\max \,\{\langle u,x\rangle :x\in P\}$ takes integer values for all $u\in \mathcal {S}^{d-1}$. As before, we will speak about faces $P^u$ of P. In the case when $P^u$ is a facet, u is called the primitive outer facet normal of $P^u$.

One of the central functionals that we want to consider on $\mathcal {P}(\Lambda )$ is the Euclidean volume (i.e., the d-dimensional Lebesgue measure restricted to $\mathcal {P}(\Lambda )$). Working with lattice polytopes, along with the Euclidean volume $\mathrm{vol}\,$, one also considers the so-called normalized volume $\mathrm{Vol}\,$ relative to $\Lambda$, scaled so that the volume of a unimodular simplex equals one. The advantage of the scaling is that $\mathrm{Vol}\,P\in {\mathbb {Z}}$ for each $P \in \mathcal {P}(\Lambda )$. For $P\in \mathcal {P}({\mathbb {Z}}^d)$ we have $\mathrm{Vol}\,P=d!\mathrm{vol}\,P$ as $\mathrm{vol}\,\Delta _d=1/d!$

2.4 Mixed Volumes

Let E be a d-dimensional Euclidean space. There exists a uniquely defined functional

$$\begin{aligned} v:\,\underbrace{\mathcal {P}(E) \times \cdots \times \mathcal {P}(E)}_d \rightarrow {\mathbb {R}}, \end{aligned}$$

with $v(P_1,\ldots ,P_d)$ being invariant under permutations of $P_1,\ldots ,P_d \in \mathcal {P}(E)$, such that the equality

$$\begin{aligned} \mathrm{vol}\,{(\lambda _1 P_1 + \cdots + \lambda _k P_k)} = \sum _{i_1=1}^k\!\cdots \!\sum _{i_d=1}^k\, \lambda _{i_1}\!\cdots \lambda _{i_d}v(P_{i_1},\ldots ,P_{i_d}) \end{aligned}$$

holds for all $P_1,\ldots ,P_k\in \mathcal {P}(E)$, non-negative scalars $\lambda _1,\ldots , \lambda _k \ge 0$, and $k \in {\mathbb {N}}$ (see [25, Thm. and Defn. 5.1.7]). One can extend v to the set of d-tuples of non-empty compact convex sets, but for the purposes of this paper, it will be enough to consider the case of polytopes. The value $v(P_1,\ldots ,P_d)$ is called the Euclidean mixed volume of the d-tuple $(P_1,\ldots ,P_d)$. Replacing the Euclidean volume in the above definition with the normalized volume relative to some lattice $\Lambda \subset E$, we obtain the normalized mixed volume $V(P_1,\ldots ,P_d)$ relative to $\Lambda$. In what follows when we say “volume” or “mixed volume” we always assume “normalized volume” or “normalized mixed volume” relative to some lattice. When the lattice is not specified it is meant to be ${\mathbb {Z}}^d\subset {\mathbb {R}}^d$ for appropriate $d\in {\mathbb {N}}$.

The mixed volume satisfies a number of properties that will be useful for our considerations. Their proofs can be found in [25, Sects. 5.1, 7.3] and [17, p. 120].

Proposition 2.1

For $P_1,\ldots ,P_d,Q_1,\ldots ,Q_d \in \mathcal {P}({\mathbb {R}}^d)$ and non-negative $\lambda ,\mu \in {\mathbb {R}}$, we have:

• 1. $V(P_1,\ldots ,P_d)\ge 0$.

• 2. $V(P_1,\ldots ,P_d)=V(Q_1,\ldots ,Q_d)$ if $(P_1,\ldots ,P_d) \equiv (Q_1,\ldots ,Q_d)$ mod $\mathbb {G}_{d,d}$.

• 3. $V(\lambda P_1+\mu Q_1,P_2,\ldots ,P_d)=\lambda V(P_1,P_2,\ldots ,P_d)+\mu V(Q_1,P_2,\ldots ,P_d)$.

• 4. $V(P_1,\ldots ,P_d) \in {\mathbb {Z}}$, whenever $P_1,\ldots ,P_d \in \mathcal {P}({\mathbb {Z}}^d)$.

• 5. Inclusion–exclusion formula

  $$\begin{aligned} V(P_1,\ldots ,P_d)=\frac{1}{d!}\sum _{k=1}^d(-1)^{d+k}\!\sum _{i_1<\ldots <i_k}\mathrm{Vol}\,{(P_{i_1}+\cdots +P_{i_k})}. \end{aligned}$$

  (2.1)

• 6. Aleksandrov–Fenchel Inequality

  $$\begin{aligned} V(P_1,P_2,P_3\ldots ,P_d)^2\ge V(P_1,P_1,P_3,\ldots ,P_d)V(P_2,P_2,P_3,\ldots ,P_d).\end{aligned}$$

• 7. Monotonicity

  $$\begin{aligned} V(P_1,\ldots ,P_d)\le V(Q_1,\ldots ,Q_d)\quad \text {if}\quad P_1\subseteq Q_1,\ldots , P_d\subseteq Q_d. \end{aligned}$$

  (2.2)

Definition 2.2

We say that a d-tuple $(P_1,\dots ,P_d)\in \mathcal {P}(E)^d$ is non-degenerate if for every $I\subseteq [d]$ with $1 \le |I|\le d$ the dimension of $\sum _{i\in I}P_i$ is at least |I|. We say that a d-tuple $(P_1,\dots ,P_d)\in \mathcal {P}(E)^d$ is irreducible if for every $I\subseteq [d]$ with $1 \le |I|<d$ the dimension of $\sum _{i\in I}P_i$ is at least $|I|+1$.

It has been observed by Minkowski [25, Thm. 5.1.8] that $V(P_1,\dots ,P_d)$ is positive if and only if the d-tuple $(P_1,\dots ,P_d)$ is non-degenerate. The notion of irreducible d-tuples is related to the following decomposition property of the mixed volume.

Proposition 2.3

([25, Thm. 5.3.1]) Let $P_1,\ldots , P_d \in \mathcal {P}({\mathbb {R}}^d)$ be such that, for some $k \in [d]$, $P_1,\ldots ,P_k$ are contained in a rational linear subspace $L\subset {\mathbb {R}}^d$ of dimension k, and let $\pi _L:{\mathbb {R}}^d\rightarrow {\mathbb {R}}^d/L$ be the projection along L. Then

$$\begin{aligned} V(P_1,\ldots , P_k,P_{k+1},\ldots , P_d)=V(P_1,\ldots , P_k)V(\pi _L(P_{k+1}),\ldots ,\pi _L(P_d)). \end{aligned}$$

In the equality above, $V(P_1,\ldots , P_k)$ is the mixed volume of $(P_1,\ldots ,P_k) \in \mathcal {P}(L)^k$ relative to the sublattice $L\cap {\mathbb {Z}}^d$, while $V(\pi _L(P_{k+1}),\ldots ,\pi _L(P_d))$ is the mixed volume of $(\pi _L(P_{k+1}),\ldots ,\pi _L(P_d)) \in \mathcal {P}({\mathbb {R}}^d/L)^{d-k}$ relative to the quotient lattice ${\mathbb {Z}}^d/(L\cap {\mathbb {Z}}^d)$. Proposition 2.3 allows one to reduce the problem of classifying all d-tuples $(P_1,\ldots ,P_d)\in \mathcal {P}({\mathbb {Z}}^d)^d$ with given mixed volume to classifying such irreducible tuples. The following result by Esterov paves the way for solving Classification Problem 1.3 via enumeration.

Theorem 2.4

([14, Thm. 1.7]) Given $m\in {\mathbb {N}}$ there exist finitely many irreducible d-tuples $(P_1,\ldots , P_d)\in \mathcal {P}({\mathbb {Z}}^d)^d$ with $V(P_1,\ldots , P_d)=m$, up to $\mathbb {G}_{d,d}$-equivalence.

Note that this statement fails for non-irreducible tuples. For a simple example one can take $A={{\,\mathrm{conv}\,}}{(0,e_2)}$ and $B_n={{\,\mathrm{conv}\,}}{(0,e_1,ne_2)}$ for $n\in {\mathbb {N}}$. Then, by Proposition 2.3, $V(A,B_n)=1$ regardless of n.

2.5 Surface Area Measure and Mixed Area Measure

Let $P \subset {\mathbb {R}}^d$ be a polytope with unit outer facet normals $\xi _1,\ldots , \xi _N$ and support function $h_P$. We have

$$\begin{aligned} \mathrm{vol}\,P=\frac{1}{d}\sum _{i=1}^N h_P(\xi _i) \mathrm{vol}\,P^{\xi _i}. \end{aligned}$$

(2.3)

Indeed, assuming 0 lies in the interior of P, the above sum represents the volume of P as the sum of the volumes of pyramids over the facets of P. This motivates the notion of the Euclidean surface area measure $s_P$ on the sphere $\mathbb {S}^{d-1}$ with finite support $\{\xi _1,\ldots ,\xi _N\}\subset \mathbb {S}^{d-1}$ and values $s_P(\xi _i)=\mathrm{vol}\,P^{\xi _i}$, the $(d-1)$-dimensional Euclidean volume of the facet $P^{\xi _i}$, for $1\le i\le N$. Thus, (2.3) can be written as

$$\begin{aligned} \mathrm{vol}\,P=\frac{1}{d}\int _{\mathbb {S}^{d-1}}\!h_P(\xi )\,ds_P(\xi ). \end{aligned}$$

(2.4)

We refer to [25, Sect. 5.1] for details. For lattice polytopes $P\in \mathcal {P}({\mathbb {Z}}^d)$ one modifies the definition as follows. Let $\{u_1,\ldots ,u_N\}$ be primitive outer facet normals of P. Then we have

$$\begin{aligned} \mathrm{Vol}\,P=\sum _{i=1}^N h_P(u_i) \mathrm{Vol}\,P^{u_i}, \end{aligned}$$

(2.5)

where $\mathrm{Vol}\,P^{u_i}$ is the $(d-1)$-dimensional volume relative to the lattice ${{\,\mathrm{aff}\,}}(P^{u_i})\cap {\mathbb {Z}}^d$. The advantage of (2.5) is that all the terms in the sum are non-negative integers. For this reason we introduce the (normalized) surface area measure $S_P$ on the set of primitive vectors $\mathcal {S}^{d-1}$ with support ${\text {supp}}S_P=\{u_1,\ldots ,u_N\}$ and values $S_P(u)=\mathrm{Vol}\,P^{u}$. Then (2.5) can be written as

$$\begin{aligned} \mathrm{Vol}\,P=\int _{\mathcal {S}^{d-1}}\!h_P(u)\,dS_P(u). \end{aligned}$$

(2.6)

See also Fig. 5 for an illustration.

Fig. 5

Surface area measure $s_P$ and the normalized surface area measure $S_P$ of the triangle $P={{\,\mathrm{conv}\,}}{(0,2 e_1, 2 e_2)}$. While $s_P$ provides the Euclidean edge length for each outer unit edge normal vector, $S_P$ provides the lattice length for each outer primitive edge normal vector

Let $P_1,\ldots , P_{d-1}$ be polytopes in ${\mathbb {R}}^d$. The Euclidean mixed area measure $s_{P_1,\ldots ,P_{d-1}}$ is defined by the inclusion–exclusion formula [25, p. 281]:

$$\begin{aligned} s_{P_1,\ldots , P_{d-1}}=\frac{1}{(d-1)!}\sum _{k=1}^{d-1}(-1)^{d-1-k} \!\!\sum _{i_1<\ldots <i_k}\!\!s_{P_{i_1}+\cdots +P_{i_k}}. \end{aligned}$$

Then, by (2.1) in Proposition 2.1, we have

$$\begin{aligned} s_{P_1,\ldots , P_{d-1}}(\xi )=v(P_1^\xi ,\ldots , P_{d-1}^\xi ) \end{aligned}$$

(2.7)

for any $\xi \in \mathbb {S}^{d-1}$. This implies that $s_{P_1,\ldots , P_{d-1}}$ has support consisting of vectors $\xi \in \mathbb {S}^{d-1}$ such that $(P_1^\xi ,\ldots , P_{d-1}^\xi )$ is non-degenerate (see the remark after Definition 2.2). Note that the support is contained in the set of outer facet normals of $P_1+\cdots +P_{d-1}$, and hence is finite. Furthermore, let ${\text {supp}}s_{K_1,\ldots , K_{d-1}}=\{\xi _1,\ldots , \xi _N\}$. We have mixed analogs of (2.3) and (2.4) (see [25, Thm. 5.1.7]):

$$\begin{aligned} v(P_1,\ldots , P_{d-1},P)&=\frac{1}{d}\sum _{i=1}^Nh_P(\xi _i)v(P_1^{\xi _i},\ldots , P_{d-1}^{\xi _i}), \end{aligned}$$

(2.8)

$$\begin{aligned} v(P_1,\ldots , P_{d-1},P)&=\frac{1}{d}\int _{\mathbb {S}^{d-1}}\!h_P(\xi )\,ds_{P_1,\ldots , P_{d-1}}(\xi ), \end{aligned}$$

(2.9)

for any polytope $P\subset {\mathbb {R}}^d$. Now let $u_i$ be the primitive vector in the direction of $\xi _i$, for $1\le i\le N$. Normalizing (2.8) we obtain

$$\begin{aligned} V(P_1,\ldots , P_{d-1},P)=\sum _{i=1}^Nh_P(u_i)V(P_1^{u_i},\ldots , P_{d-1}^{u_i}). \end{aligned}$$

(2.10)

Similarly to the unmixed case we define the (normalized) mixed area measure $S_{P_1,\ldots , P_{d-1}}$ on the set of primitive vectors $\mathcal {S}^{d-1}$ with support ${\text {supp}}S_{P_1,\ldots , P_{d-1}}$ and values $S_{P_1,\ldots , P_{d-1}}(u)=V(P_1^{u},\ldots , P_{d-1}^{u})$. As before, the terms in (2.10) are non-negative integers whenever P is also a lattice polytope. We write (2.10) as

$$\begin{aligned} V(P_1,\ldots , P_{d-1},P)=\int _{\mathcal {S}^{d-1}}\!h_P(u)\,dS_{P_1,\ldots , P_{d-1}}(u). \end{aligned}$$

(2.11)

Note that, similarly to the mixed volume, the mixed area measure is Minkowski linear in each of its arguments.

The following proposition relates the notions of irreducible tuples and mixed area measure.

Proposition 2.5

A tuple $(P_1,\ldots ,P_{d-1})\in \mathcal {P}({\mathbb {R}}^d)^{d-1}$ can be extended to an irreducible d-tuple if and only if ${\text {supp}}s_{P_1,\ldots ,P_{d-1}}$ positively spans ${\mathbb {R}}^d$.

Proof

Let s denote the measure $s_{P_1,\ldots ,P_{d-1}}$. First note that

$$\begin{aligned} \sum _{\xi \in {\text {supp}}s}\!\xi s(\xi )=0. \end{aligned}$$

(2.12)

Indeed, by (2.9) for any $x\in {\mathbb {R}}^d$ and a polytope P we have

$$\begin{aligned} 0=v(P_1,\ldots ,P_{d-1},P+x)-v(P_1,\ldots ,P_{d-1},P) =\frac{1}{d}\int _{\mathbb {S}^{d-1}}\!\langle \xi ,x\rangle \,ds(\xi ). \end{aligned}$$

Since x is arbitrary we obtain (2.12). Therefore, ${\text {supp}}s$ positively spans ${\mathbb {R}}^d$ if and only if ${\text {supp}}s$ linearly spans ${\mathbb {R}}^d$.

Now suppose ${\text {supp}}s$ is contained in $\mathbb {S}^{d-1}\cap H$ for some $(d-1)$-dimensional linear subspace H. Then, by (2.9), for any segment I orthogonal to H we have $v(P_1,\ldots ,P_{d-1},I)=0$ since $h_I(\xi )=0$ for any $\xi \in {\text {supp}}s$. This means that the d-tuple $(P_1,\ldots ,P_{d-1},I)$ is degenerate, which is impossible if $(P_1,\ldots ,P_{d-1})$ can be extended to an irreducible d-tuple.

Conversely, suppose $(P_1,\ldots ,P_{d-1})$ cannot be extended to an irreducible d-tuple. Without loss of generality we may assume that $P_1+\cdots +P_k$ is contained in a k-dimensional subspace L for some $1\le k\le d-1$. For any $\xi \in {\text {supp}}s$,

$$\begin{aligned} s_{P_1,\ldots ,P_{d-1}}(\xi )=v(P_1^\xi ,\ldots ,P_{d-1}^\xi )>0, \end{aligned}$$

and so $(P_1^\xi ,\ldots ,P_{d-1}^\xi )$ is non-degenerate. In particular,

$$\begin{aligned} k\le \dim {(P_1^\xi +\cdots +P_{k}^\xi )}=\dim {(P_1+\cdots +P_{k})^\xi }\le \dim {(P_1+\cdots +P_k)}=k. \end{aligned}$$

Therefore, $P_1+\cdots +P_k=(P_1+\cdots +P_{k})^\xi$ which implies $\xi \in L^\perp$. Thus, ${\text {supp}}s$ is a subset of $\mathbb {S}^{d-1}\cap L^\perp$. $\square$

Remark 2.6

One readily sees that, for tuples of lattice polytopes, the above proposition can be restated as follows: A tuple $(P_1,\ldots ,P_{d-1})\in \mathcal {P}({\mathbb {Z}}^d)^{d-1}$ can be extended to an irreducible d-tuple if and only if ${\text {supp}}S_{P_1,\ldots ,P_{d-1}}$ positively spans ${\mathbb {R}}^d$.

3 ${\mathbb {Z}}$-Maximal and ${\mathbb {R}}$-Maximal Tuples

Recall that the mixed volume $V(P_1,\ldots ,P_d)$ is monotonic with respect to inclusion, see (2.2). This motivates the notion of ${\mathbb {Z}}$-maximal and ${\mathbb {R}}$-maximal d-tuples of polytopes. In the definition below R denotes either ${\mathbb {Z}}$ or ${\mathbb {R}}$.

Definition 3.1

Let $i \in [d]$. A d-tuple $(P_1,\ldots ,P_d) \in \mathcal {P}(R^d)^d$ is called R-maximal in $P_i$, if for all $Q_i \in \mathcal {P}(R^d)$ with $P_i\subseteq Q_i$ the equality

$$\begin{aligned} V(P_1,\ldots ,P_{i-1},P_i,P_{i+1},\ldots ,P_d) = V(P_1,\ldots ,P_{i-1},Q_i,P_{i+1},\ldots ,P_d) \end{aligned}$$

(3.1)

implies $P_i=Q_i$. We call $(P_1,\ldots ,P_d)$ R-maximal if it is R-maximal in each of the polytopes $P_1,\ldots ,P_d$.

In view of the inclusion ${\mathbb {Z}}\subset {\mathbb {R}}$, if a d-tuple of lattice polytopes is ${\mathbb {R}}$-maximal, then it is also ${\mathbb {Z}}$-maximal. The converse is not true in general as Figs. 6 and 7 illustrate.

Fig. 6

A pair (A, B) of lattice polygons, which is ${\mathbb {R}}$-maximal in A and ${\mathbb {Z}}$-maximal but not ${\mathbb {R}}$-maximal in B. The dashed line depicts how B can be enlarged to a non-lattice polygon $B' = {{\,\mathrm{conv}\,}}{(0 , 3 e_1, ({3}/{2})e_2)}$ such that the pair $(A,B' )$ has the same mixed volume as (A, B)

Fig. 7

A ${\mathbb {Z}}$-maximal pair (A, B) of lattice polygons for which both A and B can be enlarged to non-lattice polygons $A'$ and $B'$ such that $(A',B')$ has the same mixed volume as (A, B)

Remark 3.2

There is a natural poset structure on the set $\mathcal {P}({\mathbb {Z}}^d)^d/\mathbb {G}_{d,d}$ of $\mathbb {G}_{d,d}$-equivalence classes of d-tuples of polytopes in $\mathcal {P}({\mathbb {Z}}^d)$. Let $\mathbf {P}$ and $\mathbf {Q}$ be classes containing $(P_1,\ldots , P_d)$ and $(Q_1,\ldots , Q_d)$, respectively. We say that $\mathbf {P}\le \mathbf {Q}$ if and only if $\psi (P_1,\ldots , P_d)$ is contained in $(Q_1,\ldots , Q_d)$ component-wise for some $\psi \in \mathbb {G}_{d,d}$. One can directly verify that this relation defines a partial order on $\mathcal {P}({\mathbb {Z}}^d)^d/\mathbb {G}_{d,d}$. Given $m\in {\mathbb {N}}$, one may restrict this poset to the sub-poset of $\mathbb {G}_{d,d}$-equivalence classes of d-tuples $(P_1,\ldots ,P_d)$ of normalized mixed volume m. Maximal elements of this poset correspond to ${\mathbb {Z}}$-maximal d-tuples of polytopes in the sense of Definition 3.1. Note that if $(P_1,\ldots ,P_d)$ is not irreducible then the corresponding class is not a maximal element of the poset. Indeed, suppose $P_1,\ldots , P_k$ lie in a k-dimensional subspace L, for some $1\le k<d$. Then, by Proposition 2.3, adding any segment whose direction vector lies in $L^\perp$ to $P_{k+1},\ldots , P_d$ does not change the mixed volume of $(P_1,\ldots ,P_d)$ and, hence, the class of $(P_1,\ldots ,P_d)$ lies in an infinite chain in the poset. This shows that maximal elements are, in particular, irreducible. In what follows, however, we will always write “maximal irreducible tuples” for clarity. Theorem 2.4 asserts that the subposet of classes of irreducible d-tuples of lattice polytopes with fixed mixed volume is finite and, in particular, has finitely many maximal elements.

The following proposition describes irreducible d-tuples $(P_1,\ldots , P_{d})$ that are ${\mathbb {Z}}$-maximal in $P_d$. In particular, it provides an algorithmic way of finding all possible $P_d\in \mathcal {P}({\mathbb {Z}}^d)$ such that $V(P_1,\ldots ,P_{d})=m$ and $(P_1,\ldots , P_{d})$ is ${\mathbb {Z}}$-maximal in $P_d$, given a value of m and a $(d-1)$-tuple $(P_1,\ldots , P_{d-1})\in (\mathcal {P}({\mathbb {Z}}^d))^{d-1}$ such that ${\text {supp}}s_{P_1,\ldots ,P_{d-1}}$ positively spans ${\mathbb {R}}^d$ (see Algorithm 7.2).

Proposition 3.3

Let $(P_1,\ldots , P_{d})\in (\mathcal {P}({\mathbb {Z}}^d))^d$ be an irreducible tuple which is ${\mathbb {Z}}$-maximal in $P_d$. Let $\{u_1,\ldots ,u_r\}$ be the support of the mixed area measure $S_{P_1,\ldots ,P_{d-1}}$. Then

$$\begin{aligned} P_d={{\,\mathrm{conv}\,}}{\{x\in {\mathbb {Z}}^d : \langle u_i,x\rangle \le h_i,\, i\in [r]\}} \end{aligned}$$

(3.2)

for some $h_1,\ldots ,h_r\in {\mathbb {Z}}_{\ge 0}$ satisfying

$$\begin{aligned} \sum _{i=1}^rh_iS_{P_1,\ldots ,P_{d-1}}(u_i)=V(P_1,\ldots ,P_{d}). \end{aligned}$$

(3.3)

Proof

Let $h_i=h_{P_d}(u_i)$, the value of the support function of $P_d$ at $u_i$. Then (3.3) follows directly from (2.11). Consider

$$\begin{aligned} Q=\{x\in {\mathbb {R}}^d : \langle u_i,x\rangle \le h_i \ \text {for all} \ i\in [r]\}. \end{aligned}$$

This is a rational polytope. (Its boundedness follows from Proposition 2.5.) Clearly, $P_d\subseteq Q$ and $h_{P_d}$ coincides with $h_Q$ on the support of $S_{P_1,\ldots ,P_{d-1}}$. Therefore, by (2.11) and (3.3),

$$\begin{aligned} V(P_1,\ldots , P_{d-1},Q)=\sum _{i=1}^rh_iS_{P_1,\ldots ,P_{d-1}}(u_i)=V(P_1,\ldots ,P_{d}). \end{aligned}$$

Now the ${\mathbb {Z}}$-maximality in $P_d$ and the monotonicity of the mixed volume imply that $P_d$ must contain all lattice points of Q, i.e., (3.2) holds. $\square$

Remark 3.4

Note that the values for the $h_i$ in Proposition 3.3 are given by $h_{P_d}(u_i)$, that is the values of the support function of $P_d$ at the support vector $u_i$ of the mixed area measure $S_{P_1,\ldots ,P_{d-1}}$. This fact can be used to determine whether a given d-tuple $(P_1,\ldots ,P_d) \in (\mathcal {P}({\mathbb {Z}}^d))^d$ is ${\mathbb {Z}}$-maximal in $P_d$ by checking whether the equality $P_d = {{\,\mathrm{conv}\,}}{\{x\in {\mathbb {Z}}^d : \langle u_i,x\rangle \le h_{P_d}(u_i),\, i\in [r]\}}$ holds.

A similar argument provides the corresponding statement for tuples $(P_1,\ldots , P_{d})\in (\mathcal {P}({\mathbb {R}}^d))^d$ that are ${\mathbb {R}}$-maximal in $P_d$.

Proposition 3.5

Let $(P_1,\ldots , P_{d})\in (\mathcal {P}({\mathbb {R}}^d))^d$ be an irreducible tuple that is ${\mathbb {R}}$-maximal in $P_d$. Let $\{\xi _1,\ldots ,\xi _r\}$ be the support of the mixed area measure $s_{P_1,\ldots ,P_{d-1}}$. Then

$$\begin{aligned} P_d=\{x\in {\mathbb {R}}^d : \langle \xi _i,x\rangle \le h_i \ \text {for all} \ i\in [r]\} \end{aligned}$$

for some $h_1,\ldots ,h_r\in {\mathbb {R}}_{\ge 0}$ satisfying

$$\begin{aligned} \sum _{i=1}^rh_is_{P_1,\ldots ,P_{d-1}}(\xi _i)=v(P_1,\ldots ,P_{d}). \end{aligned}$$

For ${\mathbb {R}}$-maximality we also have the following simple criterion based on comparing the mixed area measures. Since the mixed area measures of d-tuples of polytopes can be computed using (2.7), Proposition 3.6 gives a computational test for ${\mathbb {R}}$-maximality of d-tuples which we utilize in our algorithms.

Proposition 3.6

Let $(P_1,\ldots ,P_d)\in \mathcal {P}({\mathbb {R}}^d)^d$ be an irreducible tuple and assume that $P_d$ is d-dimensional. Then $(P_1,\ldots ,P_d)$ is ${\mathbb {R}}$-maximal in $P_d$ if and only if ${\text {supp}}s_{P_d} \subseteq {\text {supp}}s_{P_1,\ldots ,P_{d-1}}$.

Proof

To simplify notation, let $s_1={\text {supp}}s_{P_d}$ and $s_2={\text {supp}}s_{P_1,\ldots ,P_{d-1}}$. Suppose $s_1\subseteq s_2$. Consider a polytope $Q_d\in \mathcal {P}_d({\mathbb {R}}^d)$ such that $P_d\subseteq Q_d$ and

$$\begin{aligned} v(P_1,\ldots ,P_{d-1},P_d)=v(P_1,\ldots , P_{d-1},Q_d). \end{aligned}$$

By (2.8) we have $h_{P_d}(\xi )=h_{Q_d}(\xi )$ for every $\xi \in s_2$. Then

$$\begin{aligned} Q_d\subseteq \bigcap _{\xi \in s_2}\{x\in {\mathbb {R}}^d : \langle \xi ,x\rangle \le h_{P_d}(\xi )\}\subseteq \bigcap _{\xi \in s_1}\{x\in {\mathbb {R}}^d : \langle \xi ,x\rangle \le h_{P_d}(\xi )\}=P_d, \end{aligned}$$

since $s_1$ coincides with the set of outer facet normals of $P_d$. (Here we use the assumption that $P_d$ is d-dimensional.) Therefore $Q_d=P_d$.

Conversely, let $(P_1,\ldots ,P_d)$ be ${\mathbb {R}}$-maximal in $P_d$. Consider

$$\begin{aligned} Q_d=\bigcap _{\xi \in s_2}\{x\in {\mathbb {R}}^d : \langle \xi ,x\rangle \le h_{P_d}(\xi )\}. \end{aligned}$$

Since $(P_1,\ldots ,P_d)$ is irreducible, by Proposition 2.5, $s_2$ positively spans ${\mathbb {R}}^d$, hence, $Q_d$ defines a polytope. Clearly, $P_d\subseteq Q_d$ and $h_{P_d}(\xi )=h_{Q_d}(\xi )$ for every $\xi \in s_2$. Therefore, by (2.8), we have

$$\begin{aligned} v(P_1,\ldots , P_{d-1},P_d)=v(P_1,\ldots , P_{d-1},Q_d). \end{aligned}$$

This implies that $P_d=Q_d$, and, hence, the set $s_1$ of primitive facet normals of $P_d$ is contained in $s_2$. $\square$

In the remainder of the section we show that the two constructions appearing in parts (2) and (3) of Theorem 1.8 produce ${\mathbb {R}}$-maximal triples. Clearly, the triples in part (1) are also ${\mathbb {R}}$-maximal, and one can use Propositions 3.3 and 3.5 to see that the triples in part (4) are ${\mathbb {Z}}$-maximal, but not ${\mathbb {R}}$-maximal.

Proposition 3.7

Consider polytopes $P_1,P_2,P_3 \in \mathcal {P}_3({\mathbb {R}}^3)$ such that $P_1 = P + \alpha I$, $P_2 = P + \beta I$, and $P_3 = P + \gamma I$, where $P \in \mathcal {P}({\mathbb {R}}^3)$, $\dim P\ge 2$, $I \in \mathcal {P}_1({\mathbb {R}}^3)$, and $\alpha ,\beta ,\gamma$ are non-negative real numbers at most one of which is 0. Then the triple $(P_1,P_2,P_3)$ is ${\mathbb {R}}$-maximal.

Proof

We verify that the triple $(P_1,P_2,P_3)$ is ${\mathbb {R}}$-maximal in $P_3$ using Proposition 3.6. One has

$$\begin{aligned} s_{P_1,P_2} = s_{P+\alpha I, P + \beta I} = s_{P} + (\alpha +\beta ) s_{P,I}, \end{aligned}$$

using the linearity, $s_{P,P}=s_P$, and the fact that the measure $s_I=s_{I,I}$ is zero. Furthermore, one similarly obtains

$$\begin{aligned} s_{P_3} = s_{P_3,P_3} = s_P + 2 \gamma s_{P,I}. \end{aligned}$$

As $\alpha ,\beta ,\gamma$ are nonnegative integers at most one of which can be 0, this shows ${\text {supp}}s_{P_3} \subseteq {\text {supp}}s_{P_1,P_2}$. Showing ${\mathbb {R}}$-maximality in $P_1$ and $P_2$ is completely analogous. $\square$

Proposition 3.8

Let $P' \in \mathcal {P}_3({\mathbb {R}}^3)$ be a combinatorial pyramid with base P and let I be a segment parallel to two edges of P. Then the triple $(P',P',P'+ I)$ is ${\mathbb {R}}$-maximal.

Proof

We are going to show that $P'$ and $P' + I$ have the same sets of facet normals and, hence, the conditions of Proposition 3.6 are satisfied. Let $P'={{\,\mathrm{conv}\,}}{(P\cup \{v\})}$ for some $v\in {\mathbb {R}}^3$ and assume $I=[0,w]$ for some $w\in {\mathbb {R}}^3$. Let $J_1$ and $J_2$ be two edges of P parallel to I. Note that each edge of $P+I$ is equal to either an edge of P, or the sum of w and an edge of P, or $J_i+ I$ for $i=1,2$. Similarly, each facet of $P'+I$ is equal to either a facet of $P'$, or the sum of w and a facet of P, or ${{\,\mathrm{conv}\,}}{((J_i+I)\cup (v+I))}$, for $i=1,2$. This implies $P'$ and $P' + I$ have the same facet normals. $\square$

4 Tests for Equivalence of Tuples of Polytopes Modulo Group Actions

As we deal with enumeration up to equivalences with respect to action of various groups as described in Sect. 2.2, we need to be able to decide algorithmically whether two lattice polytopes are equivalent with respect to a given group action. More generally, we need tests for equivalence of tuples of lattice polytopes modulo group actions. In this section we present such tests for the groups of linear and affine unimodular transformations and the group $\mathbb {G}_{d,k}$.

4.1 Equivalence of Polytopes Modulo ${\text {GL}}({\mathbb {Z}}^d)$

The literature contains several algorithms that test equivalence of two lattice polytopes $P,Q \in \mathcal {P}_d({\mathbb {Z}}^d)$ modulo ${\text {GL}}({\mathbb {Z}}^d)$. See, for example, [21] and [18], where the latter also provides an overview of existing techniques. The algorithm of Kreuzer and Skarke from [21], relying on the so-called normal form of a lattice polytope, was implemented in Sagemath. The normal form of a lattice polytope P is uniquely determined by P. It encodes a sequence of vertices $(v_1,\ldots ,v_t)$ of a polytope ${{\,\mathrm{conv}\,}}{(v_1,\ldots ,v_t)}$ that coincides with P up to ${\text {GL}}({\mathbb {Z}}^d)$. Two polytopes $P,Q\in \mathcal {P}_d({\mathbb {Z}}^d)$ coincide up to ${\text {GL}}({\mathbb {Z}}^d)$ if and only if their normal forms are the same. See also [18, Exam. 3.4]. Using the normal form, each polytope in $\mathcal {P}_d({\mathbb {Z}}^d)$ can be brought into a normal ${\text {GL}}({\mathbb {Z}}^d)$ -position. In other words, in each equivalence class from $\mathcal {P}_d({\mathbb {Z}}^d)/{\text {GL}}({\mathbb {Z}}^d)$ a unique representative is chosen. Using such a normal position in enumeration algorithms is convenient because, for avoiding repetitions modulo ${\text {GL}}({\mathbb {Z}}^d)$, it suffices to bring each newly found polytope into its normal position.

4.2 Equivalence of Polytopes Modulo ${\text {Aff}}({\mathbb {Z}}^d)$

Testing equivalence of two polytopes $P, Q \in \mathcal {P}_d({\mathbb {Z}}^d)$ modulo ${\text {Aff}}({\mathbb {Z}}^d)$ can be reduced to testing equivalence modulo ${\text {GL}}({\mathbb {Z}}^{d+1})$. Indeed, $P, Q \in \mathcal {P}_d({\mathbb {Z}}^d)$ are equivalent modulo ${\text {Aff}}({\mathbb {Z}}^d)$ if and only if the respective pyramids ${{\,\mathrm{conv}\,}}{(\{0_{d+1}\}\cup (P\times \{1\}))},{{\,\mathrm{conv}\,}}{(\{0_{d+1}\}\cup (Q\times \{1\}))} \in \mathcal {P}_{d+1}({\mathbb {Z}}^{d+1})$ are equivalent modulo ${\text {GL}}({\mathbb {Z}}^{d+1})$. This approach works well in many cases, but for the purpose of implementation it is more convenient to introduce a normal ${\text {Aff}}({\mathbb {Z}}^d)$-position of polytopes in $\mathcal {P}_d({\mathbb {Z}}^d)$ by choosing a representative in each of the equivalence classes modulo ${\text {Aff}}({\mathbb {Z}}^d)$.

Our construction is as follows. For a polytope $P \in \mathcal {P}_d({\mathbb {R}}^d)$, consider

$$\begin{aligned} c_P:= \frac{1}{|{{\text {vert}}(P)}|} \sum _{v \in {\text {vert}}(P)}\!v, \end{aligned}$$

which is the barycenter of the set of vertices of P. Furthermore, we can order points of ${\mathbb {R}}^d$ lexicographically: $x=(x_1,\ldots ,x_d)$ is lexicographically smaller than $y=(y_1,\ldots ,y_d)$ if, for the smallest $i\in [d]$ with $x_i\ne y_i$, one has $x_i<y_i$. For a compact subset X of ${\mathbb {R}}^d$, let ${\text {lexmin}}(X)$ denote the lexicographic minimum of the set X. It is not hard to see that for a polytope $P \in \mathcal {P}({\mathbb {R}}^d)$ one has ${\text {lexmin}}(P) \in {\text {vert}}(P)$. This can be shown by observing that (in notation of Sect. 2.3) ${\text {lexmin}}(P)$ is obtained by first taking the face $P^{e_1}$ of P, then the face $(P^{e_1})^{e_2}$ of $P^{e_1}$, and so on. Continuing this iteration we eventually arrive at the unique point $(\,{\ldots }(P^{e_1})^{e_2}{\ldots }\,)^{e_d} = {\text {lexmin}}(P)$. In particular, if $P \in \mathcal {P}({\mathbb {Z}}^d)$ is a lattice polytope then ${\text {lexmin}}(P)$ is a lattice point. Based on these notions we can now introduce a normal ${\text {Aff}}({\mathbb {Z}}^d)$-position.

Proposition 4.1

Let $P \in \mathcal {P}_d({\mathbb {Z}}^d)$ be a polytope with N vertices and let $P'$ be the normal ${\text {GL}}({\mathbb {Z}}^d)$-position of the lattice polytope $N (P- c_P)$. Then the lattice polytope

$$\begin{aligned} P'' := \frac{1}{N}P' - {\text {lexmin}}(P')) \end{aligned}$$

is ${\text {Aff}}({\mathbb {Z}}^d)$-equivalent to P.

Proof

Let $\phi \in {\text {GL}}({\mathbb {Z}}^d)$ be a linear unimodular transformation sending $P'$ to $N(P-c_P)$. Using the fact that, for any compact set $X \subset {\mathbb {R}}^d$, one has ${\text {lexmin}}{(X - x)} = {\text {lexmin}}(X)-x$ and ${\text {lexmin}}(kX)=k{\text {lexmin}}(X)$ for all $x \in {\mathbb {R}}^d$ and $k \in {\mathbb {R}}_{\ge 0}$, one obtains:

$$\begin{aligned} \phi (P'')&= \frac{1}{N}\phi (P') - \frac{1}{N} \phi ({\text {lexmin}}(P'))\\&= P - c_P - \phi ({\text {lexmin}}(\phi ^{-1}(P-c_P)))= P - \phi ({\text {lexmin}}(\phi ^{-1}(P))). \end{aligned}$$

As $\phi ({\text {lexmin}}{(\phi ^{-1}(P))})$ is a lattice point, this proves the claim. $\square$

The polytope $P''$ in Proposition 4.1 is uniquely determined by P. We call $P''$ the normal ${\text {Aff}}({\mathbb {Z}}^d)$-position of P. Note that two lattice polytopes are equivalent modulo ${\text {Aff}}({\mathbb {Z}}^d)$ if and only if their normal ${\text {Aff}}({\mathbb {Z}}^d)$-positions coincide.

Remark 4.2

Grinis and Kasprzyk [18, Sect. 3.3] suggest to use an affine normal form, obtained by slightly modifying the definition of a normal form of a lattice polytope. They also observe in [18, Sect. 2.4] that $P,Q \in \mathcal {P}_d({\mathbb {Z}}^d)$ coincide modulo ${\text {Aff}}({\mathbb {Z}}^d)$ if and only if $c_P - c_Q \in {\mathbb {Z}}^d$ and $P - c_P$ and $Q - c_Q$ are equivalent modulo ${\text {GL}}({\mathbb {Z}}^d)$. This observation is related to our Proposition 4.1.

4.3 Equivalence of k-Tuples of d-Dimensional Polytopes Modulo $\mathbb {G}_{d,k}$

To test equivalence of k-tuples $(P_1,\ldots ,P_k)$ of lattice polytopes modulo $\mathbb {G}_{d,k}$, we use the so-called Cayley polytope of $(P_1,\ldots ,P_k)$, which is defined as

$$\begin{aligned} {\text {Cay}}{(P_1,\ldots ,P_k)}:= {{\,\mathrm{conv}\,}}{(P_1 \times \{e_1\} \cup \ldots \cup P_k \times \{e_k\})} \in \mathcal {P}({\mathbb {Z}}^{d}\times {\mathbb {Z}}^k). \end{aligned}$$

The following statement shows how equivalence of tuples of lattice polytopes modulo $\mathbb {G}_{d,k}$ can be checked using the Cayley polytope construction.

Proposition 4.3

Let $(P_1,\ldots ,P_k)$, $(Q_1,\ldots ,Q_k) \in \mathcal {P}({\mathbb {Z}}^d)^k$ be k-tuples of non-empty lattice polytopes such that $\dim {(P_1 + \cdots + P_k)}=d$. Then the following conditions are equivalent:

• (i) $(P_1,\ldots ,P_k)\equiv (Q_1,\ldots ,Q_k)$ mod $\mathbb {G}_{d,k}$,

• (ii) ${\text {Cay}}{(2 P_1,\ldots , 2 P_k)}\equiv {\text {Cay}}{(2 Q_1, \ldots , 2 Q_k)}$ mod ${\text {GL}}({\mathbb {Z}}^{d+k})$.

Proof

Let $P:={\text {Cay}}{(2 P_1,\ldots , 2 P_k)}$ and $Q:= {\text {Cay}}{( 2 Q_1,\ldots , 2 Q_k)}$. The implication (i) $\Rightarrow$ (ii) is straightforward as translating one of the polytopes $P_i$ corresponds to unimodular shearing the Cayley polytope along a coordinate hyperplane and applying a common linear unimodular transformation $\phi :{\mathbb {R}}^d \rightarrow {\mathbb {R}}^d$ to each $P_i$ corresponds to applying the linear unimodular transformation $\phi \times {{\,\mathrm{Id}\,}}_k :{\mathbb {R}}^d \times {\mathbb {R}}^k \rightarrow {\mathbb {R}}^d \times {\mathbb {R}}^k$ to the Cayley polytope. Finally, applying a permutation $\sigma$ to the $P_i$ corresponds to permuting the last k coordinates of the Cayley polytope. All of these transformations are in ${\text {GL}}({\mathbb {Z}}^{d+k})$.

Let us show the converse implication (ii) $\Rightarrow$ (i). For a lattice polytope F, consider the following property $(*)$: for all $a,b \in {\text {vert}}(F)$, the point $(a+b)/2$ is a lattice point. This property is invariant under affine unimodular transformations. It is easy to see that a face F of P has property $(*)$ if and only if $F \subseteq (2 P_i) \times \{e_i\}$ for some $i \in [k]$. In particular, $(2 P_i) \times \{e_i\}$ are inclusion-maximal faces of P that have property $(*)$. Similarly, $(2 Q_i) \times \{e_i\}$ are inclusion-maximal faces of Q that have property $(*)$. We thus see that a linear unimodular transformation $\phi$ that sends P to Q sends each $(2 P_i) \times \{e_i\}$, with $i \in [k]$, to some $(2 Q_{\sigma (i)}) \times \{e_{\sigma (i)}\}$, where $\sigma$ is a permutation on [k]. Let $\phi :{\mathbb {R}}^d \times {\mathbb {R}}^k \rightarrow {\mathbb {R}}^d \times {\mathbb {R}}^k$ be a linear unimodular transformation mapping P to Q. Without loss of generality we may assume that $(2 P_i) \times \{e_i\}$ is mapped onto $(2 Q_i) \times \{e_i\}$ and that $0 \in 2P_i$ (in particular, $\phi (0,e_i)=(t_i,e_i)$ for some $t_i \in {\mathbb {R}}^d$) for all $i \in [k]$. As $\phi$ maps $(2P_1 + \cdots + 2P_k) \times \{e_1 + \cdots + e_k\}$ onto $(2Q_1 + \cdots + 2Q_k) \times \{e_1 + \cdots + e_k\}$ and $\dim {(2P_1 + \cdots + 2P_k)}=d$ by assumption, the map $\phi$ preserves the affine subspace ${\mathbb {R}}^d \times \{e_1 + \cdots + e_k\} \subset {\mathbb {R}}^d \times {\mathbb {R}}^k$. One easily verifies that this implies that the linear subspace ${\mathbb {R}}^d \times \{0\} \subset {\mathbb {R}}^d \times {\mathbb {R}}^k$ is also preserved. Hence, with respect to the standard basis of ${\mathbb {R}}^{d+k}$, the map $\phi$ has the form

$$\begin{aligned} \begin{pmatrix} U &{} T \\ 0 &{} {{\,\mathrm{Id}\,}}_k \end{pmatrix} \in {\text {GL}}({\mathbb {Z}}^{d+k}) \end{aligned}$$

for a unimodular matrix $U \in {\text {GL}}({\mathbb {Z}}^d)$ and a matrix $T \in {\mathbb {Z}}^{d \times k}$ whose i-th column equals $t_i$ for $i \in [k]$. In particular, for any $i \in [k]$, the map $x \mapsto Ux + t_i$ is an affine unimodular transformation mapping $P_i$ onto $Q_i$, which proves the claim. $\square$

Remark 4.4

Considering the Cayley polytopes of the second dilates of $P_1,\ldots ,P_k\in \mathcal {P}({\mathbb {Z}}^d)$ is crucial. Consider for example the polygons $\Delta _2$ and $\square _2=[0,1]^2$. Based on these polygons we construct two pairs of 3-dimensional polytopes $P_1=P_2={{\,\mathrm{conv}\,}}{((\Delta _2\times \{0\})\cup (\square _2\times \{1\}))}$ and $Q_1=\Delta _2 \times [0,1]$, $Q_2=\square _2\times [0,1]$. The pairs $(P_1,P_2)$ and $(Q_1,Q_2)$ are clearly not $\mathbb {G}_{3,2}$-equivalent but ${\text {Cay}}{(P_1,P_2)}$ and ${\text {Cay}}{(Q_1,Q_2)}$ are equivalent.

5 The Enumeration Algorithm for Full-Dimensional Polytopes in Dimension Three

In this section we present an algorithm for solving the following enumeration problem. Throughout this and subsequent sections the word “maximal” will mean “${\mathbb {Z}}$-maximal”.

Enumeration Problem 5.1

Given $m \in {\mathbb {N}}$, enumerate, up to $\mathbb {G}_{3,3}$-equivalence, all maximal triples $(P_1,P_2,P_3)\in \mathcal {P}_3({\mathbb {Z}}^3)^3$ of full-dimensional lattice polytopes satisfying $V(P_1,P_2,P_3) = m$.

As we mentioned in the introduction, the main idea for solving Enumeration Problem 5.1 is to produce upper bounds for the mixed volumes $V(P_i,P_j,P_k)$ for all choices $i,j,k \in [3]$ instead of an upper bound for the volume of the Minkowski sum $P_1+P_2+P_3$. As an illustration, consider the triple $(P_1,P_2,P_3)=(\Delta _3,\Delta _3,m \Delta _3)$ which has mixed volume m. While the volumes of $P_3$ and the Minkowski sum $P_1+P_2+P_3$ are large, one has $V(P_1,P_1,P_2)=1$, i.e., some of the mixed volumes $V(P_i,P_j,P_k)$ are small. Such relations are enforced by the Aleksandrov–Fenchel inequality. Proposition 5.2 below characterizes this phenomenon in general.

Proposition 5.2

Let $(P_1,P_2,P_3) \in \mathcal {P}_3({\mathbb {Z}}^3)^3$ satisfy $V(P_1,P_2,P_3)=m$ for a given $m \in {\mathbb {N}}$. Then, up to relabeling, either $V(P_1,P_1,P_2)<m$ or $V(P_i,P_i,P_j)=m$ for all $i,j \in [3]$ with $i \ne j$. In the latter case $\mathrm{Vol}\,P_i\le m$ for all $i \in [3]$.

Proof

Suppose there are $i,j \in [3]$ with $i \ne j$ such that $V(P_i,P_i,P_j) \ne m$. After possibly reordering we may assume $V(P_1,P_1,P_2) \ne m$. If $V(P_1,P_1,P_2)$ is strictly smaller than m, we have proven the claim. So let us assume $V(P_1,P_1,P_2)>m$. In this case, by the Aleksandrov–Fenchel inequality $V(P_1,P_2,P_3)^2 \ge V(P_1,P_1,P_2)V(P_3,P_3,P_2)$, we have $V(P_2,P_3,P_3)<m$ so that the claim holds for the ordering $(P_3,P_2,P_1)$. It is left to prove that, if $V(P_i,P_i,P_j)=m$ for all $i,j \in [3]$ with $i \ne j$, then $\mathrm{Vol}\,P_i\le m$ for all $i \in [3]$. This is a direct consequence of the Aleksandrov–Fenchel inequality, as $V(P_i,P_i,P_j)^2 \ge V(P_i,P_j,P_j)\mathrm{Vol}\,P_i$ holds for any $i,j \in [3]$ with $i \ne j$. $\square$

Remark 5.3

Note that in the case $V(P_i,P_i,P_j)=m$ for all $i,j \in [3]$ with $i \ne j$ and $\mathrm{Vol}\,P_i=m$ for all $i \in [3]$, the Aleksandrov–Fenchel inequalities $V(P_i,P_i,P_j) \ge V(P_i,P_j,P_j)\mathrm{Vol}\,P_i$ become equalities, which implies that $P_1=P_2=P_3$. This is due to the characterization of the equality case in Minkowski’s inequality, see [25, Thm. 7.2.1].

Let us now present an algorithm to solve Enumeration Problem 5.1. Note that Proposition 5.2 justifies the restriction in Step 2 to a case a. relying on an inductive enumeration of maximal triples of lower mixed volume (see Step 1) and a very specific case b.

Algorithm 5.4

(Enumeration of full-dimensional triples)

• Input: $m \in {\mathbb {N}}$.

• Output: A list of all maximal triples of full-dimensional polytopes $(P_1,P_2,P_3) \in \mathcal {P}_3({\mathbb {Z}}^3)^3$ with $V(P_1,P_2,P_3)=m$, up to $\mathbb {G}_{3,3}$-equivalence.

• Step 1: If $m=1$, return the triple $(\Delta _3,\Delta _3,\Delta _3)$. Else, recursively run Algorithm 5.4 for all input values $m' < m$ in order to obtain a list of all maximal triples of full-dimensional polytopes $(P_1,P_2,P_3) \in \mathcal {P}_3({\mathbb {Z}}^3)^3$ with $V(P_1,P_2,P_3) < m$, up to $\mathbb {G}_{3,3}$-equivalence.

• Step 2:

  • a. Enumerate, up to $\mathbb {G}_{3,2}$-equivalence, all pairs $(P_1,P_2) \in \mathcal {P}_3({\mathbb {Z}}^3)^2$ such that $V(P_1,P_1,P_2)<m$ (see Remark 5.5).

  • b. Enumerate, up to $\mathbb {G}_{3,2}$-equivalence, all pairs $(P_1,P_2) \in \mathcal {P}_3({\mathbb {Z}}^3)^2$ such that $V(P_1,P_1,P_2)=V(P_2,P_2,P_1)=m$ (see Algorithm 5.6).

• Step 3: For a given pair $(P_1,P_2) \in \mathcal {P}_3({\mathbb {Z}}^3)^2$ as in Step 2a enumerate, up to translations, all $P_3 \in \mathcal {P}_3({\mathbb {Z}}^3)$ such that $V(P_1,P_2,P_3)=m$ and such that the triple $(P_1,P_2,P_3)$ is maximal in $P_3$ (see Algorithm 7.2).

• Step 4: Given a triple $(P_1,P_2,P_3) \in \mathcal {P}_3({\mathbb {Z}}^3)^3$ as in Step 2b check whether it is maximal in $P_1$ and $P_2$ and, if so, add it modulo $\mathbb {G}_{3,3}$-equivalence to the final list of maximal triples of mixed volume m.

Remark 5.5

(De-maximization procedure)We may obtain the pairs of Step 2a from the list of all maximal triples of full-dimensional lattice polytopes $(P_1,P_2,P_3) \in \mathcal {P}_3({\mathbb {Z}}^3)^3$ with $V(P_1,P_2,P_3) < m$, as recursively obtained in Step 1. Note that we need to consider not only those pairs $(P_1,P_2) \in \mathcal {P}_3({\mathbb {Z}}^3)^2$ for which the triple $(P_1,P_1,P_2)$ is maximal, but all pairs such that $V(P_1,P_1,P_2)=m'<m$. These can be obtained by iteratively pealing off vertices of the maximal triples of mixed volume less than m and searching among them for triples of the form $(P_1,P_1,P_2)$ up to permutations and translations. The running time of this task is very reasonable for values $m' \in \{1,2,3\}$ but is growing very fast in $m'$ and would also be growing extensively if we were to consider higher dimensions.

Dealing with Step 2b is more involved and, hence, we employ the following algorithm:

Algorithm 5.6

(Step 2b of Algorithm 5.4)

• Input: $m \in {\mathbb {N}}$.

• Output: A list of all pairs $(P_1,P_2) \in \mathcal {P}_3({\mathbb {Z}}^3)^2$ such that $V(P_1,P_1,P_2) = V(P_1,P_2,P_2) = m$, up to $\mathbb {G}_{3,2}$-equivalence.

• Step 1: Enumerate, up to ${\text {Aff}}({\mathbb {Z}}^3)$-equivalence, all $P_1 \in \mathcal {P}_3({\mathbb {Z}}^3)$ with $\mathrm{Vol}\,P_1\le m$ (Enumeration Problem 7.7).

• Step 2: Given $P_1 \in \mathcal {P}_3({\mathbb {Z}}^3)$ with $\mathrm{Vol}\,P_1\le m$, determine, up to translations, all $Q \in \mathcal {P}_3({\mathbb {Z}}^3)$ such that $V(P_1,P_1,Q)=m$ and the triple $(P_1,P_1,Q)$ is maximal in Q (see Algorithm 7.2).

• Step 3: Given a pair $(P_1,Q) \in \mathcal {P}_3({\mathbb {Z}}^3)^2$ as in Step 2, determine all subpolytopes $P_2 \subseteq Q$ such that $\mathrm{Vol}\,P_2\le m$ and $V(P_1,P_1,P_2)=V(P_2,P_2,P_1)=m$ (see Sect. 7.4).

• Step 4: Given a pair $(P_1,P_2) \in \mathcal {P}_3({\mathbb {Z}}^3)^2$ with $P_2 \subseteq Q$ as above, add it modulo $\mathbb {G}_{3,2}$-equivalence to the final list.

6 Extension to the General Case in Dimension Three

In this section we present an extension of Algorithm 5.4 allowing us to enumerate general maximal irreducible triples $(P_1,P_2,P_3)\in P({\mathbb {Z}}^3)^3$ without assuming that all $P_i$ are full-dimensional, thus solving the following enumeration problem.

Enumeration Problem 6.1

Given $m \in {\mathbb {N}}$, enumerate, up to $\mathbb {G}_{3,3}$-equivalence, all maximal irreducible triples $(P_1,P_2,P_3)\in \mathcal {P}({\mathbb {Z}}^3)^3$ satisfying $V(P_1,P_2,P_3) = m$.

While we may still formulate a statement analogous to Proposition 5.2, the existence of $i,j \in [3]$ with $i \ne j$ such that $V(P_i,P_i,P_j)<m$ does not necessarily allow us to build upon the enumeration for lower mixed volumes as in Remark 5.5. The reason is that, if one has $\dim P_i=2$, the triple $(P_i,P_i,P_j)$ is not irreducible anymore and therefore may not be contained in one of the maximal irreducible triples of lower mixed volume. Hence, we carry out a different case distinction for triples that contain at least one polytope of dimension 2. For a 2-dimensional lattice polytope $P \in \mathcal {P}({\mathbb {Z}}^3)$, let $\mathrm{Vol}\,_2P$ denote the normalized 2-dimensional volume relative to the affine lattice ${{\,\mathrm{aff}\,}}(P)\cap {\mathbb {Z}}^3$.

Proposition 6.2

Let $(P_1,P_2,P_3) \in \mathcal {P}({\mathbb {Z}}^3)^3$ be irreducible, $V(P_1,P_2,P_3)=m$, and at least one of the $P_i$ be 2-dimensional. Then there exist distinct indices $i,j \in [3]$ such that one of the following holds:

• (a) $(P_i,P_i,P_j)$ is irreducible and satisfies $V(P_i,P_i,P_j)<m$,

• (b) $\dim P_i=2$, $\dim P_j=3$, $V(P_i,P_i,P_j) \le m$, and $V(P_j,P_j,P_i)=m$,

• (c) $\dim P_i=\dim P_j=2$, $V(P_i,P_i,P_j) \le m$, and $V(P_j,P_j,P_i) \le m^2$.

Proof

Without loss of generality we may assume $\dim P_1\le \dim P_2\le \dim P_3$. We distinguish cases based on the dimensions of the polytopes in the triple. Assume first $\dim P_1=2$ and $\dim P_2=\dim P_3=3$. Consider the Aleksandrov–Fenchel inequality $m^2=V(P_1,P_2,P_3)^2\ge V(P_2,P_2,P_1)V(P_3,P_3,P_1)$. If $V(P_2,P_2,P_1)<m$ or $V(P_3,P_3,P_1)<m$, one has (a) for $(i,j)=(2,1)$ or $(i,j)=(3,1)$, respectively. Otherwise, one has $V(P_2,P_2,P_1)=V(P_3,P_3,P_1)=m$. Now, if (a) does not hold for $(i,j)=(2,3)$ then $V(P_2,P_2,P_3)\ge m$ and the Aleksandrov–Fenchel inequality $m^2\ge V(P_1,P_1,P_3)V(P_2,P_2,P_3)$ implies $V(P_1,P_1,P_3)\le m$, i.e., (b) holds for $(i,j)=(1,3)$. Similarly, we show that either (a) holds for $(i,j)=(3,2)$ or (b) holds for $(i,j)=(1,2)$.

Assume now that $\dim P_1\!=\!\dim P_2\!=\!2$ and $\dim P_3\!=\!3$. Consider the Aleksandrov–Fenchel inequality $m^2 \ge V(P_2,P_2,P_1) V(P_3,P_3,P_1)$. If $V(P_3,P_3,P_1)<m$, then (a) holds for $(i,j)=(3,1)$. Otherwise, one has $V(P_2,P_2,P_1)\le m$. As, additionally, $V(P_1,P_1,P_2)V(P_3,P_3,P_2) \le m^2$, in this case (c) holds for $(i,j)=(2,1)$.

Let us finally assume that $\dim P_1=\dim P_2=\dim P_3=2$. Then the inequality $m^2 \ge V(P_1,P_1,P_2)V(P_3,P_3,P_2)$ yields that either $V(P_1,P_1,P_2) \le m$ or $V(P_3,P_3,P_2) \le m$. Analogously to the above, one also has $V(P_2,P_2,P_1) \le m^2$ and $V(P_2,P_2,P_3) \le m^2$. Therefore, case (c) holds for $(i,j)=(1,2)$ or $(i,j)=(3,2)$.

$\square$

Let us now present an extension of Algorithm 5.4 that allows us to solve Enumeration Problem 6.1. In particular, Algorithm 6.3 is used to enumerate maximal irreducible triples of a given mixed volume containing at least one 2-dimensional lattice polytope. Note that Proposition 6.2 justifies the restriction to the three cases a., b., and c. in Step 2.

Algorithm 6.3

(Extension of Algorithm 5.4 to general maximal irreducible triples)

• Input: $m \in {\mathbb {N}}$.

• Output: A list of all maximal irreducible triples $(P_1,P_2,P_3) \in \mathcal {P}({\mathbb {Z}}^3)^3$ with $V(P_1,P_2,P_3) = m$ and $\dim P_i=2$ for at least one $i\in [3]$, up to $\mathbb {G}_{3,3}$-equivalence.

• Step 1: If $m=1$, return an empty list (as the only maximal irreducible triple of mixed volume 1 is $(\Delta _3,\Delta _3,\Delta _3)$ and therefore full-dimensional). Else, recursively run Algorithms 5.4 and 6.3 for input values $m' < m$ in order to obtain a list of all maximal irreducible triples $(P_1,P_2,P_3) \in \mathcal {P}({\mathbb {Z}}^3)^3$ with $V(P_1,P_2,P_3) < m$, up to $\mathbb {G}_{3,3}$-equivalence.

• Step 2:

  • a. Enumerate, up to $\mathbb {G}_{3,2}$-equivalence, all pairs $(P_1,P_2) \in \mathcal {P}({\mathbb {Z}}^3)^2$ such that the triple $(P_1,P_1,P_2)$ is irreducible with $V(P_1,P_1,P_2)<m$ (see Remark 6.4).

  • b. Enumerate, up to $\mathbb {G}_{3,2}$-equivalence, all pairs $(P_1,P_2) \in \mathcal {P}({\mathbb {Z}}^3)^2$ with $\dim P_1=2$, $\dim P_2=3$, $V(P_1,P_1,P_2) \le m$, and $V(P_2,P_2,P_1)=m$ (see Algorithm 6.5).

  • c. Enumerate, up to $\mathbb {G}_{3,2}$-equivalence, all pairs $(P_1,P_2) \in \mathcal {P}({\mathbb {Z}}^3)^2$ where $\dim P_1=\dim P_2=2$, $V(P_1,P_1,P_2) \le m$, and $V(P_1,P_2,P_2) \le m^2$ (see Algorithm 6.5).

• Step 3: For a given pair $(P_1,P_2) \in \mathcal {P}({\mathbb {Z}}^3)^2$ as in Step 2, enumerate, up to translations, all $P_3 \in \mathcal {P}({\mathbb {Z}}^3)$ such that $V(P_1,P_2,P_3)=m$ and the triple $(P_1,P_2,P_3)$ is irreducible and maximal in $P_3$ (see Algorithm 7.2).

• Step 4: Given an irreducible triple $(P_1,P_2,P_3) \in \mathcal {P}({\mathbb {Z}}^3)^3$ as in Step 3, check whether it is also maximal in $P_1$ and $P_2$ and, if so, add it modulo $\mathbb {G}_{3,3}$-equivalence to the final list of maximal triples of mixed volume m.

Remark 6.4

The enumeration in Step 2a can be obtained from the list of maximal triples of mixed volume $m'<m$ of Step 1 analogously to the procedure described in Remark 5.5.

In order to treat Steps 2b and 2c of Algorithm 6.3 we apply the following algorithm. While we treat both cases in a similar way, the separation in some of the steps has an important computational advantage. This is because it allows us to have relatively small bounding boxes in which one has to perform a rather expensive search for full-dimensional subpolytopes (Step 2b), while we may restrict the search inside larger bounding boxes to lower-dimensional subpolytopes (Step 2c).

Algorithm 6.5

(Steps 2b and 2c of Algorithm 6.3)

• Input: $m \in {\mathbb {N}}$.

• Output:

  • for b: A list of all pairs $(P_1,P_2) \in \mathcal {P}({\mathbb {Z}}^3)^2$, up to $\mathbb {G}_{3,2}$-equivalence, with $\dim P_1=2$, $\dim P_2=3$, $V(P_1,P_1,P_2) \le m$, and $V(P_2,P_2,P_1)=m$.

  • for c: A list of all pairs $(P_1,P_2) \in \mathcal {P}({\mathbb {Z}}^3)^2$, up to $\mathbb {G}_{3,2}$-equivalence, with $\dim P_1=\dim P_2=2$, $V(P_1,P_1,P_2) \le m$, and $V(P_2,P_2,P_1) \le m^2$.

• Step 1: Enumerate all $P_1:=P'_1 \times \{0\} \in \mathcal {P}({\mathbb {Z}}^3)$ for $P'_1 \in \mathcal {P}_2({\mathbb {Z}}^2)$ with $\mathrm{Vol}\,_2P_1\le m$ up to equivalence (Enumeration Problem 7.7).

• Step 2:

  • for b: Given $P_1$ as above determine bounding boxes $Q_1,\ldots ,Q_r \in \mathcal {P}_3({\mathbb {Z}}^3)$ containing, up to shearing along the affine span of $P_1$ and translations, all $P_2$ that satisfy $V(P_1,P_1,P_2) \le m$ and $V(P_1,P_2,P_2)=m$ (see Lemma 7.5).

  • for c: Given $P_1$ as above determine bounding boxes $R_1,\ldots ,R_s \in \mathcal {P}_3({\mathbb {Z}}^3)$ containing, up to shearing along the affine span of $P_1$ and translations, all $P_2$ satisfying $V(P_1,P_1,P_2) \le m$ and $V(P_1,P_2,P_2) \le m^2$ (see Lemma 7.5).

• Step 3:

  • for b: Determine all subpolytopes $P_2 \in \mathcal {P}_3({\mathbb {Z}}^3)$ of the bounding boxes $Q_1,\ldots ,Q_r$ that satisfy $V(P_1,P_1,P_2) \le m$ and $V(P_1,P_2,P_2)=m$ (see Remark 7.6 and Algorithm 7.12).

  • for c: Determine all subpolytopes $P_2 \in \mathcal {P}_2({\mathbb {Z}}^3)$ of the bounding boxes $R_1,\ldots ,R_s$ that satisfy $V(P_1,P_1,P_2) \le m$ and $V(P_1,P_2,P_2) \le m^2$ (see Remark 7.6 and Algorithm 7.12).

• Step 4: Given $P_1$ and $P_2$ as above add the pair $(P_1,P_2) \in \mathcal {P}({\mathbb {Z}}^3)^2$ modulo $\mathbb {G}_{3,2}$-equivalence to the final list.

7 Details of the Enumeration Algorithms

In this section we provide further details of the enumeration algorithms presented in the previous sections.

7.1 Finding Maximal $P_3$

A problem that we have to solve in various steps of the enumeration algorithm is the following.

Enumeration Problem 7.1

Let $m \in {\mathbb {N}}$ and let $(P_1,P_2) \in \mathcal {P}({\mathbb {Z}}^3)^2$ be a pair of lattice polytopes satisfying $\dim P_1,\dim P_2\ge 2$, and $\dim {(P_1+P_2)}=3$. Enumerate, up to translations, all lattice polytopes $P_3 \in \mathcal {P}({\mathbb {Z}}^3)$ such that $V(P_1,P_2,P_3)=m$ and such that the triple $(P_1,P_2,P_3)$ is irreducible and ${\mathbb {Z}}$-maximal in $P_3$.

We solve this enumeration problem by making use of Proposition 3.3.

Algorithm 7.2

(Finding maximal $P_3$)

• Input: A pair $(P_1,P_2) \in \mathcal {P}({\mathbb {Z}}^3)^2$ with $\dim P_1,\dim P_2\ge 2$, and $\dim {(P_1+P_2)}=3$, and a number $m \in {\mathbb {N}}$.

• Output: A list of all lattice polytopes $P_3 \in \mathcal {P}({\mathbb {Z}}^3)$, up to translations, such that $V(P_1,P_2,P_3)=m$ and the triple $(P_1,P_2,P_3)$ is ${\mathbb {Z}}$-maximal in $P_3$.

• Step 1: Compute the mixed area measure of $P_1$ and $P_2$. That is, compute the normalized mixed areas $V(P_1^{u},P_2^{u})$ for all $u \in {\mathbb {Z}}^3$ that are primitive outer normal vectors of a facet of the Minkowski sum $P_1 + P_2$. Obtain a vector $a = (a_1,\ldots ,a_r) \in {\mathbb {N}}^r$ of the mixed areas of $P_1,P_2$ with respect to those primitive normal vectors $u_1,\ldots ,u_r \in {\mathbb {Z}}^3$ that yield a positive mixed area.

• Step 2: Determine all vectors $h = (h_1,\ldots ,h_r) \in ({\mathbb {Z}}_{\ge 0})^r$ with $\sum _{i=1}^r h_ia_i = m$.

• Step 3: Given a vector $h \in ({\mathbb {Z}}_{\ge 0})^r$ as above, compute $P_3:={{\,\mathrm{conv}\,}}\{x \in {\mathbb {Z}}^3 : \langle u_i , x \rangle \le h_i\ \text {for all}\ i \in [r]\}$ and check whether the triple $(P_1,P_2,P_3)$ is irreducible and satisfies $V(P_1,P_2,P_3)=m$. If it does, append $P_3$ modulo translations to the final list.

Remark 7.3

Algorithm 7.2 allows us to profit from the restriction to maximal triples (or triples that are maximal in at least one polytope) in our enumeration. For example, fixing the pair $(\Delta _3,\Delta _3) \in \mathcal {P}_3({\mathbb {Z}}^3)^2$ and mixed volume $m=4$, Algorithm 7.2 directly determines $Q = 4 \Delta _3$ as the unique maximal lattice polytope such that $V(\Delta _3,\Delta _3,Q)=4$.

Remark 7.4

A slight modification of Algorithm 7.2 can be used in order to enumerate maximal pairs of polygons $(P_1,P_2) \in \mathcal {P}_2({\mathbb {Z}}^2)$ of a given mixed volume m. By the Aleksandrov–Fenchel inequality in the two-dimensional case one may assume without loss of generality that $\mathrm{Vol}\,P_1\le m$. For any fixed $P_1 \in \mathcal {P}_2({\mathbb {Z}}^2)$ one may compute the area measure and, analogously to Steps 2 and 3 of Algorithm 7.2, determine a list of all $P_2 \in \mathcal {P}_2({\mathbb {Z}}^2)$ such that $(P_1,P_2)$ is ${\mathbb {Z}}$-maximal in $P_2$ and $V(P_1,P_2)=m$.

7.2 Bounding $P_2$ Given a Lower-Dimensional $P_1$

In the lemma below $A-A$ denotes the difference set $A+(-A)$ of a convex set $A\subset {\mathbb {R}}^d$ and $A^*=\{y\in {\mathbb {R}}^d : \langle x,y\rangle \le 1\ \text {for all}\ x\in A\}$ denotes the polar dual convex set.

Lemma 7.5

Let $(P_1,P_2) \in \mathcal {P}({\mathbb {Z}}^3)^2$ be a pair of lattice polytopes such that $P_1$ is 2-dimensional of the form $P_1 = P' \times \{0\} \subset {\mathbb {R}}^2 \times \{0\}$, and $P_2$ has dimension at least 2 and positive width w in the direction of $e_3$. Assume $V(P_1,P_1,P_2)=m_1$ and $V(P_2,P_2,P_1) \le m_2$ for some $m_1,m_2\in {\mathbb {N}}$. Then, up to a shearing along ${\mathbb {R}}^2 \times \{0\}$ and a lattice translation, $P_2$ is contained in the bounding polytope

$$\begin{aligned} R_{q_1,q_2} :={{\,\mathrm{conv}\,}}\left( \left\{ \begin{pmatrix} x_1 \\ x_2 \\ x_3 \end{pmatrix} \in {\mathbb {Z}}^2 \times \{0,\ldots ,w-1\}\,:\,\begin{pmatrix} x_1 \\ x_2\end{pmatrix}\in Q' + \frac{1}{w}\begin{pmatrix}q_1 x_3\\ q_2 x_3\end{pmatrix}\right\} \right) , \end{aligned}$$

where $q_1, q_2 \in \{0,\ldots ,w-1\}$ and

$$\begin{aligned} Q' :=\begin{pmatrix} 0 &{} {m_2}/{w} \\ -{m_2}/{w} &{} 0 \end{pmatrix} (P'-P')^{*}. \end{aligned}$$

Proof

We may assume $0 \in P_2$ and $h_{P_2}(-e_3)=0$, and therefore Proposition 2.3 yields $m_1 = V(P_1,P_1,P_2) = h_{P_2}(e_3)\mathrm{Vol}\,_2P'$. Then $P_2$ contains a lattice point $(q_1,q_2,w)$ at height w and, up to shearing, we may assume that $q_1,q_2 \!\in \! \{0,\ldots ,w-1\}$. Let $x = (x_1,x_2,x_3) \in P_2 \cap {\mathbb {Z}}^3$ be another lattice point of $P_2$ and consider the triangle $T_x := {{\,\mathrm{conv}\,}}{((0,0,0),(q_1,q_2,w),(x_1,x_2,x_3))} \subseteq P_2$. Let $(n_1,n_2,n_3) = (q_1,q_2,w) \times (x_1,x_2,x_3)$ be the normal vector to ${{\,\mathrm{aff}\,}}(T_x)$ with lattice length equal to $\mathrm{Vol}\,_2T_x$. Then (2.10) yields

$$\begin{aligned} V(T_x,T_x,P_1)&= h_{P_1}((n_1,n_2,n_3)) + h_{P_1}(-(n_1,n_2,n_3))\\&= h_{P'}((n_1,n_2)) + h_{P'}(-(n_1,n_2))= h_{P'-P'}((n_1,n_2)). \end{aligned}$$

Explicit computation of the cross product yields $(n_1,n_2)= (-wx_2,wx_1) + (q_2 x_3, -q_1 x_3)$. By the monotonicity of the mixed volume we have $m_2 \ge V(P_2,P_2,P_1) \ge V(T_x,T_x,P_1)$ and, hence,

$$\begin{aligned} (-wx_2,wx_1) + (q_2 x_3, -q_1 x_3)= (n_1,n_2) \in m_2 (P'-P')^{*}. \end{aligned}$$

This is equivalent to

$$\begin{aligned} \begin{pmatrix} x_1\\ x_2 \end{pmatrix}\in \begin{pmatrix} 0 &{} {m_2}/{w} \\ -{m_2}/{w} &{} 0 \end{pmatrix}(P'-P')^{*} + \frac{1}{w}\begin{pmatrix} q_1 x_3 \\ q_2 x_3\end{pmatrix}, \end{aligned}$$

which shows the assertion. $\square$

Remark 7.6

Note that, in the setting of Lemma 7.5, the bounding box $R_{q_1,q_2}$ is actually constructed under the assumption that $P_2$ contains the segment $I_{q_1,q_2} = {{\,\mathrm{conv}\,}}{((0,0,0),(q_1,q_2,w))}$. Therefore in order to enumerate the set of all suitable $P_2$ we may restrict to searching for all $q_1,q_2 \in \{0,\ldots ,w-1\}$ for lattice polytopes inside $R_{q_1,q_2}$ that contain $I_{q_1,q_2}$. We use this fact when we apply Algorithm 7.12. Also note that any lattice polytope $P \in \mathcal {P}({\mathbb {Z}}^3)$ with $I_{q_1,q_2} \subset P\subseteq R_{q_1,q_2}$ satisfies $V(P_1,P_1,P) = m_1$ by construction of $R_{q_1,q_2}$, while the upper bound of $m_2$ on $V(P,P,P_1)$ may in general not be satisfied for some subpolytope $P \subseteq R_{q_1,q_2}$.

7.3 Enumeration of Lattice Polytopes by Volume Using Sandwich Factory Algorithm

Enumeration Problem 7.7

Given $m \in {\mathbb {N}}$ and $d \in {\mathbb {N}}$, enumerate up to affine unimodular transformations all polytopes $P \in \mathcal {P}_d({\mathbb {Z}}^d)$ with $1 \le \mathrm{Vol}\,P\le m$.

7.3.1 Sandwich-Factory Based Approach

We present a relatively simple algorithm to Enumeration Problem 7.7 which we also found to lead to very reasonable running times. The running time of the Sagemath [28] implementation of our algorithm on a regular desktop computer was just a few minutes for $d=3$ and $m=4$. For $d=3$, $m=6$ our implementation terminates within an hour. For $d=2$, much larger values of m can be handled with an hour time limit. Even more important in the context of our original enumeration problem about mixed volumes is the fact that we use our algorithm for solving Enumeration Problem 7.7 as a template for solving further similar enumeration problems by appropriately modifying the basic steps of the algorithm.

We also refer to [5] for an alternative approach to enumeration of lattice polytopes by their volume. Note also that [16] provides an explicit description of lattice polytopes of arbitrary dimension d with the normalized volume at most 4.

We call a pair (A, B) of polytopes $A,B \in \mathcal {P}_d({\mathbb {Z}}^d)$ a sandwich if A is a subset of B. The basic principle of our algorithm is to capture all possible polytopes in a set of sandwiches (A, B). If for $P \in \mathcal {P}_d({\mathbb {Z}}^d)$ the inclusion $A \subseteq P \subseteq B$ holds, we say that P occurs in the sandwich (A, B). The algorithm maintains a sandwich factory, which is a set of sandwiches with the property that each P in question occurs in some of the sandwiches from the set. We call the difference $\mathrm{Vol}\,B-\mathrm{Vol}\,A$ the volume gap of a sandwich (A, B). The algorithm starts with a sandwich factory containing sandwiches with a large volume gap. In the main part, the algorithm iteratively replaces sandwiches with a large volume gap by sandwiches with a smaller volume gap. Eventually, only sandwiches with volume gap 0 remain; such sandwiches correspond to polytopes P with $\mathrm{Vol}\,P\le m$. Thus, as soon as there are no sandwiches with positive volume gap, the enumeration task is completed.

7.3.2 Initialization of the Sandwich Factory

It is clear that every lattice polytope $P \in \mathcal {P}_d({\mathbb {Z}}^d)$ contains an empty lattice simplex A, that is a simplex with exactly $d+1$ lattice points. Also, if $\mathrm{Vol}\,P\le m$ then, clearly, $\mathrm{Vol}\,A\le m$. Thus, we start with a set of sandwiches (A, B) which involves all possible empty simplices A with $1 \le \mathrm{Vol}\,A\le m$. In dimension $d=2$ there is only one empty simplex up to ${\text {Aff}}({\mathbb {Z}}^2)$-equivalence, namely, the triangle $A=\Delta _2$. In dimension $d=3$, by White’s classification (see [27] or [24, Thm. 5]), every empty 3-dimensional simplex is ${\text {Aff}}({\mathbb {Z}}^3)$-equivalent to either the standard simplex $\Delta _3$ or ${{\,\mathrm{conv}\,}}{(0, e_1, e_3, e_3 + a e_1 + b e_2)}$ with $a,b \in {\mathbb {N}}$, $a<b$, and ${\text {gcd}}(a,b)=1$. Thus, it suffices to determine such simplices A with $1 \le \mathrm{Vol}\,A\le m$. To complete the initialization of the sandwich factory, one needs to choose an appropriate B for each A so that (A, B) is a sandwich, which contains all lattice polytopes P with $1 \le \mathrm{Vol}\,P\le m$ and the property $A \subseteq P$. It is intuitively clear that if a point x is far away from A, then the volume ${{\,\mathrm{conv}\,}}{(A \cup \{x\})}$ must be large. This informal idea is expressed explicitly in the following lemma.

Lemma 7.8

Let A be a d-dimensional simplex and let $m \ge \mathrm{Vol}\,A$. Then

$$\begin{aligned} \{x \in {\mathbb {R}}^d: \mathrm{Vol}\,{({{\,\mathrm{conv}\,}}{(A \cup \{x\})})} \le m\}\subseteq \lambda A + (1-\lambda )c_A, \end{aligned}$$

where $c_A$ is the barycenter of A and $\lambda = (d+1) ({m}/({\mathrm{Vol}\,A}) -1) + 1$.

Proof

The proof for $d=3$ can be found in [3, Lem. 13]. The proof extends directly to the case of an arbitrary dimension $d \in {\mathbb {N}}$. $\square$

In view of Lemma 7.8, one can fix B to be the integral hull of $\lambda A + (1-\lambda )c_A$, that is

$$\begin{aligned} B={{\,\mathrm{conv}\,}}{ ( (\lambda A + (1-\lambda )c_A ) \cap {\mathbb {Z}}^d)}. \end{aligned}$$

It may still be the case that B is chosen too large in the sense that B may contain vertices v with the property that $\mathrm{Vol}\,{({\text {conv}}(A \cup \{v\}))} > m$. Clearly, if a polytope $P \in \mathcal {P}_d({\mathbb {Z}}^d)$ occurs in (A, B) and has the property $\mathrm{Vol}\,P\le m$ then P cannot contain v as above. This means that one can iteratively make B smaller by removing vertices v as above, as long as such vertices exist. More precisely, while v as above exists, one iteratively replaces B by ${{\,\mathrm{conv}\,}}{((B \cap {\mathbb {Z}}^d) \setminus \{v\})}$. We call this procedure the reduction of B relative to A. Having carried out the above reduction of B for each A, we complete the initialization of the sandwich factory.

7.3.3 Iterative Updates of Sandwich Factory

The purpose of the iterative procedure is to reduce the maximum volume gap occurring in a sandwich factory. That is, as long as there are sandwiches with positive volume gap, one considers those sandwiches (A, B) in the sandwich factory, for which the volume gap $\mathrm{Vol}\,B- \mathrm{Vol}\,A$ is maximized. For each such sandwich (A, B) one picks a vertex v of B not belonging to A. Every polytope $P \in \mathcal {P}_d({\mathbb {Z}}^d)$ with $\mathrm{Vol}\,P\le m$ occurring in (A, B) may or may not contain v. If P contains v, we can enclose P into the sandwich $({\text {conv}}(A \cup \{v\}),B)$ with a smaller volume gap. If P does not contain v, we can enclose P into the sandwich $(A,{\text {conv}}((B\cap {\mathbb {Z}}^d) \setminus \{v\}))$, whose volume gap is also smaller. Thus, we can remove the sandwich (A, B) from the factory and replace it by two other sandwiches (see also Fig. 8).

Fig. 8

Replacing a sandwich by two other sandwiches with a smaller volume gap

Here it should also be noticed that, when we let A grow, by considering $({\text {conv}}(A \cup \{v\}), B)$, we can make B smaller. Indeed, B may contain vertices w with the property that $\mathrm{Vol}\,{({\text {conv}}(A \cup \{v,w\}))} > m$. Then rather than adding the sandwich $({\text {conv}}(A \cup \{v\}), B)$, we first reduce B relative to ${{\,\mathrm{conv}\,}}(A \cup \{v\})$, to a potentially smaller polytope $B'$ and then add $({\text {conv}}(A \cup \{v\}), B')$ to the sandwich factory.

7.3.4 Equivalent Sandwiches

While the above algorithmic steps can already be used for finding all polytopes $P \in \mathcal {P}_d({\mathbb {Z}}^d)$ with $\mathrm{Vol}\,P\le m$, its efficiency would not be very good as one would generate many polytopes that are equivalent up to affine unimodular transformation. When for two sandwiches (A, B) and $(A',B')$ there exists an affine unimodular transformation $\phi$ with $\phi (A)=A'$ and $\phi (B) = B'$, then, up to affine unimodular transformations, the lattice polytopes occurring in (A, B) also occur in $(A',B')$ and vice versa. We call such sandwiches (A, B) and $(A',B')$ equivalent. Thus, if a sandwich (A, B) is already present in the sandwich factory, there is no need to add $(A',B')$. Based on this idea, we add a new sandwich (A, B) to the sandwich factory only if the factory does not already contain a sandwich equivalent to (A, B). The test for equivalence of sandwiches can be reduced to the test for equivalence of lattice polytopes as follows. If (A, B) is a sandwich then, by embedding 3A into ${\mathbb {R}}^{d}\times {\mathbb {R}}$ at heights 1 and $-1$ and 3B at height 0, we obtain the polytope

$$\begin{aligned} P_{A,B} \,:=\, {{\,\mathrm{conv}\,}}{\bigl ( \underbrace{(3 A ) \times \{1\}}_{\text {height} \ 1} \cup \underbrace{(3 B) \times \{0\}}_{\text {height} \ 0} \cup \underbrace{(3 A) \times \{-1\}}_{\text {height} \ -1} \bigr )} \,\in \, \mathcal {P}_{d+1}({\mathbb {Z}}^{d}\times {\mathbb {Z}}), \end{aligned}$$

see also Fig. 9.

Fig. 9

An example of a sandwich (A, B) in dimension two (left) and the three-dimensional lattice polytope $P_{A,B}$ assigned to this sandwich (right), whose affine normal form is used to distinguish sandwiches up to affine unimodular transformations

Lemma 7.9

Two sandwiches (A, B) and $(A',B')$ are equivalent if and only if the polytopes $P_{A,B}$ and $P_{A',B'}$ are equivalent up to affine unimodular transformations.

Proof

The first implication is direct. If (A, B) and $(A',B')$ are equivalent, then there exists an affine unimodular transformation $\phi \in {\text {Aff}}({\mathbb {Z}}^d)$ such that $\phi (A)=A'$ and $\phi (B)=B'$. The map $\phi \times {{\,\mathrm{Id}\,}}\in {\text {Aff}}{({\mathbb {Z}}^d \times {\mathbb {Z}})}$ then satisfies $(\phi \times {{\,\mathrm{Id}\,}})(P_{A,B}) = P_{A',B'}$. In order to show the reverse implication assume that $P_{A,B}$ and $P_{A',B'}$ are equivalent and let $\psi \in {\text {Aff}}{({\mathbb {Z}}^d \times {\mathbb {Z}})}$ be an affine unimodular transformation such that $\psi (P_{A,B})=P_{A',B'}$. Note that for both $P=P_{A,B}$ and $P=P_{A',B'}$ the vector $v = e_{d+1}$ is the unique direction such that $|h_P(v)-h_P(-v)| = 2$. Therefore $\psi$ maps the intersection of $P_{A,B}$ with any of the hyperplanes ${\mathbb {R}}^d \times \{-1\}$, ${\mathbb {R}}^d \times \{0\}$, and ${\mathbb {R}}^d \times \{1\}$ to the intersection of $P_{A',B'}$ with the respective hyperplane. Here we use that, as $P_{A,B}$ is symmetric with respect to the hyperplane ${\mathbb {R}}^d \times \{0\}$, we may assume that the intersections with the hyperplanes ${\mathbb {R}}^d \times \{-1\}$ and ${\mathbb {R}}^d \times \{1\}$ are not permuted by $\psi$. In particular, $\psi (3B \times \{0\}) = 3B'\times \{0\}$ and, as B is full-dimensional, $\psi ({\mathbb {R}}^d \times \{h\}) = {\mathbb {R}}^d \times \{h\}$ for any $h \in {\mathbb {R}}$. Furthermore, we may assume that $\psi$ is linear and, hence, with respect to the standard basis of ${\mathbb {R}}^{d+1}$ to be of the form

$$\begin{aligned} \begin{pmatrix} U &{} t \\ 0 &{} 1 \end{pmatrix} \in {\text {GL}}({\mathbb {Z}}^{d+1}), \end{aligned}$$

for a unimodular matrix $U \in {\text {GL}}({\mathbb {Z}}^d)$ and an integer vector $t \in {\mathbb {Z}}^d$. Denote by $\phi \in {\text {GL}}({\mathbb {Z}}^d)$ the linear unimodular transformation corresponding to U. Then $\psi (3A \times \{-1\}) = (\phi (3A)-t) \times \{-1\}$ and $\psi (3A \times \{1\}) = (\phi (3A)+t) \times \{1\}$. In particular, $\phi (3A)-t = 3A' = \phi (3A)+t$ and therefore one has $t=0$. So $\phi$ is a unimodular transformation such that $\phi (A) = A'$ and $\phi (B) = B'$, and hence the sandwiches (A, B) and $(A',B')$ are equivalent. $\square$

7.3.5 Summary of the Sandwich-Factory Algorithm

We give a complete description of the algorithm we have developed above.

Algorithm 7.10

(Sandwich-factory algorithm)

• Input: Dimension $d \in {\mathbb {N}}$ and volume bound $m \in {\mathbb {N}}$.

• Output: A list of all full-dimensional lattice polytopes $P \in \mathcal {P}_d({\mathbb {Z}}^d)$ with $\mathrm{Vol}\,P\le m$, up to affine unimodular transformations.

• Step 1: Enumerate, up to affine unimodular transformations, all empty lattice simplices A with $\mathrm{Vol}\,A \le m$.

  • For each A as above, choose B to be the integral hull ${{\,\mathrm{conv}\,}}( (\lambda A + (1-\lambda )c_A ) \cap {\mathbb {Z}}^d)$, where $c_A$ is the barycenter of A and

    $$\begin{aligned} \lambda = (d+1) \biggl ( \frac{m}{\mathrm{Vol}\,A} - 1 \biggr ) + 1 \end{aligned}$$

    and then reduce B relative to A.

  • Initialize the sandwich factory $\mathcal {F}$ with all pairs (A, B) obtained as above.

• Step 2: While $\mathcal {F}$ contains sandwiches with a positive volume gap, carry out the following steps for sandwiches (A, B) whose volume gap is maximized:

  • pick a vertex v of B, not contained in A,

  • fix $A' := {{\,\mathrm{conv}\,}}{(A \cup \{v\})}$,

  • compute the reduction $B'$ of B relative to $A'$,

  • fix $B'' := {{\,\mathrm{conv}\,}}{((B \cap {\mathbb {Z}}^d) \setminus \{v\})}$,

  • add $(A',B')$ to $\mathcal {F}$, unless $\mathcal {F}$ already contains a sandwich equivalent to $(A',B')$,

  • add $(A,B'')$ to $\mathcal {F}$, unless $\mathcal {F}$ already contains a sandwich equivalent to $(A,B'')$.

• Step 3: In this step, all sandwiches (A, B) in $\mathcal {F}$ have the form $A=B$. Return the set of all A with $(A,B) \in \mathcal {F}$. Up to affine unimodular transformations, this is the set of all polytopes $P \in \mathcal {P}_d({\mathbb {Z}}^d)$ with $1 \le \mathrm{Vol}\,P\le m$.

Remark 7.11

In Sect. 7.3.2 we described an efficient implementation of Step 1 for dimension two and three (the dimensions we are interested in, in the context of this paper). For higher dimensions we do not specify how to implement Step 1 and only notice that it can be implemented algorithmically. We also note that rather than using empty lattice simplices of normalized volume at most m, one can start with all lattice simplices of normalized volume at most m. Such simplices can be enumerated using the Hermitian normal form (see [26, Sect. 4.1]) rather easily.

7.4 Sandwich-Type Search for Subpolytopes

In this section we describe our approach towards the task of finding all subpolytopes fulfilling certain conditions inside a given bounding polytope. For our purposes we found it computationally efficient to employ an algorithm similar to the one presented in Sect. 7.3. Our enumeration employs three different variations of this task that we solve using three different variations a., b., and c. of Algorithm 7.12. In particular, variation a. is employed for the search for full-dimensional subpolytopes inside of a maximal polytope as obtained using Algorithm 7.2, while variations b. and c. deal with the search for full-dimensional or 2-dimensional subpolytopes of a bounding polytope as in Lemma 7.5. Note that a sandwich-type search seems particularly natural for the search inside bounding polytopes as in Lemma 7.5, as by Remark 7.6 it suffices to search for those subpolytopes that contain a given segment I depending on the bounding polytope.

For a sandwich $(A,B) \in \mathcal {P}({\mathbb {Z}}^3)^2$ we call the number $|B\cap {\mathbb {Z}}^3|-|A\cap {\mathbb {Z}}^3|\ge 0$ the lattice point gap of (A, B). Furthermore, for a lattice polytope $P \in \mathcal {P}({\mathbb {Z}}^3)$ let $\mathrm{Vol}\,_r\!P$ denote the normalized k-dimensional volume relative to the affine lattice ${{\,\mathrm{aff}\,}}(P)\cap {\mathbb {Z}}^3$ where $k \in [3]$ is the dimension of P. Generalizing the concept of the reduction of a lattice polytope $B\supseteq A$ relative to A used in Algorithm 7.10, we define the reduction of B relative to A with respect to the conditions $\mathrm{Vol}\,_r\le m_1$ and $V(\,{\cdot }\,,\,{\cdot }\,,P_1) \le m_2$ to be the polytope

$$\begin{aligned} B'={{\,\mathrm{conv}\,}}{\bigl \{ x \in B \cap {\mathbb {Z}}^3 : A_x}&:={{\,\mathrm{conv}\,}}{(A \cup \{x\})}\ \text {satisfies}\\ \mathrm{Vol}\,_rA_x&\le m_1,\,V(A_x,A_x,P_1) \le m_2\bigr \}. \end{aligned}$$

Algorithm 7.12

(Sandwich approach to subpolytopes)

• Input:

  • for a: A bounding box $M \in \mathcal {P}_3({\mathbb {Z}}^3)$, a lattice polytope $P_1 \in \mathcal {P}({\mathbb {Z}}^3)$, and a bound $m \in {\mathbb {N}}$.

  • for b/c: A bounding box $M \in \mathcal {P}_3({\mathbb {Z}}^3)$, a segment $I \subset M$, a lattice polytope $P_1 \in \mathcal {P}({\mathbb {Z}}^3)$, and bounds $m_1,m_2 \in {\mathbb {N}}$.

• Output:

  • for a: A list of all full-dimensional lattice polytopes $P_2$, up to translations, with $P_2 \subseteq M$ such that $\mathrm{Vol}\,P_2\le m$ and $V(P_2,P_2,P_1) \le m$.

  • for b: A list of all full-dimensional lattice polytopes $P_2$, up to translations, with $I \subset P_2 \subseteq M$ such that $\mathrm{Vol}\,P_2\le m_1$ and $V(P_2,P_2,P_1) \le m_2$.

  • for c: A list of all 2-dimensional lattice polytopes $P_2$, up to translations, with $I \subset P_2 \subset M$ such that $\mathrm{Vol}\,_2 P_2\le m_1$ and $V(P_2,P_2,P_1) \le m_2$.

• Step 1:

  • for a: Initialize the sandwich factory $\mathcal {F}$ with pairs (S, M) where S ranges over all empty simplices in M satisfying the bounding conditions (in particular, $V(S,S,P_1) \le m$). Set $m_1=m_2=m$.

  • for b/c: Initialize the sandwich factory $\mathcal {F}$ with the pair $(I,M')$, where $M'$ is the reduction of M relative to I with respect to the conditions $\mathrm{Vol}\,_r\le m_1$ and $V(\,{\cdot }\,,\,{\cdot }\,,P_1) \le m_2$.

• Step 2: While $\mathcal {F}$ contains sandwiches with positive lattice point gap, carry out the following steps for sandwiches (A, B) having the maximal lattice point gap among the sandwiches in $\mathcal {F}$:

  • pick a vertex v of B, not contained in A,

  • fix $A' := {{\,\mathrm{conv}\,}}{(A \cup \{v\})}$ (note that, as B is reduced relative to A with respect to the conditions $\mathrm{Vol}\,_r\le m_1$ and $V(\,{\cdot }\,,\,{\cdot }\,,P_1) \le m_2$, the polytope $A'$ is ensured to satisfy the bounding conditions),

  • (for c:) if $\dim A'=2$ and $\dim B=3$, set $B := B \cap {{\,\mathrm{aff}\,}}(A')$,

  • compute the reduction $B'$ of B relative to A with respect to the conditions $\mathrm{Vol}\,_r\le m_1$ and $V(\,{\cdot }\,,\,{\cdot }\,,P) \le m_2$,

  • fix $B'' := {{\,\mathrm{conv}\,}}{((B \cap {\mathbb {Z}}^d) \setminus \{v\})}$,

  • add $(A',B')$ to $\mathcal {F}$, unless $\mathcal {F}$ already contains a translate of $(A',B')$,

  • add $(A,B'')$ to $\mathcal {F}$, unless $\mathcal {F}$ already contains a translate of $(A,B'')$,

  • remove (A, B) from $\mathcal {F}$.

• Step 3: In this step all sandwiches (A, B) have lattice point gap 0 and therefore fulfill $A=B$. Return a list of A for all sandwiches $(A,B) \in \mathcal {F}$.

Remark 7.13

While the overall structure of Algorithm 7.12 above is very similar to Algorithm 7.10, there are some differences. In Algorithm 7.12 we also work with sandwiches (A, B) for which $\dim A<\dim B$ and therefore the volume gap is not necessarily strictly decreasing in our iterative steps. We deal with this by considering the lattice point gap of a sandwich instead. Furthermore, we identify two sandwiches (A, B) and $(A',B')$ only if there is a translation vector $t \in {\mathbb {Z}}^3$ such that $(A',B') = (A+t,B+t)$. Also note that, in addition to a volume bound for $P_2$, we also have a bound for the mixed volume $V(P_2,P_2,P_1)$ and therefore perform a slightly different reduction step.