Ehrhart Positivity of Tesler Polytopes and Berline–Vergne’s Valuation

Abstract

For ${\varvec{a}} \in {\mathbb {R}}_{\ge 0}^{n}$, the Tesler polytope ${\text {Tes}}_{n}({\varvec{a}})$ is the set of upper triangular matrices with non-negative entries whose hook sum vector is ${\varvec{a}}$. Motivated by a conjecture of Morales, we study the questions of whether the coefficients of the Ehrhart polynomial of ${\text {Tes}}_n(1,1,\dots ,1)$ are positive. We attack this problem by studying a certain function constructed by Berline–Vergne and its values on faces of a unimodularly equivalent copy of ${\text {Tes}}_n(1,1,\dots ,1)$. We develop a method of obtaining the dot products appeared in formulas for computing Berline–Vergne’s function directly from facet normal vectors. Using this method together with known formulas, we are able to show Berline–Vergne’s function has positive values on codimension 2 and 3 faces of the polytopes we consider. As a consequence, we prove that the third and fourth coefficients of the Ehrhart polynomial of ${\text {Tes}}_{n}(1,\dots ,1)$ are positive. Using the Reduction Theorem by Castillo and the second author, we generalize the above result to all deformations of ${\text {Tes}}_{n}(1,\dots ,1)$ including all the integral Tesler polytopes.

1 Introduction

A subset P of ${\mathbb {R}}^{m}$ is a polyhedron if it is the intersection of finitely many half spaces, usually defined by linear inequalities. A polytope is a bounded polyhedron. Equivalently, a polytope $P \subset {\mathbb {R}}^{m}$ may be defined as the convex hull of finitely many points in ${\mathbb {R}}^{m}$. We assume that readers are familiar with the basic concepts related to polytopes, such as face and dimension, presented in [2, 30].

A polytope $P \subset {\mathbb {R}}^{m}$ is called integral if all of its vertices are integer points, i.e., points in ${\mathbb {Z}}^m$. In 1962, Ehrhart discovered that for any integral polytope P of dimension d, the function $E_{P}$ which maps any non-negative integer $t \in {\mathbb {Z}}_{\ge 0}$ to the number of integer points in tP (the t-th dilate of P) is a polynomial in t of degree d. We call $E_{P}$ the Ehrhart polynomial of P. For each $1\le i\le d$, let $e_i(P)$ be the coefficient of $t^i$ in $E_P(t)$, so

$$\begin{aligned} E_P(t)=e_{d}(P)t^{d}+e_{d-1}(P)t^{d-1}+\dots + e_0(P). \end{aligned}$$

In [12], Ehrhart showed that for any d-dimensional integral polytope P, the leading coefficient $e_{d}(P)$ of the Ehrhart polynomial $E_P(t)$ is the normalized volume of P, the second coefficient $e_{d-1}(P)$ is one half of the sum of the normalized volumes of the facets of P, and the constant term $e_0(P)$ is 1. Thus, these three coefficients are always positive. However, the remaining coefficients of $E_P(t)$ are not always positive. We say an integral polytope P is Ehrhart positive, if all the coefficients of $E_P(t)$ are positive. (See [19] for a survey on Ehrhart positivity.)

An affine transformation from an affine space $A \subseteq {\mathbb {R}}^{m}$ to an affine space $B \subseteq {\mathbb {R}}^{l}$ is a unimodular transformation if it induces a bijection from integer points in A to integer points in B. Two polytopes $P \subset {\mathbb {R}}^{m}$ and $Q \subset {\mathbb {R}}^{l}$ are said to be unimodularly equivalent if there exists a unimodular transformation $\phi$ from an affine space containing P to an affine space containing Q such that $\phi (P)=Q$. Clearly, if two polytopes are unimodularly equivalent, they have the same Ehrhart polynomial.

1.1 Tesler Polytopes and Morales Conjecture

For any ${\varvec{a}}=(a_1,\dots ,a_n) \in {\mathbb {R}}_{\ge 0}^{n}$, the Tesler polytope of hook sum ${\varvec{a}}$, denoted by ${\text {Tes}}_{n}({\varvec{a}})$, is the set of all $n\times n$ upper triangular matrices with non-negative entries such that its “hook sum vector” is ${\varvec{a}}$. (See Definition 4.1 for a complete definition.) When ${\varvec{a}}= \varvec{1}:=(1,\dots ,1) \in {\mathbb {R}}^n$, the integer points of ${\text {Tes}}_{n}({\varvec{a}})$ are called Tesler matrices, which were initially introduced by Tesler, and then rediscovered by Haglund in his work of expressing the diagonal Hilbert series as a weighted sum over these matrices [15]. Consequently, Tesler matrices play an important role in the field of diagonal harmonics [1, 13, 14, 16, 28]. Intrigued by these works, Mészáros et al. [21] defined and studied the Tesler polytopes of hook sum ${\varvec{a}}$ defined as above for ${\varvec{a}}\in {\mathbb {Z}}_{>0}^n$. In this paper, we extend the domain of ${\varvec{a}}$ to ${\mathbb {R}}_{\ge 0}^{n}$, noticing that important features remain the same. The work in this paper was initially motivated by a conjecture of Morales:

Conjecture 1.1

[22] Tesler polytopes ${\text {Tes}}_{n}(\varvec{1})$ and ${\text {Tes}}_{n}(1,0,\dots , 0)$ are both Ehrhart positive for any positive integer n.

We remark that both families of polytopes in Morales conjecture are fascinating objects both for their own interesting combinatorial properties and for their connection to other fields of mathematics. We have already discussed above the importance of ${\text {Tes}}_{n}(\varvec{1})$ in the study of diagonal harmonics. Meanwhile, it is known [8] that ${\text {Tes}}_{n}(1,0,\dots ,0)$ is unimodularly equivalent to the Chan–Robbins–Yuen (CRY) polytope, whose volume is the product of the first $n-2$ Catalan numbers [7, 29]. As of today, still no simple proofs for this surprising result is known. Furthermore, the CRY polytope is a face of the Birkhoff polyope which is also a well-studied subject. In particular, computing volumes of Birkhoff polytopes is an extremely hard problem and has attracted a lot of recent research [4, 10, 23].

1.2 McMullen’s Formula and $\alpha$-Positivity

In this paper, we will use a technique developed by Castillo and the 2nd author [6] to attack Morales’ positivity conjecture and its generalizations. The technique is based on the existence of “McMullen’s formula”. In 1975, Danilov questioned, in the context of toric varieties, whether it is possible to construct a function $\alpha$ such that for any integral polytope $P \subset {\mathbb {R}}^{m}$, the following equation holds

$$\begin{aligned}|P\cap \mathbb {Z}^{m}|\,=\!\sum _{\begin{array}{c} \text {{\textit{F}} a non-empty}\\ \text {face of }P \end{array}} \!\!\alpha (F,P){\text {nvol}}(F), \end{aligned}$$

where ${\text {nvol}}(F)$ is the normalized volume of F and $\alpha (F,P)$ only depends on the normal cone of P at F [9]. McMullen was the first to confirm that it is possible to construct such a function $\alpha$ (in a non-constructive way). Hence, we refer to the above formula as McMullen’s formula [20]. Since $\alpha$ only depends on normal cones and normal cones are invariant under dilations, we obtain the following expression for $e_i(P)$, the coefficient of $t_i$ in $E_P(t)$, as a weighted sum of $\alpha (F,P)$:

$$\begin{aligned} e_{i}(P)\,=\!\sum _{\begin{array}{c} F\text { an }i\text {-dimen-}\\ \text {sional face of }P \end{array}}\!\!\!\alpha (F,P){\text {nvol}}(F),\qquad \text { for any }\ 0 \le i \le \dim (P). \end{aligned}$$

(1.1)

One sees that, as a consequence of the above formula, if $\alpha (F,P)>0$ for every i-dimensional face F of P, then $e_{i}(P)$ is positive. Moreover, if $\alpha (F,P)>0$ for every face of P, then P is Ehrhart positive. We say a polytope P is $\alpha$-positive if $\alpha (F,P)>0$ for every face F of P.

One cannot discuss $\alpha$-positivity without fixing a construction of the $\alpha$ function for McMullen’s formula. Currently, at least three different constructions for $\alpha$ are known; they are given by Pommersheim and Thomas [24], by Berline and Vergne [3], and by Ring and Schürmann [26]. Following [6], we will use Berline–Vergne’s construction for the $\alpha$ function, and for simplicity we refer to their construction as the BV-$\alpha$ function, denoted by ${\alpha ^{\textrm{BV}}}$.

Based on the discussion above, one sees that Ehrhart positivity can be studied through $\alpha$-positivity or more specifically through BV-$\alpha$-positivity, which is the approach we will take. To simplify the calculation of BV-$\alpha$ values, we work on the “projected Tesler polytope” ${\text {PTes}}_{n}({\varvec{a}})$, which is unimodularly equivalent to ${\text {Tes}}_{n}({\varvec{a}})$ and thus has the same Ehrhart polynomial as that of ${\text {Tes}}_n({\varvec{a}})$. (See Sect.  4.2 for the definition of ${\text {PTes}}_{n}({\varvec{a}})$.) Therefore, if ${\text {PTes}}_{n}({\varvec{a}})$ is BV-$\alpha$-positive then ${\text {Tes}}_{n}({\varvec{a}})$ is Ehrhart positive. The following is one of our main results:

Theorem 1.2

Let $n \ge 3$ be a positive integer. Then ${\alpha ^{\textrm{BV}}}(F, {\text {PTes}}_n({\varvec{1}}))$ is positive for all the codimension 2 and 3 faces F of ${\text {PTes}}_{n}(\varvec{1})$. Therefore, $e_{d-2}({\text {PTes}}_{n}(\varvec{1}))$ and $e_{d-3}({\text {PTes}}_{n}(\varvec{1}))$ are positive, where $d=\left( {\begin{array}{c}n\\ 2\end{array}}\right)$ is the dimension of ${\text {PTes}}_{n}(\varvec{1})$.

One benefit of discussing BV-$\alpha$-positivity is that via the Reduction Theorem 3.4 established by Castillo and the second author, one can prove Ehrhart positivity of all deformations of a polytope P by proving that it is BV-$\alpha$-positive. (See Definition 2.5 for the definition of deformations.) Applying the Reduction Theorem together with the result that any invertible affine transformation preserves deformations (Lemma 2.6), we obtain the following result:

Corollary 1.3

Let P be a d-dimensional integral polytope that is a deformation of ${\text {Tes}}_{n}(\varvec{1})$. Then the following statements are true:

• (i) If $d = \left( {\begin{array}{c}n\\ 2\end{array}}\right)$, i.e., $\dim (P)=\dim ({\text {Tes}}_{n}(\varvec{1}))$, then $e_{d-2}(P)>0$ and $e_{d-3}(P)>0$.

• (ii) If $d = \left( {\begin{array}{c}n\\ 2\end{array}}\right) -1$, i.e., $\dim (P)=\dim ({\text {Tes}}_{n}(\varvec{1}))-1$, then $e_{d-2}(P)>0$.

It is known that ${\text {Tes}}_{n}({\varvec{a}})$ is a deformation of ${\text {Tes}}_{n}(\varvec{1})$ for all ${\varvec{a}}\in {\mathbb {R}}_{\ge 0}^{n}$, and ${\text {Tes}}_n({\varvec{a}})$ is integral for ${\varvec{a}}\in {\mathbb {Z}}_{\ge 0}^n$. Therefore, we are able to generalize the positivity result stated above to all Tesler polytopes, including ${\text {Tes}}_{n}(1,0,\dots ,0)$—the other polytope in Conjecture 1.1.

Corollary 1.4

Let ${\varvec{a}}\in {\mathbb {Z}}_{\ge 0}^n$. Then $e_{d-2}({\text {Tes}}_n({\varvec{a}}))$ and $e_{d-3}({\text {Tes}}_n({\varvec{a}}))$ are both positive (assuming these coefficients exist), where $d = \dim ({\text {Tes}}_n({\varvec{a}}))$.

Given the results in Corollary 1.3, it is natural to ask which polytopes are deformations of ${\text {Tes}}_{n}(\varvec{1})$. Please see [18, Sect. 4] for some discussion on this.

1.3 Computing BV-$\alpha$ Values

Our method of proving Theorem 1.2 is by computing ${\alpha ^{\textrm{BV}}}(F,{\text {PTes}}_n({\varvec{1}}))$ for all the codimension 2 and 3 faces F of ${\text {PTes}}_n({\varvec{1}})$ using known formulas for BV-$\alpha$ values given in Lemmas 3.1 and 3.2. One difficulty of applying these formulas to find the BV-$\alpha$ values for faces F of ${\text {PTes}}_{n}({\varvec{1}})$ (or any other polytope) is that we need to compute dot products between the generators of “pointed feasible cones” of ${\text {PTes}}_{n}({\varvec{a}})$ at F. Since these formulas are provided for pointed feasible cones that are unimodular with respect to projections of standard lattices, one has to be extremely careful to determine their generating rays.

In Sect. 3.2, we develop a method for obtaining dot products appearing in formulas similar to those in Lemmas 3.1 and 3.2, without calculating the generators of pointed feasible cones. Instead we obtain them via computing dot products between the generators of normal cones. (The key result is summarized in Corollary 3.7.) This approach simplifies our verification procedure significantly. In general, our method can be applied to any totally unimodular polytope, and it is efficient if we know the inequality descriptions of these polytopes which are often helpful for figuring out their normal cones. Hence, we expect that our result will be useful in computing BV-$\alpha$ values for other families of totally unimodular polytopes.

1.4 Organization of the Paper

In Sect. 2, we provide background on polyhedra theory. In Sect. 3, after giving a brief description for the construction of BV-$\alpha$ functions, we develop the aforementioned method that helps to calculate the BV-$\alpha$ values arising from totally unimodular polytopes. In Sect. 4, we first formally introduce Tesler polytopes and review relevant results, and then define projected Tesler polytopes ${\text {PTes}}_n({\varvec{1}})$ and discuss its properties. By applying the method developed in Sect. 3 to ${\text {PTes}}_{n}(\varvec{1})$, we obtain a proof for Theorem 1.2. Finally, we prove Corollaries 1.3 and 1.4 using the Reduction Theorem 3.4.

2 Basic Definitions and Results in Polyhedra Theory

In this section, we review terminologies and results related to polytopes/polyhedra that are relevant to this article. We assume that $\langle \,{\cdot },\,{\cdot }\,\rangle$ is the dot product on ${\mathbb {R}}^{m}$, and let $P \subset {\mathbb {R}}^m$ be a polytope. If a facet F of P contains a face G, we call F a supporting facet of G.

Given a finite set of vectors $R=\{\varvec{r}_1, \varvec{r}_2, \dots , \varvec{r}_k\} \subset {\mathbb {R}}^m$, a (polyhedral) cone $K \subset {\mathbb {R}}^m$ generated by R is

$$\begin{aligned} K :=\left\{ \varvec{x}\in {\mathbb {R}}^m:\varvec{x}= \sum _{i=1}^k c_i \varvec{r}_i, \, c_i \ge 0\right\} . \end{aligned}$$

A cone $K \subset {\mathbb {R}}^m$ can also be defined by homogeneous linear inequalities (thus it is polyhedral). A pointed cone is a cone that does not contain a line. The polar cone of a cone $K \subset {\mathbb {R}}^m$ is the cone:

$$\begin{aligned} K^{\circ }=\{\varvec{y}\in W:\langle \varvec{x},\varvec{y}\rangle \le 0, \, \forall \varvec{x}\in K\}, \end{aligned}$$

where W is the subspace of ${\mathbb {R}}^m$ spanned by K. It is a well-known result that the polar of the polar of a pointed cone is itself.

For any subset S of ${\mathbb {R}}^m$, let ${\text {lin}}(S)$ be the translation of the affine span of S to the origin, and let ${\mathbb {R}}^m\!/{{\text {lin}}(S)}$ be the orthogonal complement of ${\text {lin}}(S)$. In this paper, ${\text {lin}}(S)$ is always a rational subspace of ${\mathbb {R}}^m$, i.e., it is a subspace that can be defined by linear equalities with integer coefficients. For any subset or an element B of ${\mathbb {R}}^m$, we use $B/{{\text {lin}}(S)}$ to denote the canonical projection of B onto ${\mathbb {R}}^m\!/{{\text {lin}}(S)}$. Note that when ${\text {lin}}(S)$ is a rational subspace, ${\mathbb {Z}}^m\!/{{\text {lin}}(S)}$ is a lattice in ${\mathbb {R}}^m\!/{{\text {lin}}(S)}$.

A cone $K \subset {\mathbb {R}}^m\!/{{\text {lin}}(S)}$ is unimodular with respect to the lattice ${\mathbb {Z}}^m\!/{{\text {lin}}(S)}$ if it can be generated by a set of vectors $\{\varvec{d}_{1},\dots ,\varvec{d}_{l}\}$ that can be extended to a basis for the lattice ${\mathbb {Z}}^m\!/{{\text {lin}}(S)}$. In this case, we call $\varvec{d}_{1},\dots ,\varvec{d}_{l}$ the primitive generators for the unimodular cone K. We say K is unimodular if it is unimodular with respect to ${\mathbb {Z}}^m$. (Note that for a unimodular cone K, each of its primitive generators is a primitive vector, i.e., an integer vector with the property that the greatest common divisor of its components is 1.)

Definition 2.1

Suppose $P \subset {\mathbb {R}}^m$ is a polytope.

• For any face F of P, the feasible cone of P at F is

  $$\begin{aligned} {\text {fcone}}(F,P):=\{\varvec{y}\in {\mathbb {R}}^m:\varvec{x}+\epsilon \varvec{y}\in P~\text {for some}~\epsilon >0\}, \end{aligned}$$

  where $\varvec{x}$ is any interior point of F. (It can be shown that the definition does not depend on the choice of $\varvec{x}$.) The pointed feasible cone of P at F is

  $$\begin{aligned} {\text {fcone}}^{\varvec{p}}(F,P):={\text {fcone}}(F,P)/{{\text {lin}}(F)}. \end{aligned}$$

• For any face F of P, the normal cone of P at F is

  $$\begin{aligned} {\text {ncone}}(F, P) := \{\varvec{u}\in {\text {lin}}(P):\langle \varvec{u},\varvec{p}_{1} \rangle \ge \langle \varvec{u},\varvec{p}_{2} \rangle ,\, \forall \varvec{p}_{1} \in F,\, \forall \varvec{p}_{2} \in P \}. \end{aligned}$$

  The normal fan of P is the collection of all normal cones of P at its non-empty faces.

The following is one important connection between normal cones and pointed feasible cones:

Lemma 2.2

[5, Lemma 2.4] Suppose $P \subset {\mathbb {R}}^m$ is a polytope and F is a codimension k face of P. Then both ${\text {fcone}}^p(F,P)$ and ${\text {ncone}}(F,P)$ are full-dimensional pointed cones in the k-dimensional space ${\text {lin}}(P)/{{\text {lin}}(F)}$. Furthermore, they are polar to one another, that is,

$$\begin{aligned} {\text {ncone}}(F,P)^{\circ }={\text {fcone}}^{p}(F,P)\quad \text { and }\quad {\text {fcone}}^{p}(F,P)^\circ = {\text {ncone}}(F,P). \end{aligned}$$

(2.1)

Definition 2.1 gives one common definition for pointed feasible cones and normal cones. The following lemma provides another way of constructing/defining them, which we state without proof.

Lemma 2.3

Suppose $P \subset {\mathbb {R}}^m$ is a polytope and F is a codimension k face of P.

• (i) If ${\varvec{v}}$ is a vertex of F, then ${\text {fcone}}^p(F,P) = {\text {fcone}}({\varvec{v}},P)/{{\text {lin}}(F)}$.

• (ii) Assume the supporting facets of F are $\{F_1, \dots , F_k\}$ and for each i, let $\varvec{n}_i \in {\text {lin}}(P)$ be an outer normal vector for the facet $F_i$. Then ${\text {ncone}}(F,P)$ is generated by $\varvec{n}_1, \dots ,\varvec{n}_k$.

Definition 2.4

We say a polytope P in ${\mathbb {R}}^{m}$ is totally unimodular if it is an integral polytope and ${\text {fcone}}({\varvec{v}},P)$ is unimodular for all the vertices ${\varvec{v}}$ of P.

We remark that any totally unimodular polytope P is a simple polytope, that is, each vertex of P is contained in exactly $\dim (P)$ many facets, or equivalently is contained in exactly $\dim (P)$ many edges.

Definition 2.5

Let P and Q be two polytopes. Then Q is a deformation of P if there exists a surjective map $\phi$ from the set of the vertices of P to that of Q and $r_{i,j} \in {\mathbb {R}}_{\ge 0}$ such that $\phi ({\varvec{v}}_{i})-\phi ({\varvec{v}}_{j})=r_{i,j}({\varvec{v}}_{i}-{\varvec{v}}_{j})$ whenever ${\varvec{v}}_{i}$ and ${\varvec{v}}_{j}$ are adjacent vertices of P.

It is a classical result [25] that there is an alternative but equivalent way of defining deformations in terms of normal fans. However, it requires considering normal fans of P and Q with respect to a same underlining space, which we do not include in this paper. We finish this part with the following lemma that will be used in proving Corollary 1.3.

Lemma 2.6

Let P and Q be two polytopes and $\psi$ is an invertible affine transformation from the affine hull of Q to the affine hull of P. Then Q is a deformation of P if and only if $\psi (Q)$ is a deformation of $\psi (P)$.

Proof

For the forward direction, let ${\varvec{v}}, {\varvec{w}}$ be any pair of adjacent vertices of P and ${\varvec{v}}', {\varvec{w}}'$ the corresponding vertices of Q. Then by Definition 2.5, there exists $r \in {\mathbb {R}}_{\ge 0}$ such that ${\varvec{v}}'-{\varvec{w}}'=r({\varvec{v}}-{\varvec{w}})$. Since $\psi$ is an affine transformation, we have that $\psi ({\varvec{v}}')-\psi ({\varvec{w}}')=r(\psi ({\varvec{v}})-\psi ({\varvec{w}}))$. Therefore, $\psi (Q)$ is a deformation of $\psi (P)$. The backward direction can be proven by the same argument. $\square$

3 Computing Berline–Vergne’s $\alpha$-Construction

In the first part of this section, we provide more details on Berline–Vergne’s $\alpha$-function construction for McMullen’s formula, formulas for computing it, as well as the Reduction Theorem. In the second part, we develop an alternative method of obtaining dot products appearing in formulas for BV-$\alpha$ values, which is one of the main results of this paper. This method and the Reduction Theorem are both key ingredients in our proofs for Theorem 1.2 and Corollary 1.3 that will be presented in Sect. 4.

3.1 Berline–Vergne’s Construction

In [3], Berline and Vergne associate to every rational affine cone c an analytic function $\phi (c)$ on ${\mathbb {R}}^{m}$ which is recursively defined with respect to the dimension of cones. They show that $\phi$ is a valuation and $\phi (c)$ is analytic near the origin. Then they set ${\alpha ^{\textrm{BV}}}(F,P)$ to be the residue of $\phi ({\text {fcone}}^{p}(F,P))$ around 0, and prove that it is a valid $\alpha$-construction for McMullen’s formula.

One sees that this procedure of computing ${\alpha ^{\textrm{BV}}}$ or $\phi$ is complicated. Using the valuation property of $\phi$, one can reduce the problem of computing ${\alpha ^{\textrm{BV}}}(F,P)$ to the cases when ${\text {fcone}}^{p}(F,P)$ is unimodular with respect to ${\mathbb {Z}}^m\!/{{\text {lin}}(F)}$. However, even for these cases, simple formulas for ${\alpha ^{\textrm{BV}}}(F,P)$ are only known if the dimension of ${\text {fcone}}^{p}(F,P)$ is at most 3. Note that it [2, Exam. 19.2] follows immediately from Berline–Vergne’s construction that

$$\begin{aligned} \begin{aligned} {\alpha ^{\textrm{BV}}}(F,P)&=1,&\quad \text {when}\ \ \dim ({\text {fcone}}^{p}(F,P))&=0,\\ {\alpha ^{\textrm{BV}}}(F,P)&=\frac{1}{2},&\quad \text {when}\ \ \dim ({\text {fcone}}^{p}(F,P))&=1.\end{aligned} \end{aligned}$$

We include formulas for faces of codimensions 2 and 3 below.

Lemma 3.1

[2, Exam. 19.3] Let F be a codimension 2 face of an integral polytope $P \subset {\mathbb {R}}^m$. Suppose ${\text {fcone}}^{p}(F,P)$ is a unimodular cone with respect to the lattice ${\mathbb {Z}}^{m}\!/{{\text {lin}}(F)}$, and $\varvec{u}_1$ and $\varvec{u}_2$ are its primitive generators. Then

$$\begin{aligned} {\alpha ^{\textrm{BV}}}(F,P)=\frac{1}{4}+\frac{1}{12}\biggl (\frac{\langle \varvec{u}_1,\varvec{u}_2\rangle }{\langle \varvec{u}_1,\varvec{u}_1\rangle }+\frac{\langle \varvec{u}_1,\varvec{u}_2\rangle }{\langle \varvec{u}_2,\varvec{u}_2\rangle }\biggr ). \end{aligned}$$

Lemma 3.2

[6, Lem. 3.10] Let F be a codimension 3 face of an integral polytope $P\subset {\mathbb {R}}^m$. Suppose ${\text {fcone}}^{p}(F,P)$ is a unimodular cone with respect to the lattice ${\mathbb {Z}}^{m}\!/{{\text {lin}}(F)}$, and $\varvec{u}_1$, $\varvec{u}_2$, and $\varvec{u}_3$ are its primitive generators. Then

$$\begin{aligned} {\alpha ^{\textrm{BV}}}(F,P)=\frac{1}{8}+\frac{1}{24}{} & {} \!\!\!\biggl (\frac{\langle \varvec{u}_1,\varvec{u}_2\rangle }{\langle \varvec{u}_1,\varvec{u}_1\rangle }+\frac{\langle \varvec{u}_1,\varvec{u}_2\rangle }{\langle \varvec{u}_2,\varvec{u}_2\rangle }+\frac{\langle \varvec{u}_1,\varvec{u}_3\rangle }{\langle \varvec{u}_1,\varvec{u}_1\rangle } \\ {}{} & {} \quad +\frac{\langle \varvec{u}_1,\varvec{u}_3\rangle }{\langle \varvec{u}_3,\varvec{u}_3\rangle }+\frac{\langle \varvec{u}_2,\varvec{u}_3\rangle }{\langle \varvec{u}_2,\varvec{u}_2\rangle }+\frac{\langle \varvec{u}_2,\varvec{u}_3\rangle }{\langle \varvec{u}_3,\varvec{u}_3\rangle }\biggl ). \end{aligned}$$

Remark 3.3

When $\dim ({\text {fcone}}^{p}(F,P)) \ge 4$, the formulas for ${\alpha ^{\textrm{BV}}}(F,P)$ that one can obtain by directly applying Berline–Vergne’s algorithm become way more complicated. However, by the nature of the algorithm, these formulas are still in terms of $\langle \varvec{u}_{i}, \varvec{u}_{j} \rangle$, the dot products between primitive generators of the unimodular cone ${\text {fcone}}^p(F,P)$. Therefore, it is important to know the values of these dot products.

Despite the difficulty and complication of obtaining formulas for computing the BV-$\alpha$ values, there is one great benefit of studying the question of Ehrhart positivity via the BV-$\alpha$-positivity approach. In [6], Castillo and the second author obtained the following theorem—the Reduction Theorem—from the valuation property of $\phi$:

Reduction Theorem 3.4

Suppose P and Q are two integral polytopes in ${\mathbb {R}}^{m}$. Assume further that Q is a deformation of P. Then for any fixed k, if ${\alpha ^{\textrm{BV}}}(F,P) > 0$ for every k-dimensional face F of P, then ${\alpha ^{\textrm{BV}}}(G, Q) > 0$ for every k-dimensional face G of Q. Therefore, BV-$\alpha$-positivity of P implies BV-$\alpha$-positivity of Q.

By the Reduction Theorem, if we show that all the k-dimensional faces of an integral polytope P is BV-$\alpha$-positive (which together with (1.1) implies that the corresponding Ehrhart coefficient is positive), then the same is true for any integral deformation of P.

3.2 Computing BV-$\alpha$ Values of Totally Unimodular Polytopes

Recall that if a polytope P in ${\mathbb {R}}^{m}$ is totally unimodular, then for any vertex v of P, the feasible cone ${\text {fcone}}(v,P)$ is unimodular. As a consequence, for any face F of P, the pointed feasible cone ${\text {fcone}}^{p}(F,P)$ is unimodular with respect to the lattice ${\mathbb {Z}}^{m}\!/{{\text {lin}}(F)}$. (See Remark 3.6 below.) Thus, we can apply any known BV-$\alpha$ formulas (such as those presented in Lemmas 3.1 and 3.2) to calculate the BV-$\alpha$ values for P. In this part, we present a way to obtain the dot products $\langle \varvec{u}_i,\varvec{u}_j\rangle$ appeared in these formulas. As stated in Remark 3.3, these dot products are generally important for computing BV-$\alpha$ values, so we expect our method will be useful when formulas for computing BV-$\alpha$ values for faces of other codimensions become known.

Below is the key lemma of this section. (Recall that for an edge of P connecting two vertices ${\varvec{v}}$ and ${\varvec{w}}$, the primitive edge direction from ${\varvec{v}}$ to ${\varvec{w}}$ is the vector $\varvec{d}= a({\varvec{w}}-{\varvec{v}})$ where $a\in {\mathbb {R}}_{>0}$ is chosen so that $\varvec{d}$ is a primitive vector.)

Lemma 3.5

Let P be a totally unimodular polytope in ${\mathbb {R}}^{m}$, and assume that P is full-dimensional. Suppose F is a codimension k face of P, and its supporting facets are $F_{1},\dots ,F_{k}$ (so F is the intersection of $F_{1},\dots ,F_{k}$). Further assume that $\varvec{n}_i$ is the primitive outer normal vector of $F_{i}$ for each $1 \le i \le k$. (Hence, $\{\varvec{n}_1, \dots , \varvec{n}_k\}$ is a set of rays that generates ${\text {ncone}}(F,P)$.)

Fix a vertex ${\varvec{v}}$ of F. For each i, let ${\varvec{v}}_{i}$ be the vertex of P that is adjacent to ${\varvec{v}}$ but not in $F_{i}$, and $\varvec{d}_{i}$ the primitive edge direction from ${\varvec{v}}$ to ${\varvec{v}}_{i}$. (Note that the uniqueness of the choice of ${\varvec{v}}_i$ follows from the fact that P is simple.) Define $\varvec{u}_{i}:=\varvec{d}_{i}/{{\text {lin}}(F)}$ to be the projection of $\varvec{d}_i$ onto the orthogonal complement of ${\text {lin}}(F)$. Then the following statements are true:

• (i) $\langle \varvec{n}_{i}, \varvec{u}_{i} \rangle = -1$ for each $1 \le i \le k$.

• (ii) $\langle \varvec{n}_{j}, \varvec{u}_{i} \rangle = 0$ for each pair of distinct i and j in $\{1, 2, \dots , k\}$.

• (iii) $\{\varvec{u}_{1},\dots ,\varvec{u}_{k}\}$ generates ${\text {fcone}}^{p}(F,P)$.

• (iv) $\{\varvec{u}_{1},\dots ,\varvec{u}_{k}\}$ is a basis for the lattice ${\mathbb {Z}}^m\!/{{\text {lin}}(F)}$.

Remark 3.6

Note that conclusions (iii) and (iv) of the lemma above imply that ${\text {fcone}}^p(F,P)$ is a unimodular cone with respect to the lattice ${\mathbb {Z}}^m\!/{{\text {lin}}(F)}$, and $\varvec{u}_1, \dots , \varvec{u}_k$ are its primitive generators.

Proof of Lemma 3.5

Clearly $F_1, \dots , F_k$ are supporting facets of ${\varvec{v}}$. Since P is simple and full-dimensional in ${\mathbb {R}}^m$, the vertex ${\varvec{v}}$ is supported by m facets. Assume $F_{k+1}, \dots , F_m$ are the other $m-k$ supporting facets of v. For each $k+1 \le i \le m$, we similarly let $\varvec{n}_i$ be the primitive outer normal vector of $F_{i}$, let ${\varvec{v}}_{i}$ be the vertex of P that is adjacent to ${\varvec{v}}$ but not in $F_{i}$, and $\varvec{d}_{i}$ the primitive edge direction from ${\varvec{v}}$ to ${\varvec{v}}_{i}$. By the choices of these vectors and the fact that P is totally unimodular, we have the following:

• (a) $\{ \varvec{d}_1, \dots , \varvec{d}_m\}$ forms a basis for the lattice ${\mathbb {Z}}^m$.

• (b) $\{\varvec{d}_1,\dots , \varvec{d}_m\}$ generates ${\text {fcone}}({\varvec{v}},P) = {\text {fcone}}^p({\varvec{v}},P)$.

• (c) $\{\varvec{n}_1, \dots , \varvec{n}_m\}$ generates ${\text {ncone}}({\varvec{v}},P)$.

• (d) For each $k+1 \le j \le m$, the vertex ${\varvec{v}}_j \in F$ and thus $\varvec{d}_j \in {\text {lin}}(F)$.

By Lemma 2.3 (i), we have that ${\text {fcone}}^p(F,P) = {\text {fcone}}({\varvec{v}},P)/{{\text {lin}}(F)}$. This together with (a), (b), and (d) implies conclusions (iii) and (iv) of the lemma.

Before we prove conclusions (i) and (ii) of the lemma, we prove the following claim first: for each $1\le i\le m$,

$$\begin{aligned} (\textrm{I})\ \ \langle \varvec{n}_{\textit{i}},\varvec{d}_{\textit{i}}\rangle =-1,\qquad \quad (\textrm{II})\ \ \langle \varvec{n}_{\textit{j}},\varvec{d}_{\textit{i}} \rangle = 0~~\text {if }{\textit{j}}\ne {\textit{i}}. \end{aligned}$$

By Lemma 2.2, we have that ${\text {fcone}}^p({\varvec{v}},P)$ and ${\text {ncone}}({\varvec{v}},P)$ are polar to one another. Therefore, we immediately have (II), and that $\langle \varvec{n}_{i},\varvec{d}_{i}\rangle =-a_i$ for some positive integer $a_i >0$. However, since $\varvec{n}_i$ is primitive, by elementary number theory, there exists an integer point/vector $\varvec{x}\in {\mathbb {Z}}^{m}$ such that $\langle \varvec{n}_{i}, \varvec{x}\rangle =-1$. But by (a), the vector $\varvec{x}$ is a ${\mathbb {Z}}$-linear combination of $\varvec{d}_1, \dots , \varvec{d}_m$. This together with (II) implies that $a_i = 1$. So (I) holds.

Now let $i = 1, 2, \dots , k$. Since $\varvec{u}_i = \varvec{d}_i/{{\text {lin}}(F)}$, we have that $\varvec{u}_i - \varvec{d}_i \in {\text {lin}}(F)$ which is contained in ${\text {lin}}(F_j)$ for each $1 \le j \le k$. Hence, $\langle \varvec{n}_j, \varvec{u}_{i}-\varvec{d}_{i} \rangle = 0$, or equivalently,

$$\begin{aligned} \langle \varvec{n}_{j}, \varvec{u}_{i} \rangle = \langle \varvec{n}_{j}, \varvec{d}_{i} \rangle \quad \ \text {for any pair of }i \text { and }j \text { in }\{1, 2, \dots , k\}. \end{aligned}$$

Therefore, conclusions (i) and (ii) follow from (I) and (II). $\square$

Corollary 3.7

Suppose F, P, $\{\varvec{n}_1, \dots , \varvec{n}_k\}$, and $\{\varvec{u}_1, \dots , \varvec{u}_k\}$ are as given in Lemma 3.5. Let $C=(c_{i,j})$ be the $k \times k$ matrix whose (i, j)-th entry is $c_{i,j}=\langle \varvec{n}_{i}, \varvec{n}_{j} \rangle$ and $M=(m_{i,j})$ be the $k \times k$ matrix whose (i, j)-th entry is $m_{i,j}=\langle \varvec{u}_{i}, \varvec{u}_{j} \rangle$. Then C and M are inverse to one another.

Proof

Let $X= (\varvec{n}_1, \dots , \varvec{n}_k)$ be the $m \times k$ matrix whose columns are $\varvec{n}_i$’s, and $Y = (\varvec{u}_1, \dots , \varvec{u}_k)$ be the $m \times k$ matrix whose columns are $\varvec{u}_i$’s. Then by parts (i) and (ii) of Lemma 3.5, we have

$$\begin{aligned} X^t Y = - I. \end{aligned}$$

By Lemma 2.2, the normal cone ${\text {ncone}}(F,P)$ is full-dimensional in ${\mathbb {R}}^m\!/{{\text {lin}}(F)}$. Hence, its generating set $\{\varvec{n}_1,\dots , \varvec{n}_k\}$ is a basis for ${\mathbb {R}}^m\!/{{\text {lin}}(F)}$. Therefore, for each $1 \le j \le k$, we have

$$\begin{aligned} \varvec{u}_j = a_{1,j} \varvec{n}_1 + \dots + a_{k,j} \varvec{n}_k \end{aligned}$$

for some real numbers $a_{1,j}, \dots , a_{k,j}$. Now applying Lemma 3.5, (i) and (ii), again, we get

$$\begin{aligned} \langle \varvec{u}_{i}, \varvec{u}_{j} \rangle =\langle \varvec{u}_i, a_{1,j}\varvec{n}_{1}+\cdots + a_{i,j}\varvec{n}_i + \cdots +a_{k,j}\varvec{n}_{k}\rangle = -a_{i,j}. \end{aligned}$$

(3.1)

Hence,

$$\begin{aligned} Y\!\!{} & {} =(\varvec{u}_1, \dots , \varvec{u}_k)=(\varvec{n}_1,\dots ,\varvec{n}_k)(a_{i,j})_{1\le i,j\le k}\\ {}{} & {} =X(-\langle \varvec{u}_{i}, \varvec{u}_{j} \rangle )_{1 \le i, j\le k}=X(-M)=-XM. \end{aligned}$$

Therefore,

$$\begin{aligned} -I = X^t Y = X^t (-XM) = - (X^t X)M. \end{aligned}$$

Clearly, we have $X^t X = C$. Thus, $C M = I$. $\square$

For convenience, we call the matrix C and the matrix M appearing in the above corollary the matrix of dot products (MDP) of ${\text {ncone}}(F,P)$ and ${\text {fcone}}^p(F,P)$, respectively. Then we can restate Corollary 3.7 as the following:

Corollary 3.8

Suppose P is a totally unimodular polytope and is full-dimensional in ${\mathbb {R}}^m$, and F is a face of P. Then the MDP of ${\text {ncone}}(F,P)$ and the MDP of ${\text {fcone}}^p(F,P)$ are inverse to one another.

Remark 3.9

We remark that results given in Lemma 3.5 and its corollaries are stated for totally unimodular polytopes that are full-dimensional in ${\mathbb {R}}^m$. If a totally unimodular polytope P in ${\mathbb {R}}^m$ is not full-dimensional, one can consider a unimodularly equivalent copy of P that is full-dimensional in its ambient space and apply our results. This is the approach we will take when we discuss Tesler polytopes in the next section. We want to note that it is possible to modify the choice of generators of ${\text {ncone}}(F,P)$ in Lemma 3.5 to obtain results that work for totally unimodular polytopes that are not full-dimensional. But since the setup would be more complicated, we only include the current simpler version that we use in this article.

4 Positivity of the (Projected) Tesler Polytope of Hook Sums $(1,\dots ,1)$

In this section, we start by formally defining Tesler polytopes ${\text {Tes}}_n({\varvec{a}})$ and reviewing known results on them. We then define projected Tesler polytopes ${\text {PTes}}_n({\varvec{a}})$ and discuss their facet normal vectors. Next, we apply the method developed in Sect.  3.2 together with Lemmas 3.1 and 3.2 to show that the BV-$\alpha$ values of all the codimension 2 and 3 faces of ${\text {PTes}}_{n}(\varvec{1})$ are positive. Finally, as a consequence, we complete the proof for our main results—Theorem 1.2 and Corollaries 1.3 and 1.4. We always assume that n is a positive integer.

4.1 Background on Tesler Polytopes

In this part, we review definitions and results related to Tesler polytopes that are relevant to this paper. Majority of them are given in [21]. Let ${\mathbb {U}}(n)$ be the set of $n\times n$ upper triangular matrices. If we ignore the zeros below the diagonal, then ${\mathbb {U}}(n)\cong {\mathbb {R}}^{\genfrac(){0.0pt}1{n+1}{2}}$ as a vector space.

Definition 4.1

Let $M=(m_{i,j})\in {\mathbb {U}}(n)$. The k-th hook-sum of M is the sum of the entries on the k-th row of M minus the sum of the entries on the k-th column of M above the diagonal element, that is,

$$\begin{aligned} \eta _k(M) :=(m_{k,k}+m_{k,k+1}+\dots +m_{k,n}) - (m_{1,k}+m_{2,k}+\dots +m_{k-1,k}). \end{aligned}$$

The hook-sum vector of M is defined to be $\varvec{\eta }(M) :=(\eta _1(M), \eta _2(M),\dots , \eta _n(M))$. For any ${\varvec{a}}=(a_1,\dots ,a_n)\in {\mathbb {R}}_{\ge 0}^{n}$, let

$$\begin{aligned} \mathbb {H}_n({\varvec{a}}):=\{M \in {\mathbb {U}}(n):\varvec{\eta }(M)={\varvec{a}}\} \end{aligned}$$

(4.1)

be the affine subspace of ${\mathbb {U}}(n)$ defined by the hook sum conditions determined by ${\varvec{a}}$, and then define the Tesler polytope of hook sum ${\varvec{a}}$ to be

$$\begin{aligned} {\text {Tes}}_{n}({\varvec{a}})\,:=\, \{M=(m_{i,j}) \in \mathbb {H}_n({\varvec{a}}):m_{i,j} \ge 0\ \text {for }1 \le i \le j \le n \}. \end{aligned}$$

(4.2)

Lemma 4.2

Let ${\varvec{a}}=(a_1,a_2,\dots ,a_n) \in {\mathbb {R}}_{\ge 0}^{n}$. Suppose that the first p entries of ${\varvec{a}}$ are zero and $a_{p+1}>0$, and let ${\varvec{b}}=(a_{p+1},a_{p+2}, \dots , a_n)$, then ${\text {Tes}}_{n}({\varvec{a}})$ is isomorphic to ${\text {Tes}}_{n-p}({\varvec{b}})$, which is a non-empty polytope of dimension $\left( {\begin{array}{c}n-p\\ 2\end{array}}\right)$. Furthermore, if $a_1 > 0$, i.e., ${\varvec{a}}\in {\mathbb {R}}_{>0} \times {\mathbb {R}}_{\ge 0}^{n-1}$, then ${\text {Tes}}_n({\varvec{a}})$ is full-dimensional in $\mathbb {H}_n({\varvec{a}})$.

It was shown in [21, Cor. 2.6] that the dimension of ${\text {Tes}}_n({\varvec{a}})$ is $\left( {\begin{array}{c}n\\ 2\end{array}}\right)$ for ${\varvec{a}}\in {\mathbb {Z}}_{>0}\times {\mathbb {Z}}_{\ge 0}^{n-1}$. Even though the proof works for ${\varvec{a}}\in {\mathbb {R}}_{>0} \times {\mathbb {R}}_{\ge 0}^{n-1}$ as well, their arguments are quite involved. Hence, we provide an alternative and more straightforward proof below. We need the following result on $\mathbb {H}_n({\varvec{a}})$, which follows from elementary linear algebra.

Lemma 4.3

For any ${\varvec{a}}\in {\mathbb {R}}_{\ge 0}^n$, the affine space $\mathbb {H}_n({\varvec{a}})$ has dimension $\left( {\begin{array}{c}n\\ 2\end{array}}\right)$.

Proof of Lemma 4.2

First, ${\text {Tes}}_n({\varvec{a}})$ is clearly bounded, and hence is a polytope. Next, it is easy to check that ${\text {Tes}}_n({\varvec{a}})$ is isomrphic to ${\text {Tes}}_{n-p}({\varvec{b}})$. Given these and that ${\text {Tes}}_n({\varvec{a}})$ lives in $\mathbb {H}_n({\varvec{a}})$ (which has dimension $\left( {\begin{array}{c}n\\ 2\end{array}}\right)$ by Lemma 4.3), one sees that it suffices to show that if $a_1>0$, then ${\text {Tes}}_n({\varvec{a}})$ has dimension $\left( {\begin{array}{c}n\\ 2\end{array}}\right)$.

Assume $a_1 > 0$. We first show that ${\text {Tes}}_n({\varvec{a}})$ contains a matrix $X=(x_{i,j})$ with all positive entries by construction:

• (1) Let $c_1 = a_1/n$ and let $x_{1,j} = c_1$ for all $1 \le j \le n$.

• (2) Let $c_2 = (c_1 + a_2)/(n-1)$, and let $x_{2,j} = c_2$ for all $2 \le j \le n$.

• (3) Let $c_3 = (c_1+c_2+a_3)/(n-2)$, and let $x_{3,j} = c_3$ for all $3 \le j \le n$.

• $\dots$ $\dots \dots \dots \dots$

• (n) Let $c_n = (c_1+c_2 + \dots + c_{n-1} + a_n)/1$, and let $x_{n,n} = c_n$.

It is easy to verify that $X=(x_{i,j}) \in {\text {Tes}}_n({\varvec{a}})$ with all positive entries, as desired. Now observe that for any $M=(m_{i,j}) \in \mathbb {H}_n(0,0,\dots ,0)$ with each $|m_{i,j}|$ sufficiently small, we have that $X +M \in {\text {Tes}}_n({\varvec{a}})$. By Lemma 4.3, the vector space $\mathbb {H}_n(0,0,\dots ,0)$ has dimension $\left( {\begin{array}{c}n\\ 2\end{array}}\right)$. Hence, we conclude that ${\text {Tes}}_n({\varvec{a}})$ is of dimension $\left( {\begin{array}{c}n\\ 2\end{array}}\right)$. $\square$

For any ${\varvec{a}}\in {\mathbb {Z}}_{\ge 0}^n$, Mészáros et al. [21] gave the characterization for the face poset of ${\text {Tes}}_{n}({\varvec{a}})$ using the concept of support. Their characterization can be easily generalized to any ${\varvec{a}}\in {\mathbb {R}}_{\ge 0}^{n}$ by the same proof.

Definition 4.4

For $1 \le i \le j \le n$, let $H_{i,j}^n$ be the hyperplane consisting of upper triangular matrices whose (i, j)-th entry is 0, that is,

$$\begin{aligned} H_{i,j}^{n}\,:=\,\{M=(m_{l,k})\in {\mathbb {U}}(n):m_{i,j}=0\}. \end{aligned}$$

(4.3)

We say the intersection $H_{i_{1},j_{1}}^n\cap \cdots \cap H_{i_{k},j_{k}}^n$ does not make any zero rows if the intersection is not contained in $H_{i,i}^{n} \cap H_{i,i+1}^n\cap \cdots \cap H_{i,n}^{n}$ for every $1\le i\le n$.

Theorem 4.5

[21, Thm. 2.5] Let ${\varvec{a}}\in {\mathbb {R}}_{>0}^n$. If F is a codimension k face of ${\text {Tes}}_n({\varvec{a}})$, then F is of the form

$$\begin{aligned} {\text {Tes}}_n({\varvec{a}}) \cap H_{i_{1},j_{1}}^{n} \cap \cdots \cap H_{i_{k},j_{k}}^{n}, \end{aligned}$$

(4.4)

where $H_{i_{1},j_{1}}^{n} \cap \cdots \cap H_{i_{k},j_{k}}^{n}$ does not make any zero rows. Conversely, any such form is a codimension k face of ${\text {Tes}}_{n}({\varvec{a}})$.

We remark that when we apply Theorem 4.5, the hyperplane $H_{n,n}^n$ should never appear in the expression (4.4) as it automatically makes a zero row. Hence, ${\text {Tes}}_n({\varvec{a}}) \cap H_{n,n}^n$ is not a facet of ${\text {Tes}}_n({\varvec{a}})$. On the other hand, it is easy to see that if $(i,j) \ne (n,n)$, the intersection ${\text {Tes}}_n({\varvec{a}}) \cap H_{i,j}^n$ is a facet. Hence, we have the following result as a consequence to Theorem 4.5.

Corollary 4.6

Let ${\varvec{a}}\in {\mathbb {R}}_{>0}^n$. Then the Tesler polytope ${\text {Tes}}_n({\varvec{a}})$ has the following inequality description:

$$\begin{aligned} {\text {Tes}}_{n}({\varvec{a}}) =\{M= & {} (m_{i,j}) \in \mathbb {H}_n({\varvec{a}}): \\ {}{} & {} m_{i,j} \ge 0~\text { for}~{1 \le i < j \le n \text { or } 1 \le i = j \le n-1}\}, \end{aligned}$$

in which each inequality defines a facet.

The following lemma states well-known results about the Tesler polytopes. However, we are not able to find a direct reference for it. So we provide a sketch of its proof without introducing all the terminologies and results involved.

Lemma 4.7

Let ${\varvec{a}}\in {\mathbb {R}}_{\ge 0}^n$. Then the Tesler polytope ${\text {Tes}}_n({\varvec{a}})$ is integral if and only if ${\varvec{a}}\in {\mathbb {Z}}_{\ge 0}^n$. Furthermore, if ${\varvec{a}}\in {\mathbb {Z}}_{>0}^n$, the Tesler polytope ${\text {Tes}}_n({\varvec{a}})$ is a totally unimodular polytope.

Sketch of proof

Note that if we treat M as a $\left( {\begin{array}{c}n+1\\ 2\end{array}}\right)$-dimensional vector $\textbf{M}$, the hook sum condition $\varvec{\eta }(M) ={\varvec{a}}$ can be written as $L \textbf{M}={\varvec{a}}$, where $L=(l_{k,(i,j)})$ is an $n \times \left( {\begin{array}{c}n+1\\ 2\end{array}}\right)$ matrix obtained from the incidence matrix of a directed complete graph on n vertices by appending an identity matrix of size n. It is a classical result [27, Chap. 19] that such a matrix is totally unimodular. Hence, our first conclusion follows.

Next, for any ${\varvec{a}}\in {\mathbb {Z}}_{>0}^{n}$, it follows from [21, Thm. 1.7] that the Tesler polytope ${\text {Tes}}_{n}({\varvec{a}})$ is simple. This together with the fact that L is totally unimodular leads to the second conclusion. $\square$

4.2 Projected Tesler Polytopes and Facet Normal Vectors

We start by defining the projection map we will use in this part. For any upper triangular $X =(x_{i,j}) \in \mathbb {U}(n)$, we define $\psi _{{\text {diag}}}(X)$ to be the upper triangular matrix $Y \in \mathbb {U}(n-1)$ obtained by “erasing” the diagonal line of X shown as below:

More formally, the entries of $\psi _{{\text {diag}}}(X)=Y=(y_{i,j})$ are defined as

$$\begin{aligned} y_{i,j} = x_{i,j+1} \quad \text { for }\ 1 \le i \le j \le n-1. \end{aligned}$$

(4.5)

Now we are ready to define projected Tesler polytopes:

Definition 4.8

For any ${\varvec{a}}\in {\mathbb {R}}_{\ge 0}^n$, we call $\psi _{{\text {diag}}}({\text {Tes}}_n({\varvec{a}}))$ the projected Tesler polytope of hook sum ${\varvec{a}}$ and denote it by ${\text {PTes}}_n({\varvec{a}})$.

Lemma 4.9

Let ${\varvec{a}}=(a_{1},\dots ,a_{n}) \in {\mathbb {R}}_{\ge 0}^n$. Then the restriction map $\psi _{{\text {diag}}}|_{\mathbb {H}_n({\varvec{a}})}$ is a unimodular transformation from $\mathbb {H}_n({\varvec{a}})$ to $\mathbb {U}(n-1)$. (Recall $\mathbb {H}_n({\varvec{a}})$ is defined in (4.1).) Therefore, ${\text {PTes}}_n({\varvec{a}})$ is unimodularly equivalent to ${\text {Tes}}_n({\varvec{a}})$.

The proof of the above lemma is elementary, and thus is omitted from this paper. One can find a proof of it in [17, Thm. 3.1.1].

Recall that when ${\varvec{a}}\in {\mathbb {R}}_{>0}^n$, for any $(i,j) \ne (n,n)$, the intersection of ${\text {Tes}}_{n}({\varvec{a}})$ and the hyperplane $H_{i,j}^{n}$ (which is defined in (4.3)) is a facet of ${\text {Tes}}_n({\varvec{a}})$. From now on, we fix a notation for the facet of ${\text {PTes}}_{n}({\varvec{a}})$ that corresponds to the facet ${\text {Tes}}_n({\varvec{a}}) \cap H_{i,j}(n)$ of ${\text {Tes}}_n({\varvec{a}})$ under the transformation $\psi _{{\text {diag}}}$:

$$\begin{aligned} F_{i,j}({\varvec{a}}):=\psi _{{\text {diag}}}({\text {Tes}}_{n}({\varvec{a}}) \cap H_{i,j}^{n}). \end{aligned}$$

(4.6)

Using Lemma 4.9, we can easily translate Lemmas 4.2 and 4.7, and Theorem 4.5 to a version for projected Tesler polytopes as below:

Proposition 4.10

Let ${\varvec{a}}=(a_1, \dots , a_n) \in {\mathbb {R}}_{\ge 0}^n$. Then the following statements are true.

• (i) If $a_1 > 0$, i.e., ${\varvec{a}}\in {\mathbb {R}}_{>0} \times {\mathbb {R}}_{\ge 0}^{n-1}$, then ${\text {PTes}}_n({\varvec{a}})$ is a full-dimensional polytope in ${\mathbb {U}}(n-1)$.

• (ii) The projected Tesler polytope ${\text {PTes}}_n({\varvec{a}})$ is integral if and only if ${\varvec{a}}\in {\mathbb {Z}}_{\ge 0}^n$.

• (iii) If ${\varvec{a}}\in {\mathbb {Z}}_{> 0}^n$, then the projected Tesler polytope ${\text {PTes}}_n({\varvec{a}})$ is totally unimodular.

• (iv) If ${\varvec{a}}\in {\mathbb {R}}_{>0}^n$, then for any k facets $F_{i_{1},j_{1}}({\varvec{a}}), \dots , F_{i_{k},j_{k}}({\varvec{a}})$ of ${\text {PTes}}_n({\varvec{a}})$, they are precisely the k supporting facets of a codimension k face of ${\text {PTes}}_n({\varvec{a}})$ if and only if $H_{i_{1},j_{1}}^{n} \cap \cdots \cap H_{i_{k},j_{k}}^{n}$ does not make any zero rows.

Similarly, we can convert Corollary 4.6—the linear inequality description for ${\text {Tes}}_n({\varvec{a}})$ with all inequalities being facet-defining—to one for ${\text {PTes}}_n({\varvec{a}})$, which provides us a facet normal vector for each facet $F_{i,j}({\varvec{a}})$ of ${\text {PTes}}_{n}({\varvec{a}})$.

Definition 4.11

Let ${\varvec{e}}_{i,j}^n \in {\mathbb {U}}(n)$ be the matrix in which the (i, j)-th entry is 1 and all other entries are 0. For $1 \le k \le n$, define the k-th shifted hook sum matrix to be

$$\begin{aligned} \varvec{{ s }}_k^{n}:=\sum _{j=k}^{n} {\varvec{e}}_{k,j}^{n} - \sum _{i=1}^{k-1} {\varvec{e}}_{i,k-1}^{n} \in {\mathbb {U}}(n). \end{aligned}$$

Proposition 4.12

Let ${\varvec{a}}=(a_{1},\dots ,a_{n}) \in {\mathbb {R}}_{> 0}^n$. Then

$$\begin{aligned} {\text {PTes}}_n({\varvec{a}}) = \biggl \{Y \in {\mathbb {U}}(n-1):\!\!\!\!\begin{array}{lcl} &{}&{} \langle \varvec{{ s }}_i^{n-1},Y \rangle \le a_i \text { for all }1 \le i \le n-1\\ &{}&{} \langle -{\varvec{e}}_{i,j-1}^{n-1},Y \rangle \le 0 \text { for all }1 \le i < j \le n\\ \end{array}\!\biggl \}, \end{aligned}$$

where the inequality $\langle \varvec{{ s }}_i^{n-1},Y \rangle \le a_i$ defines the facet $F_{i,i}({\varvec{a}})$ and the inequality $\langle -{\varvec{e}}_{i,j-1}^{n-1},Y \rangle \le 0$ defines the facet $F_{i,j}({\varvec{a}})$. Hence,

$$\begin{aligned} \varvec{n}_{i,j}:={\left\{ \begin{array}{ll} \varvec{{ s }}_i^{n-1} &{} \text {if } {1 \le i=j \le n-1}, \\ -{\varvec{e}}_{i,j-1}^{n-1} &{} \text {if } {1\le i<j \le n}, \end{array}\right. } \end{aligned}$$

(4.7)

is the primitive outer normal vector of the facet $F_{i,j}({\varvec{a}})$.

Proof

By Corollary 4.6 and Lemma 4.9, it is enough to show the following two statements are true:

• (i) For any $1 \le i \le n-1$, the image of the halfspace $\{X=(x_{l,k}) \in \mathbb {H}_n({\varvec{a}}):x_{i,i} \ge 0 \}$ of $\mathbb {H}_n({\varvec{a}})$ under $\psi _{{\text {diag}}}$ is precisely the halfspace $\{Y \in {\mathbb {U}}(n-1):\langle \varvec{{ s }}_{i,}^{n-1},Y \rangle \le a_i\}$ of ${\mathbb {U}}(n-1)$.

• (ii) For any $1 \le i\! < \! j \le n$, the image of the halfspace $\{X=(x_{l,k}) \in \mathbb {H}_n({\varvec{a}}):x_{i,j} \ge 0\}$ of $\mathbb {H}_n({\varvec{a}})$ under $\psi _{{\text {diag}}}$ is precisely the halfspace $\{Y \in {\mathbb {U}}(n-1):\langle -{\varvec{e}}_{i,j-1}^{n-1},Y \rangle \le 0\}$ of ${\mathbb {U}}(n-1)$.

We only prove (i) here, as the proof for (ii) is similar (and simpler). One sees that (i) is equivalent to that for any $1 \le i \le n-1$ and for any $X = (x_{l,k}) \in \mathbb {H}_n({\varvec{a}})$,

$$\begin{aligned} x_{i,i} \ge 0 \quad \Longleftrightarrow \quad \langle \varvec{{ s }}_{i,}^{n-1}, \psi _{{\text {diag}}}(X) \rangle \le a_i. \end{aligned}$$

However, since $X \in \mathbb {H}_n({\varvec{a}})$, we have $\eta _i(X) = a_i$, i.e.,

$$\begin{aligned} (x_{i,i}+x_{i,i+1}+\dots +x_{i,n}) - (x_{1,i}+x_{2,i}+\dots +x_{i-1,i}) = a_i. \end{aligned}$$

Hence,

$$\begin{aligned} x_{i,i} \ge 0 \quad \Longleftrightarrow \quad (x_{i,i+1}+\dots +x_{i,n}) - (x_{1,i}+x_{2,i}+\dots +x_{i-1,i}) \le a_i, \end{aligned}$$

where one can verify that the latter is exactly $\langle \varvec{{ s }}_{i,}^{n-1}, \psi _{{\text {diag}}}(X) \rangle \le a_i$. $\square$

Corollary 4.13

For any ${\varvec{a}}, {\varvec{b}}\in {\mathbb {R}}_{>0}^n$, the projected Tesler polytopes ${\text {PTes}}_n({\varvec{a}})$ and ${\text {PTes}}_n({\varvec{b}})$ have the same normal fan.

Proof

By the second part of Proposition 4.12, one sees that the normal fans of ${\text {PTes}}_n({\varvec{a}})$ and ${\text {PTes}}_n({\varvec{b}})$ have exactly the same one-dimensional cones. It then follows from Lemma 2.3 (ii) and Proposition 4.10 (iv), all other dimensional cones in these two normal fans are the same. $\square$

4.3 BV-$\alpha$ Values of Projected Tesler Polytopes

By (ii) of Proposition 4.10, we know that for any ${\varvec{a}}\in {\mathbb {Z}}_{> 0}^n$, the projected Tesler polytope ${\text {PTes}}_n({\varvec{a}})$ is integral, and hence we can discuss the BV-$\alpha$ values arising from them. Moreover, it follows from Corollary 4.13 that ${\text {PTes}}_n({\varvec{a}})$ and ${\text {PTes}}_n({\varvec{1}})$ have the same normal fan, and thus by Lemma 2.2 they share exactly the same BV-$\alpha$ values. For simplicity and convenience, below we will present our positivity results on BV-$\alpha$ values on ${\text {PTes}}_n({\varvec{1}})$ only, knowing that these results are true for any ${\text {PTes}}_n({\varvec{a}})$ where ${\varvec{a}}\in {\mathbb {Z}}_{> 0}^n$.

First, we have seen in Proposition 4.12 that any facet $F_{i,j}({\varvec{1}})$ of ${\text {PTes}}_n({\varvec{1}})$ with $i<j$ has its outer normal as a member of the standard basis up to sign. Then using [5, Example 3.15], we get the BV-$\alpha$ values for every face whose supporting facets are all of this form.

Lemma 4.14

Suppose F is a codimension k face of ${\text {PTes}}_{n}({\varvec{1}})$, and its supporting facets are $F_{i_{1},j_{1}}({\varvec{1}}),\dots ,F_{i_{k},j_{k}}({\varvec{1}})$ where $i_{l}<j_{l}$ for all $1 \le l \le k$. Then

$$\begin{aligned}{\alpha ^{\textrm{BV}}}(F, {\text {PTes}}_n({\varvec{1}}))=\frac{1}{2^k}.\end{aligned}$$

By Proposition 4.10, the projected Tesler polytope ${\text {PTes}}_{n}({\varvec{1}})$ is totally unimodular, and is full-dimensional in ${\mathbb {U}}(n-1)$. Therefore, we can apply the procedure described in § 3.2 to calculate the BV-$\alpha$ values of codimension 2 and codimension 3 faces of ${\text {PTes}}_{n}(\varvec{1})$. Given Lemma 4.14, we only consider faces of ${\text {PTes}}({\varvec{1}})$ where at least one of its supporting facets is $F_{l,l}({\varvec{a}})$ for some $1 \le l \le n-1$. We will calculate the BV-$\alpha$ values of faces we consider case by case, and use the following terminologies in our description of cases.

Suppose $1 \le i < j \le n$ and $1 \le l \le n-1$. We say that the position (i, j) (of an upper triangular matrix) is on the l-th hook if either $i=l$ or $j=l$. More specifically, we say (i, j) is on the row of the l-th hook if $i=l$ and (i, j) is on the column of the l-th hook if $j=l$. It is easy to verify that

$$\begin{aligned} \langle \varvec{n}_{l,l}, \varvec{n}_{i,j} \rangle =\langle \varvec{{ s }}^{n-1}_{l}, -{\varvec{e}}^{n-1}_{i,j-1}\rangle ={\left\{ \begin{array}{ll} -1 &{} \quad \text {if }(i,j) \text { is on the row of the }l\text {-th hook}, \\ 1 &{} \quad \text {if }(i,j) \text { is on the column of the }l\text {-th hook},\\ 0 &{} \quad \text {if }(i,j) \text { is not on the }l\text {-th hook}.\end{array}\right. } \end{aligned}$$

This result will be used repeatedly in calculations involved in the proof of Lemmas 4.15 and 4.16 below.

Lemma 4.15

Assume $n \ge 3$. Suppose F is a codimension 2 face of ${\text {PTes}}_{n}(\varvec{1})$, and its two supporting facets are $F_{l,l}({\varvec{1}})$, $F_{i,j}({\varvec{1}})$ (with $i\le j$). We compute ${\alpha ^{\textrm{BV}}}(F,{\text {PTes}}_{n}(\varvec{1}))$ for all possible cases below.

• (a) Suppose $i <j$. There are three subcases:

  • (a.i) If (i, j) is on the row of the l-th hook, then

    $$\begin{aligned} {\alpha ^{\textrm{BV}}}(F,{\text {PTes}}_{n}(\varvec{1}))=\frac{1}{4}+\frac{1}{12}\cdot \frac{n}{n-1}. \end{aligned}$$

  • (a.ii) If (i, j) is on the column of the l-th hook, then

    $$\begin{aligned} {\alpha ^{\textrm{BV}}}(F,{\text {PTes}}_{n}(\varvec{1}))=\frac{1}{4}-\frac{1}{12}\cdot \frac{n}{n-1}. \end{aligned}$$

  • (a.iii) If (i, j) is not on the l-th hook, then ${\alpha ^{\textrm{BV}}}(F,{\text {PTes}}_{n}(\varvec{1}))=1/4$.

• (b) Suppose $i=j$. Then

  $$\begin{aligned} {\alpha ^{\textrm{BV}}}(F,{\text {PTes}}_{n}(\varvec{1}))=\frac{1}{4}+\frac{1}{6(n-1)}. \end{aligned}$$

Hence, ${\alpha ^{\textrm{BV}}}(F,{\text {PTes}}_{n}(\varvec{1}))$ is positive for codimension 2 faces of ${\text {PTes}}_n({\varvec{1}})$.

Proof

We only provide a proof for case (a), but one can obtain the BV-$\alpha$ values for the other cases by following the same procedure. For case (a), we have $i=l<j$. By Proposition 4.12, the primitive outer normal vector of $F_{l,l}({\varvec{1}})$ is $\varvec{n}_{l,l}=\varvec{{ s }}^{n-1}_{l}$ and that of $F_{i,j}({\varvec{1}})$ is $\varvec{n}_{l,j} = -{\varvec{e}}^{n-1}_{l,j-1}$. Thus, the MDP of ${\text {ncone}}{(F, {\text {PTes}}_n(\varvec{1}))}$ is

$$\begin{aligned}\begin{bmatrix} \langle \varvec{n}_{l,l}, \varvec{n}_{l,l}\rangle \ {} &{} \langle \varvec{n}_{l,l}, \varvec{n}_{l,j}\rangle \\ \langle \varvec{n}_{l,j}, \varvec{n}_{l,l} \rangle &{}\langle \varvec{n}_{l,j}, \varvec{n}_{l,j}\rangle \end{bmatrix}{} & {} =\begin{bmatrix} \langle \varvec{{ s }}^{n-1}_{l}, \varvec{{ s }}^{n-1}_{l}\rangle &{} \langle \varvec{{ s }}^{n-1}_{l}, -{\varvec{e}}^{n-1}_{l,j-1}\rangle \\ \langle -{\varvec{e}}^{n-1}_{l,j-1}, \varvec{{ s }}^{n-1}_{l} \rangle &{}\langle -{\varvec{e}}^{n-1}_{l,j-1}, -{\varvec{e}}^{n-1}_{l,j-1}\rangle \end{bmatrix} \\ {}{} & {} = \begin{bmatrix} n-1&-1 {\varvec{1}}&1 \end{bmatrix}. \end{aligned}$$

Then we compute its inverse, which by Corollary 3.8 gives the MDP of ${\text {fcone}}^p(F, {\text {PTes}}_n(\varvec{1}))$:

$$\begin{aligned}\begin{bmatrix} n-1 &{} -1 \\ -1 &{} 1\end{bmatrix}^{-1} = \begin{bmatrix}\displaystyle \frac{1}{n-2} &{}\displaystyle \frac{1}{n-2}\ \\ \displaystyle \frac{1}{n-2} &{}\displaystyle \frac{n-1}{n-2}\, \\ \end{bmatrix}. \end{aligned}$$

Hence, if we let $\varvec{u}_1$ and $\varvec{u}_2$ be the primitive generators for the unimodular cone ${\text {fcone}}^{p}(F,P)$ (with respect to the lattice ${\mathbb {Z}}^{\left( {\begin{array}{c}n\\ 2\end{array}}\right) }\!\!/{{\text {lin}}(F)}$) and assume $|\varvec{u}_1| \le |\varvec{u}_2|$, then

$$\begin{aligned} \langle \varvec{u}_{1}, \varvec{u}_{1} \rangle = \dfrac{1}{n-2}, \quad \langle \varvec{u}_{1}, \varvec{u}_{2} \rangle = \dfrac{1}{n-2}, \quad \text { and } \quad \langle \varvec{u}_{2}, \varvec{u}_{2} \rangle = \dfrac{n-1}{n-2}. \end{aligned}$$

Therefore, applying Lemma 3.1, we obtain the value of ${\alpha ^{\textrm{BV}}}(F,{\text {PTes}}_{n}(\varvec{1}))$ as shown in the lemma.

As we mentioned above, we won’t provide detailed calculation for the other cases. However, for easy reference, we summarize involved matrices and the $\alpha$-value of each case of this lemma in Table 1. $\square$

Table 1 Data for Lemma 4.15

Lemma 4.16

Assume $n \ge 3$. Suppose F is a codimension 3 face of ${\text {PTes}}_{n}(\varvec{1})$, and its two supporting facets are $F_{l,l}({\varvec{1}}), F_{i_{1},j_{1}}({\varvec{1}})$ and $F_{i_{2},j_{2}}({\varvec{1}})$ (with $i_1 \le j_1$ and $i_2 \le j_2$). We compute ${\alpha ^{\textrm{BV}}}(F,{\text {PTes}}_{n}(\varvec{1}))$ for all possible cases below.

• (a) Suppose $i_{1} < j_{1}$ and $i_{2} < j_{2}$. There are six subcases.

  • (a.i) If both $(i_{1},j_{1})$ and $(i_{2},j_{2})$ are not on the l-th hook, then ${\alpha ^{\textrm{BV}}}(F,{\text {PTes}}_{n}(\varvec{1}))={1}/{8}$.

  • (a.ii) If one of $(i_{1},j_{1})$ and $(i_{2},j_{2})$ is on the row of the l-th hook and the other is not on the l-th hook, then

    $$\begin{aligned} {\alpha ^{\textrm{BV}}}(F,{\text {PTes}}_{n}(\varvec{1}))=\frac{1}{8} +\frac{n}{24(n-1)}. \end{aligned}$$

  • (a.iii) If one of $(i_{1},j_{1})$ and $(i_{2},j_{2})$ is on the column of the l-th hook and the other is not on the l-th hook, then

    $$\begin{aligned} {\alpha ^{\textrm{BV}}}(F,{\text {PTes}}_{n}(\varvec{1}))=\frac{1}{8}-\frac{n}{24(n-1)}. \end{aligned}$$

  • (a.iv) If both of $(i_{1},j_{1})$ and $(i_{2},j_{2})$ are on the row of the l-th hook, then

    $$\begin{aligned} {\alpha ^{\textrm{BV}}}(F,{\text {PTes}}_{n}(\varvec{1}))=\frac{1}{8} +\frac{n}{12(n-2)}. \end{aligned}$$

  • (a.v) If one of $(i_{1},j_{1})$ and $(i_{2},j_{2})$ is on the column of the l-th hook and the other is on the row of the l-th hook, then

    $$\begin{aligned} {\alpha ^{\textrm{BV}}}(F,{\text {PTes}}_{n}(\varvec{1}))=\frac{1}{8} -\frac{1}{12(n-2)}. \end{aligned}$$

  • (a.vi) If both of $(i_{1},j_{1})$ and $(i_{2},j_{2})$ are on the column of the l-th hook, then ${\alpha ^{\textrm{BV}}}(F,{\text {PTes}}_{n}(\varvec{1}))={1}/{24}$.

• (b) Suppose exactly one of $i_1=j_1$ and $i_2=j_2$ is true. Without loss of generality, we assume $i_1=j_1$ and let $m:=i_1$. There are four subcases.

  • (b.i) If $(i_{2},j_{2})$ is not on the l-th or the m-th hook, then

    $$\begin{aligned} {\alpha ^{\textrm{BV}}}(F,{\text {PTes}}_{n}(\varvec{1}))=\frac{1}{8}+\frac{1}{12(n-1)}. \end{aligned}$$

  • (b.ii) If $(i_{2},j_{2})$ is on the row of either the l-th hook or the m-th hook but is not on the hook of the other, then

    $$\begin{aligned} {\alpha ^{\textrm{BV}}}(F,{\text {PTes}}_{n}(\varvec{1}))=\frac{1}{8}+\frac{n^{2}+n-3}{24(n^{2}-3n+2)}. \end{aligned}$$

  • (b.iii) If $(i_{2},j_{2})$ is on the column of either the l-th hook or the m-th hook but is not on the hook of the other, then

    $$\begin{aligned} {\alpha ^{\textrm{BV}}}(F,{\text {PTes}}_{n}(\varvec{1}))=\frac{1}{8} - \frac{n^2 - 3n + 3}{24(n^2-3n+2)}. \end{aligned}$$

  • (b.iv) If $(i_{2},j_{2})$ is on the column of either the l-th hook or the m-th hook and is on the row of the other hook, then ${\alpha ^{\textrm{BV}}}(F,{\text {PTes}}_{n}(\varvec{1}))={1}/{8}$.

• (c) Suppose $i_1=j_1$ and $i_2=j_2$. Then

  $$\begin{aligned} {\alpha ^{\textrm{BV}}}(F,{\text {PTes}}_{n}(\varvec{1}))=\frac{1}{8}+\frac{1}{4(n-2)}. \end{aligned}$$

Hence, ${\alpha ^{\textrm{BV}}}(F,{\text {PTes}}_{n}(\varvec{1}))$ is positive for codimension 3 faces of ${\text {PTes}}_n({\varvec{1}})$.

Proof

We follow the same procedure as in the proof for case (a.i) of Lemma 4.15 to obtain formulas for ${\alpha ^{\textrm{BV}}}(F,{\text {PTes}}_{n}(\varvec{1}))$ for all cases. Please refer to Table 2 for all the matrices involved in the calculation. (We did not include $\alpha$-values in these tables so that they would fit into one single page.) We also remark that when $n=3$, cases (a.iv), (a.v), and (a.vi), and case (c) won’t occur, so $n-3$ occurring in the denominators of the MDP of ${\text {fcone}}^p(F, {\text {PTes}}_n({\varvec{1}}))$ in these cases is not a concern. $\square$

Table 2 Matrices for Lemma 4.16

We are now ready to prove Theorem 1.2.

Proof of Theorem 1.2

The first conclusion follows from Lemmas 4.15 and 4.16. The second conclusion follows from the first conclusion and (1.1). $\square$

4.4 Corollaries to Theorem 1.2

Using the Reduction Theorem 3.4, we can generalize our BV-$\alpha$-positivity results for codimension 2 and 3 faces of ${\text {PTes}}_{n}(\varvec{1})$ to any deformations of ${\text {PTes}}_{n}(\varvec{1})$:

Corollary 4.17

Let P be a d-dimensional integral polytope that is a deformation of ${\text {PTes}}_{n}(\varvec{1})$. Then the following statements are true:

• (i) If $d=\dim ({\text {PTes}}_{n}(\varvec{1}))=\left( {\begin{array}{c}n\\ 2\end{array}}\right)$, then all the codimension 2 and 3 faces of P have positive BV-$\alpha$ values. Therefore, $e_{d-2}(P)>0$ and $e_{d-3}(P)>0$.

• (ii) If $d=\dim ({\text {PTes}}_{n}(\varvec{1}))-1=\left( {\begin{array}{c}n\\ 2\end{array}}\right) -1$, then all the codimension 2 faces of P have positive BV-$\alpha$ values. Therefore, $e_{d-2}(P)>0$.

Proof

If $d=\dim ({\text {PTes}}_{n}(\varvec{1}))$, then by Lemmas 4.15 and 4.16, and Theorem 3.4, we have that ${\alpha ^{\textrm{BV}}}(F,P)>0$ for every face F of P that has dimension $d-2$ or $d-3$. Therefore, the coefficients $e_{d-2}(P)$ and $e_{d-3}(P)$ are positive by (1.1). Similarly, if $d=\dim ({\text {PTes}}_{n}(\varvec{1}))-1$, Lemma 4.16 and Theorem 3.4 imply that ${\alpha ^{\textrm{BV}}}(F,P)>0$ for all $(d-2)$-dimensional faces F of P. Therefore, we have that $e_{d-2}(P)>0$ by (1.1).$\square$

Proof of Corollary 1.3

Recall that ${\text {PTes}}_{n}(\varvec{1}) = \psi _{{\text {diag}}}( {\text {Tes}}_n(\varvec{1}))$, where by Lemma 4.9 the map $\psi _{{\text {diag}}}$ induces a unimodular transformation from $\mathbb {H}_n(\varvec{1})$ to ${\mathbb {U}}(n-1)$. Then the conclusions follow from Corollary 4.17, Lemma 2.6, and the fact that unimodular transformations are invertible and preserve Ehrhart polynomials. $\square$

Proof of Corollary 1.4

Suppose the first p entries of ${\varvec{a}}$ are zero and $a_{p+1} > 0$, and let ${\varvec{b}}= (a_{p+1}, a_{p+2}, \dots , a_n)$. By Lemma 4.2, the Tesler polytope ${\text {Tes}}_n({\varvec{a}})$ is isomorphic to the Tesler polytope ${\text {Tes}}_{n-p}({\varvec{b}})$ which has the same dimension as ${\text {Tes}}_{n-p}({\varvec{1}})$. It is a classical result that ${\text {Tes}}_{n-p}({\varvec{b}})$ is a deformation of ${\text {Tes}}_{n-p}({\varvec{1}})$. (See [11] or [18, Sect. 4].) Therefore, the conclusion follows from (i) of Corollary 1.3. $\square$