Salem numbers, spectral radii and growth rates of hyperbolic Coxeter groups

We show that not every Salem number appears as the growth rate of a cocompact hyperbolic Coxeter group. We also give a new proof of the fact that the growth rates of planar hyperbolic Coxeter groups are spectral radii of Coxeter transformations, and show that this need not be the case for growth rates of hyperbolic tetrahedral Coxeter groups.


Introduction
Let P ⊂ H n be a compact hyperbolic Coxeter polyhedron of dimension n ≥ 2. This means that P is a convex polyhedron bounded by N ≥ n + 1 hyperbolic hyperplanes H 1 , . . ., H N that either intersect under a dihedral angle of the form π/k for an integer k ≥ 2 or admit a common perpendicular in H n .The group generated by the reflections r i in the hyperplanes H i , 1 ≤ i ≤ N , is a discrete group G ⊂ IsomH n called a (cocompact) hyperbolic Coxeter group.When N is small, their quotient spaces H n /G give rise to hyperbolic orbifolds and manifold covers enjoying particularly nice extremality properties.The simplest examples of hyperbolic Coxeter groups arise from Coxeter k-gons P = (p 1 , . . ., p k ) ⊂ H 2 , for k ≥ 3 , where the integers p 1 , . . ., p k ≥ 2 satisfy 1 In particular, there exist infinitely many non-isometric Coxeter polygons.Furthermore, a well known result of Siegel states that the hyperbolic 2-orbifold of minimal volume originates from the Coxeter triangle (2,3,7).
A hyperbolic Coxeter group is a geometric realisation of a certain abstract Coxeter system.A Coxeter system (W, S) of rank N consists of a group W with finite generating set S = {s 1 , . . ., s N } and with relations (s i s j ) m ij = 1 for all indices i, j.Here the integers m ij satisfy m ii = 1 and m ij = m ji ∈ {2, 3, . . ., ∞}, otherwise.An exponent m ij = ∞ indicates that the product s i s j is of infinite order.The growth series f S (t) of (W, S) is given by where a k ∈ Z equals the number of words w ∈ W with S-length k, and it characterises the complexity of W .By a result of Steinberg, f S (t) is a rational function that depends on the set of finite subgroups of W .The inverse τ = 1/R of the radius of convergence R of f S (t) is a real algebraic integer called the growth rate of W , or also the growth rate of its Coxeter polyhedron.
For a hyperbolic Coxeter group G ⊂ IsomH n , the growth rate satisfies τ > 1 so that G is of exponential growth.More specifically, for G compact, results of Floyd, Plotnick and Parry imply that τ is a Salem number or a quadratic unit when n = 2 or n = 3. Recall that a Salem number is a real algebraic integer > 1 such that all Galois conjugates have absolute value not greater than 1 and at least one of them has absolute value equal to 1.An interesting example is given by the growth rate of the reflection group associated with the Coxeter triangle (2,3,7).It is equal to Lehmer's number α L ≈ 1.176281 with minimal polynomial L(t) = t 10 + t 9 − t 7 − t 6 − t 5 − t 4 − t 3 + t + 1. Observe that α L is the smallest Salem number known to date.
In reverse, our first main result sheds light on the realisation of Salem numbers as growth rates of hyperbolic Coxeter groups and their polyhedra.
Theorem 1 Not every Salem number is the growth rate of a compact hyperbolic Coxeter polyhedron.
Consider an abstract Coxeter system (W, S) of rank N together with its natural representation as a discrete group of reflections in GL N (V) for a certain quadratic real vector space V .A Coxeter element c ∈ W is a word of Slength N so that every generator in S appears exactly once.Its representative C ∈ GL N (V) is called a Coxeter transformation.By means of its order and the eigenvalue spectrum one can decide about the nature of W .For example, W is finite if and only if C is of finite order; see [15].
For integers p 1 , . . ., p k ≥ 2, consider the star graph Star(p 1 , . . ., p k ) given by the tree with one vertex of valency k that has k outgoing paths of respective lengths p i − 1.Such a graph describes a Coxeter system (W, S) as follows.Each node s of the graph yields a generator in S, and the relations of W = S are defined by s 2 = 1, by (st) 3 = 1 if the nodes s, t are joined by an edge, and by (st) 2 = 1, otherwise.
Our second main result establishes a bridge between the growth rates of reflection groups of planar hyperbolic Coxeter groups and the spectral radii of Coxeter transformations of suitably parametrised star graphs.
Then the growth rate of the reflection group given by the compact Coxeter kgon (p 1 , . . ., p k ) in H 2 equals the spectral radius of the Coxeter transformation of Star(p 1 , . . ., p k ).This result is implicitly stated in the work of E. Hironaka [13] and based on a connection to Alexander polynomials of pretzel links and the theory of fibered knots and links [12,14].Our method of proof of Theorem 2 is different and does not use any topology.Instead, we provide and exploit explicit recursion formulas.
Conversely, for dimensions n > 2, we show that not every growth rate of a compact Coxeter polyhedron in H n is equal to the spectral radius of a Coxeter transformation.In fact, in contrast to the Coxeter tetrahedron with Coxeter symbol [4,3,5], the growth rate of the Coxeter tetrahedron [3,5,3] is not equal to the spectral of a Coxeter transformation.Note that the tetrahedron [4,3,5] has minimal volume among all hyperbolic Coxeter tetrahedra while [3,5,3] has minimal growth rate among all Coxeter polyhedra in H 3 .
The paper is organised as follows.In Section 2, we recall in a first part 2.1 the essential concepts of Coxeter group, Coxeter graph and Coxeter transformation.In the second part 2.2, we describe hyperbolic Coxeter polyhedra and their associated reflection groups.Some examples provide a glimpse into the wealth of hyperbolic Coxeter groups which-in contrast to the spherical and affine Coxeter groups-are far from being classified; see [33] and [8], for example.In Section 3, we review the basic notions of growth series and growth rate of a Coxeter system.The partial order on the set of Coxeter systems and its implication for growth rates and minimality of certain Coxeter systems are presented.These aspects will be useful tools in some of our proofs.This first part 3.1 is completed by a brief discussion of the connection of certain growth rates with Salem numbers.In part 3.2, we identify the planar hyperbolic Coxeter group having the second smallest growth rate as a preparation to prove Theorem 1. Section 4 is devoted to the proof of Theorem 2, and in Section 5, we show that not every growth rate of a hyperbolic Coxeter group is the spectral radius of a Coxeter transformation.

Coxeter groups and Coxeter elements
A Coxeter system (W, S) is a group W with finite generating set S = {s 1 , . . ., s N } and with the relations (s i s j ) m ij = 1 for all indices i, j, where the integers m ij satisfy m ii = 1 and m ij = m ji ∈ {2, 3, . . ., ∞}, otherwise.Here, m ij = ∞ means that the product s i s j is of infinite order.The group W is a Coxeter group of rank N. Given a Coxeter system (W, S), the corresponding Coxeter diagram Γ is the weighted graph whose vertices v 1 , . . ., v N correspond to the generators s 1 , . . ., s N , and an edge of weight m ij joins v i to v j when m ij ≥ 3. Edge weights are typically omitted if they equal 3.
Coxeter groups admit a canonical geometrical representation.Let V be a real vector space with basis e 1 , . . ., e N , where the vector e i corresponds to the generator s i of S, say.Equip V with the symmetric bilinear form B defined by The geometric representation ρ : W → GL N (V) defined by associates to each generator s i the corresponding reflection r i = ρ(s i ) with respect to the subspace For a reference, see McMullen [24].We note that if all the weights of the Coxeter diagram Γ are equal to 3, then the Coxeter adjacency matrix A equals the adjacency matrix of the underlying abstract graph.

Hyperbolic Coxeter groups
Let H n denote the standard hyperbolic n-space in its linear model In this setting, a hyperbolic hyperplane H is given by the intersection of H n with the Lorentzian-orthogonal complement e L of a (space-like) vector e ∈ R n+1 normalised to be of norm q n,1 (e) = 1.The reflection r = r H with respect to the hyperbolic hyperplane H of H n is given by x → r(x) = x − 2 e, x n,1 e where x, y n,1 denotes the bilinear form of signature (n, 1) associated with q n,1 .The isometry group of H n is given by the group O • (n, 1) of positive Lorentzian matrices; see [28,Chapter 3].Notice that each isometry is a finite composition of reflections in hyperbolic hyperplanes.
A Coxeter polyhedron P ⊂ H n is a convex polyhedron all of whose dihedral angles are of the form π/m for an integer m ∈ {2, 3, . . ., ∞}.We always assume that P is of finite volume and hence bounded by finitely many hyperplanes H 1 , . . ., H N with N ≥ n + 1. Represent each hyperplane H i = e L i by a unit normal vector e i directed away from P so that the half-space x, e i n,1 ≤ 0} contains P. In particular, P = ∩ N i=1 H − i .For i = 1, . . ., N , the reflections r i with respect to H i generate a discrete group G = (G, S) of hyperbolic isometries with generating set S = {r 1 , . . ., r N }.The elements of S satisfy r 2 i = 1 and, for i = j, the rotation relations (r i r j ) m ij = 1 if m ij < ∞.In particular, the exponents m ij are symmetric with respect to i, j, and when finite, they are closely related to the dihedral angles formed by H i , H j when intersecting in H n .Products r i r j of infinite order can be described in a geometric way as well (see below).As a consequence, the group G is a Coxeter group which we call a hyperbolic Coxeter group.Furthermore, the group G can be described by a Coxeter diagram Γ as above.
Consider the Gram matrix Gram(P) = e i , e j n,1 whose entries are described as follows.
(3) e i , e j n,1 Many combinatorial and geometric features of P can be read off from its Gram matrix Gram(P); see [33].For example, the polyhedron P is a compact nsimplex if Gram(P) is an indecomposable and invertible matrix of signature (n, 1) such that all principal submatrices are positive definite.
In view of (3), a product r i r j is of infinite order if the hyperplanes H i , H j are (hyperbolic) parallel or at distance l ij in H n .We take this additional information into account and describe the Example 2 As in the 2-dimensional case, there are infinitely many nonisometric compact Coxeter polyhedra in H 3 .Their description is not of the same elementary nature as in Example 1 but there is a complete characterisation due to Andreev; see [18], for example.
Example 3 Compact Coxeter n-simplices were classified by Lannér and exist for n ≤ 4, only (see [34, Part II, Chapter 5]).Of particular interest will be the Coxeter tetrahedra given by the symbols [3,5,3] and [4,3,5].Due to work of Koszul and Chein, non-compact Coxeter n-simplices are classified as well and exist for n ≤ 9.All their volumes are computed in [17].
Example 4 Compact Coxeter polyhedra with mutually intersecting bounding hyperplanes exist in H n for n ≤ 4, only.Such a polyhedron is either a simplex or one of the seven Esselmann polyhedra.These results are due to Felikson and Tumarkin [9, Theorem A].For more detailed information, see [8].
3 Salem numbers and growth rates

Growth series and growth rates
For a Coxeter system (W, S) with generating set S = {s 1 , . . ., s N } we introduce the notion and review the relevant properties of the growth series of (W, S); for references, see [1], [19] and [20].The growth series f S of W is given by where a k ∈ Z equals the number of words w ∈ W with S-length k.By Steinberg's formula, where W T , T ⊂ S , is a finite Coxeter subgroup of W , and where W ∅ = {1}.By a result of Solomon, the associated growth polynomials f T are given explicitly in terms of their exponents n 1 , . . ., n p according to Here we use the standard notations [l] := 1 For the exponents n 1 = 1, n 2 , . . ., n p of G T , see [7, Section 9.7], for example.
In particular, the dihedral group D l 2 , l ≥ 2 , has exponents 1, l − 1 and growth polynomial [2][l].As a consequence, in its disk of convergence, the growth series f S (t) of a Coxeter system (W, S) is a rational function and quotient of coprime monic polynomials p(t), q(t) ∈ Z[t] of equal degree.The growth rate τ W = τ (W,S) is defined by and equals the inverse of the radius of convergence R of f S (t).
Growth rates satisfy a nice monotonicity property on the partially ordered set of Coxeter systems.For two Coxeter systems (W, S) and (W , S ), one declares (W, S) ≤ (W , S ) if there is an injective map ι : S → S such that m st ≤ m ι(s)ι(t) for all s, t ∈ S. If ι extends to an isomorphism between W and W , one writes (W, S) (W , S ), and (W, S) < (W , S ) otherwise.In the latter case, we often say that the Coxeter system (W , S ) dominates the system (W, S).This partial order satisfies the descending chain condition since m st ∈ {2, 3, . . ., ∞} where s = t.In particular, any strictly decreasing sequence of Coxeter systems is finite, which, in the hyperbolic case, leads to the notion of minimal hyperbolic Coxeter systems; see [24].In this context, we shall exploit the following result of Terragni [31,Section 4.3].Then, by Lemma 1, we deduce that τ [3,8] Consider As a consequence of ( 4)-( 6), the growth function of a compact hyperbolic Coxeter polygon P = (p 1 , . . ., p k ) ⊂ H 2 can be determined as follows.Since (7) 1 one deduces that (see also [10, Section 2]) ( 8) In general, the growth rate τ G = τ P = 1/R of G (and of P) is an algebraic integer which, by results of Milnor and de la Harpe, is always strictly bigger than 1.By results of Floyd, Plotnick and Parry (see also [22]), the growth rate τ P of a compact Coxeter polyhedron P ⊂ H n with n = 2 and n = 3 is either a quadratic unit or a Salem number, that is, τ P is a real algebraic integer > 1 all of whose Galois conjugates have absolute value not greater than 1 and at least one of them has absolute value equal to 1.
The smallest Salem number known to date is Lehmer's number α L ≈ 1.176281 with minimal polynomial L(t) = t 10 + t 9 − t 7 − t 6 − t 5 − t 4 − t 3 + t + 1.By a result of E. Hironaka [12] (see also [20]), Lehmer's number α L is the minimal growth rate τ 1 among all τ P with P a compact hyperbolic Coxeter polygon, and it is realised by the triangle P = (2, 3, 7) in a unique way.In this context, recall Siegel's result that the associated Coxeter group [3,7] yields the (unique) minimal volume quotient space among all hyperbolic 2-orbifolds of finite volume; the second smallest hyperbolic 2-orbifold is given by the compact quotient space In comparison with α L , the growth rate τ 2 of the triangle group [3,8] has minimal polynomial t 10 − t 7 − t 5 − t 3 + 1 and is ≈ 1.230391.By looking at the complete list (L) of all Salem numbers of degree ≤ 44, which is due to Boyd, Mossinghoff and others (for a survey, see [29]; for the list (L), see [25]), the growth rate τ 2 is the seventh smallest Salem number in the list (L), only.
For compact Coxeter polyhedra P ⊂ H 3 , the smallest growth rate has been determined by Kellerhals and Kolpakov in [20].It is realised by the Coxeter group [3,5,3] in a unique way and of value ≈ 1.350980 with minimal polynomial t 10 − t 9 − t 6 + t 5 − t 4 − t + 1.In this way, τ [3,5,3] is bigger than the first 47 smallest Salem numbers as listed in (L).
Note that for higher dimensional Coxeter polyhedra P ⊂ H n , n ≥ 4 , there are many examples whose growth rates are not Salem numbers anymore.A simple example is given by the compact right-angled 120-cell C ⊂ H 4 with f 0 = 600 vertices and f 3 = 120 dodecahedral facets.As a consequence of [21, Proposition 3.2], the growth function f (t) of C is given by whose denominator polynomial is irreducible over Z with two inversive pairs of positive real roots (see Section for more details).In particular, the growth rate τ C ≈ 112.763387 is not a Salem number.In [32], Umemoto constructed an infinite sequence of non-isometric 4-dimensional compact Coxeter polyhedra whose growth rates are real 2-Salem numbers.These are algebraic integers α > 1 which have exactly one conjugate β outside the closed unit disk, and at least one conjugate on the unit circle.Then all other conjugates of α different from α −1 , β and β −1 lie on the unit circle.As in the case of Salem numbers and their minimal polynomials, called Salem polynomials, the minimal polynomial or 2-Salem polynomial of α is an irreducible palindromic polynomial of even degree.

Remark 2
In general, it is a difficult problem to decide whether a palindromic monic polynomial p(t) ∈ Z[t] is irreducible.Specifically, for a palindromic monic polynomial with four simple roots that are positive real and the other roots on the unit circle, it is difficult to decide whether it is a 2-Salem polynomial or splits into two Salem polynomials and possibly cyclotomic polynomials over Z.
In [4, Theorem 6.3, Theorem 7.1] (see also [5,Theorem 2.12]), Cannon provides a necessary and sufficient condition for p(t) to be a Salem polynomial, and he showed that the growth rate of a compact hyperbolic 4-simplex is not a Salem number.
Remark 3 Salem numbers and real 2-Salem numbers are special Perron numbers.A Perron number is a real algebraic integer > 1 all of whose conjugates are of strictly smaller absolute value.In [21], Kellerhals and Perren formulate a conjecture which can be stated in a modified way as follows: For every n ≥ 2, the growth rate of a hyperbolic Coxeter n-polyhedron is a Perron number.By means of the software package CoxIter [11] and its webversion, both due to Guglielmetti, one verifies that the conjecture is true for all known hyperbolic Coxeter polyhedra of finite volume.

Not every Salem number appears as a growth rate
With these preliminaries we are now ready to prove that not every Salem number is the growth rate of a hyperbolic Coxeter group.As a first step, we consider hyperbolic Coxeter polygons whose growth rates are small Salem numbers and prove the following result.
where the help functions h v , v ∈ P, and their sum H(t) can be written in the form since the exponents of the group D kv 2 are equal to n 1 = 1 and n 2 = n(k) = k v − 1.By results of [20, Section 3.1], we have the following properties for the functions h k (t) and [n] for all x ∈ (0, 1].(a) For all i < j, 0 < h i (t) < h j (t) < 1 .
(b) For any positive integer l, .
To this end, we consider three cases in terms of the number of vertices f 0 ≥ 3.
Case 1.Let f 0 ≥ 5, and consider compact Coxeter polygons with at least five vertices.Here, all the interior angles may be equal to π/2.Hence, by ( 11) and (b), we get the obvious estimate Case 2. Let f 0 = 4, and consider hyperbolic Coxeter quadrilaterals P by noticing that they may have at most three right angles.Hence, by the properties (a) and (b) above, we get the estimate ( 12) Therefore, (11), (b) and (12) imply that a compact Coxeter quadrilateral has strictly bigger growth rate than the triangle [3,8].
Case 3. Let f 0 = 3, and let P be a Coxeter triangle with angular existence condition 1/p + 1/q + 1/r < 1 for integers p, q, r ≥ 2. Hence, at most one angle of P can be equal to π/2, for example.The angular existence condition and the properties (a) and (b) imply that H must satisfy at least one of the following inequalities.
We are now ready to prove our first main result as given by Theorem 1 in Section 1.It provides the answer to a question which the first author raised at the Oberwolfach Mini-Workshop on Reflection Groups in Negative Curvature in April 2019.

Proof of Theorem 1
Let P ⊂ H n be a compact Coxeter polyhedron of dimension n ≥ 2 and denote by G = (G, S) its associated Coxeter group.Let τ = τ P = τ G be the growth rate of P and of G.For n = 2, the smallest growth rate τ 1 = τ [3,7] equals the smallest known Salem number given by Lehmer's number α L , and the second smallest growth rate τ 2 = τ [3,8] is the seventh smallest Salem number by Proposition 1. Hence the five Salem numbers strictly in between τ 1 and τ 2 in the list (L) of [25] do not appear as growth rates of compact Coxeter polygons in H 2 .This holds, for example, for the fifth smallest Salem number ≈ 1.216391 with minimal polynomial t 10 − t 6 − t 5 − t 4 + 1.
For n = 3, we know by [20,Section 3] that the minimal growth rate is the Salem number with minimal polynomial t 10 − t 9 − t 6 + t 5 − t 4 − t + 1 and belongs to the Coxeter tetrahedron [3,5,3].Since τ [3,5,3] ≈ 1.350980, it follows that none of the first 47 smallest Salem numbers as listed in (L) appear as growth rates of compact Coxeter polyhedra in H 3 .
Suppose that n ≥ 4. Our strategy is to show that the growth rate τ P of a compact Coxeter polyhedron P ⊂ H n is either not a Salem number or satisfies τ P > τ [3,8] .We distinguish between the two cases that (i) P has mutually intersecting facets or that (ii) the polyhedron P has a pair of disjoint facets.
In case (i), we know by Example 4 that P is of dimension four and equals either one of the five Coxeter simplices or one of the seven Esselmann polyhedra.By Remark 2, the growth rates of the five Coxeter 4-simplices are not Salem numbers anymore.For the seven Esselmann polyhedra E i , the growth functions f i (t) , 1 ≤ i ≤ 7 , can be determined by means of Steinberg's formula; see also [27, pp. 89-90].This can also be achieved by using the software CoxIter [11].Then, by analysing the sign changes of the denominator polynomials of f (t), one checks that the biggest real pole lies -roughly -between 1.90 and 2.61.The smallest growth rate among the Esselmann polyhedra is ≈ 1.902812 and belongs to the group with Coxeter symbol [8,3,4,3,8].Hence, for 1 ≤ i ≤ 7, the growth rate τ i of E i satisfies τ i > τ [3,8] .

Remark 5
The proof of Theorem 1 shows that if the growth rate τ P of a compact hyperbolic Coxeter polyhedron P of dimension n ≥ 4 happens to be a Salem number, then τ P > τ [3,8] .However, there is no example known to date where this is the case.Recall that if all the weights of the edges of a Coxeter diagram Γ are equal to 3, then the Coxeter adjacency matrix equals the adjacency matrix of the underlying abstract graph.In this case, we will sometimes simply write "graph" instead of "Coxeter diagram".We will also write "Coxeter tree" to stress the case where the underlying abstract graph of a Coxeter diagram is a tree.
We now define specific Coxeter diagrams that we use in this section.
Let k ≥ 3 and p 1 , . . ., p k ≥ 2 be integers.Recall that the star graph Star(p 1 , . . ., p k ) is defined to be the tree with one vertex of valency k that has k outgoing paths of respective lengths p i − 1.For example, Star(2, 3, 7) the graph of the Coxeter group E 10 .

Suppose now that
and consider the Coxeter diagram Γ(p 1 , . . ., p k ) of the compact hyperbolic Coxeter polygon P = (p 1 , . . ., p k ) ⊂ H 2 ; see Example 1. Recall that its growth rate τ is a Salem number.
The goal of the section is to prove our second main result as stated by Theorem 2 in Section 1.
For a Coxeter tree Γ, we denote by C Γ the associated Coxeter transformation and by Φ Γ (t) its characteristic polynomial.Given a graph Γ and a vertex v of Γ, we denote by Γ−v the subgraph obtained by deleting v and all its adjacent edges from Γ.A leaf of a tree is a vertex of valency one.
Lemma 2 Let Γ be a Coxeter tree and let v be a leaf.Let v be the unique vertex adjacent to v, and let m be the weight of the edge connecting v and v .Then we have the following identity in R[t]: Proof Since the conjugacy class of the Coxeter transformation does not depend on the Coxeter element, we are free to choose the bipartite Coxeter transformation for our calculations.More precisely, we partition the vertices of Γ into two sets V 1 and V 2 so that all edges of Γ connect a vertex from V 1 with a vertex in V 2 .We then choose an ordering of the vertices of Γ so that all vertices in V 1 appear before the vertices of V 2 , and take the corresponding Coxeter element.See Figure 3 for an example of a bipartite ordering of the vertices of a tree.Now, let 0 X X 0 be the adjacency matrix with respect to our chosen bipartite ordering of the vertices.For the Coxeter transformation, we then get that For the characteristic polynomial, we obtain The matrix we are taking the determinant of can be chosen to have the form , where the two middle columns and rows correspond to the vertices v and v .The assertion follows by developing the column and the row that correspond to the vertex v.
We now specialise Lemma 2 to the trees given by star graphs Star(p 1 , . . ., p k ).In order to do so, we first simplify our notation as follows.Let C p 1 ,...,p k be the Coxeter transformation of Star(p 1 , . . ., p k ), and denote by Φ p 1 ,...,p k (t) the characteristic polynomial of C p 1 ,...,p k .The following statements follow directly from Lemma 2. Lemma 3 Let p 1 , . . ., p k ≥ 2 be integers.
(1) If p k ≥ 4, we have the following equality in Z[t]: (2) If p k = 3, we have the following equality in Z[t]:

Example 6
The Coxeter transformation of the star graph with k ≥ 1 arms of length one has characteristic polynomial (t + 1) k−1 (t 2 − (k − 2)t + 1).This can be verified, for example, by an inductive argument and Lemma 2.
Definition 1 For integers p 1 , . . ., p k ≥ 2, define ∆ p 1 ,...,p k (t) ∈ Z(t) by Recall that for k ≥ 3 and the growth function f P (t) of a compact hyperbolic polygon P = (p 1 , . . ., p k ) is reciprocal and given by (see ( 6)-( 8)) . In order to prove Proposition 2, we establish recursion formulas for the polynomial ∆ p 1 ,...,p k (t).These recursion formulas have the same form as the recursion formulas we gave for the polynomial Φ p 1 ,...,p k (t) in Lemma 3.This is the content of the following lemma.
(1) If p k ≥ 4, we have the following equality in Z[t]: (2) If p k = 3, we have the following equality in Z[t]: Proof We have where the polynomials A = ∆ p 1 ,...,p k−1 (t) − k−1 i=1 [p i ] and B = k−1 i=1 [p i ] do not depend on p k .We now calculate

Proof of Proposition 2
First of all, we note that we can permute the p i without changing ∆ p 1 ,...,p k (t) or Φ p 1 ,...,p k (t).Repeatedly using the recursion formulas (1) of Lemma 3 and Lemma 4, the statement can hence be reduced to the class of cases where all p i are in the set {2, 3}.Within this class, we now proceed by induction on the number N of p i = 3.
We have two base cases, N = 0 and N = 1.If N = 0, we are in the case k arbitrary and p i = 2 for all i.In this case, we are done by the Examples 6 and 7.If N = 1, there are two possibilities to consider.
we use the recursion formulas (2) of Lemma 3 and Lemma 4 to reduce the statement to the case N = 0.
For the inductive step, we assume that N ≥ 2. As before, we use the recursion formulas (2) of Lemma 3 and Lemma 4 to reduce the statement to the case N −1.This finishes the proof.
We are now ready to prove our second main result.
Proof of Theorem 2 By definition, the spectral radius of the Coxeter transformation of the star graph Star(p 1 , . . ., p k ) equals the absolute value of the largest root of Φ p 1 ,...,p k (t).By Proposition 2, this in turn equals the absolute value of the largest root of ∆ p 1 ,...,p k (t).
consider the growth series f (t) of the compact hyperbolic Coxeter polygon P = (p 1 , . . ., p k ).Since the denominator polynomial of the rational function f (t) equals ∆ p 1 ,...,p k (t), the growth rate τ of P, as given by the inverse of the radius of convergence of f (t), also equals the absolute value of the largest root of ∆ p 1 ,...,p k (t).
5 The tetrahedral groups [3,5,3] and [4,3,5] In the previous section, we have shown that the growth rates of planar hyperbolic Coxeter groups are spectral radii of Coxeter transformations.In this section, we start the investigation of this property in dimension three.By giving both an example of a growth rate that is the spectral radius of a Coxeter transformation and an example that is not, we illustrate that the question becomes more difficult.
Definition 2 For integers i, k ≥ 2 and j ≥ 1, let H(i, j, k) be the connected tree with two vertices v 1 , v 2 of valency three that are connected by a path of length j.Furthermore, v 1 has two additional outgoing paths: one of length i − 1 and one of length 1.Similarly, v 2 has two additional outgoing paths: one of length k − 1 and one of length 1.For example, Figure 3 depicts the graph H(2, 8, 3).
Consider the compact Coxeter tetrahedron [4,3,5] (see also Remark 1).The associated growth rate τ [4,3,5] is the Salem number with minimal polynomial This follows from work of Parry [26] and can be conveniently verified by means of the software CoxIter [11].In particular, the growth rate τ [4,3,5] is the largest root of p(t) and is ≈ 1.359999.This number equals the spectral radius of the Coxeter transformation associated with the graph H(2, 8, 3), depicted in Figure 3.In fact, using Equations ( 15)-( 17) from the proof of Lemma 2, one can compute the characteristic polynomial of the Coxeter transformation of H(2, 8, 3) as the determinant of a block matrix involving identity matrices and the matrix X which is defined via a bipartite adjacency matrix of the graph.In the case of H(2, 8, 3), with the numbering of the vertices indicated in Figure 3, the matrix X becomes 1 1 1 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 1 1 1 0 0 0 0 0 0 0 1 A straightforward computation yields that the characteristic polynomial of the Coxeter transformation is given by In particular, we see that the spectral radius of the Coxeter transformation associated with H(2, 8, 3) equals the Salem number with minimal polynomial p(t).

Proposition 3
The growth rate of the tetrahedral group [3,5,3] is not equal to the spectral radius of a Coxeter transformation.
The growth rate τ [3,5,3] of the compact Coxeter tetrahedron [3,5,3] is equal to the Salem number λ 0 ≈ 1.350980 with minimal polynomial t 10 − t 9 − t 6 + t 5 − t 4 − t + 1, (see Section 3.1).Our proof of Proposition 3 is based on McMullen's classification of minimal hyperbolic Coxeter systems (see Section 3.1).Firstly, we note that by Proposition 7.5 of McMullen [24], if a Coxeter transformation has spectral radius smaller than the golden ratio (1 + √ 5)/2, then it is the Coxeter transformation associated with a Coxeter diagram whose underlying abstract graph is a tree, with no restriction on edge weights.We need a slightly stronger statement, given by the following lemma.
Lemma 5 If λ < 1.35999 is the spectral radius of a Coxeter transformation, then λ is also the spectral radius of a Coxeter transformation of a tree with constant edge weights all equal to 3.
Proof By Proposition 7.5 of McMullen [24], we know that λ must be the spectral radius of a Coxeter transformation of a tree.We now want to show that we can assume the tree to have constant edge weights all equal to 3. To this end we assume that λ < 1.35999 is the spectral radius of a Coxeter transformation of a tree Γ with at least one edge weight ≥ 4. By McMullen's classification of the 38 minimal hyperbolic Coxeter diagrams [24], the Coxeter tree Γ must dominate either the Star (2,4,5) or the Star(2, 3, 7).Indeed, the Coxeter transformation of all the other minimal hyperbolic Coxeter trees have larger spectral radii.Now, at least one edge of Γ must have weight ≥ 4. The only possibility for this to happen is if the weight is exactly 4 and the edge weighted 4 is adjacent to a leaf of Γ.Indeed, in all other cases, a minimal diagram given in [24,Table 5] other than Star (2,4,5) or Star(2, 3, 7) would be dominated by Γ, and hence the spectral radius would have to be larger than 1.35999.To finish the proof, we use the values of spectral radii of adjacency matrices depicted in Table 1.
We first deal with the cases of the star graphs, and use that the spectral radii of adjacency matrices are monotonic with respect to taking subgraphs, see, for example, Proposition 3.1.1 in [2].
In order to treat the case 1(a), we note that the sequence of spectral radii of the adjacency matrices of the graphs H(2, j, 3) is monotonically decreasing in j.This follows from the fact that subdividing an edge that does not lie on an endpath does not increase the spectral radius, see, for example, Proposition 3.1.4in [2].From Table 1 we obtain that H(2, 10, 3) is smaller than α 0 and hence so is the spectral radius of H(2, j, 3) for all j ≥ 10.In particular, since every star of the type (2, 3, r) is the subgraph of a graph H(2, j, 3) for j large enough, also the spectral radii of the stars (2, 3, r) are smaller than α 0 .
We now deal with the graphs of type H(i, j, k).
As soon as i or k is ≥ 4 and j = 3, it follows by the classification that the star of type (2,4,6) is a subgraph.Hence the spectral radius is > α 0 , which must also be the case for j = 3, since the spectral radius is monotonically decreasing in j.This eliminates many cases.Up to graph isomorphism, the only cases that we still have to consider are the following ones.
In both cases, the spectral radius is again a decreasing sequence in the parameter j.Hence, in both cases the values given in Table 1 suffice to exclude that a spectral radius of the adjacency matrix of a graph of type H(i, j, k) equals α 0 .This concludes the proof.
We are now ready to prove Proposition 3.

Proof of Proposition 3
We want to show that λ 0 ≈ 1.350980 is not the spectral radius of a Coxeter transformation.By Lemma 5, if λ 0 was the spectral radius of a Coxeter transformation, then there would exist a tree Γ with constant edge weights all equal to 3 and such that λ 0 is the spectral radius of the Coxeter transformation associated with Γ.The spectral radius λ of the Coxeter transformation is related to the spectral radius α of the adjacency matrix of Γ by the identity (2).We note that in the case of constant edge weights equal to 3, the Coxeter adjacency matrix equals the graph-theoretic adjacency matrix of the tree.But then, by Lemma 6, the equation ( 2) does not have a solution among trees for λ = λ 0 .It follows that λ 0 cannot be the spectral radius of a Coxeter transformation.

Figure 2 :
Figure 2: A sequence of ordered Coxeter graphs.

4
The spectral radius of a Coxeter transformationConsider an (abstract) Coxeter system (W, S) of rank N with finite generating set S = {s 1 , . . ., s N } and associated Coxeter diagram Γ. Denote by c ∈ W a Coxeter element and represent it geometrically by its Coxeter transformation C = ρ(c) ∈ GL N (V).Recall that all Coxeter elements are conjugate in W if Γ is a tree.In this case, the spectral radius λ of C is related to the leading eigenvalue α of the adjacency matrix of Γ by α 2 = 2 + λ + λ −1 ; see Section 2.1.
, then we join the nodes v i , v j by a bold edge (omitting the weight ∞); if H i , H j are at distance l ij > 0 in H n , then v i , v j are joined by a dotted edge (usually without the weight l ij ).In [9, Theorem A], Felikson and Tumarkin showed that the Coxeter diagram of a compact Coxeter polyhedron in H n has always a dotted edge if n ≥ 5 (compare also with Example 3 below).In the case that the Coxeter polyhedron P is bounded by only a few hyperplanes, its description by the Coxeter symbol is more convenient.For example, [p 1 , . . ., p k ] or [q 1 , . . ., q l , ∞] with integer labels p i , q j ≥ 3 are associated with linear Coxeter diagrams with k or l + 1 edges marked by the respective weights.The Coxeter symbol [(p k , q)] describes a cyclic Coxeter diagram with k ≥ 1 consecutive edge weights p followed by the weight q; see [17, Appendix], for example.
Coxeter polyhedron P and its Coxeter group G = (G, S) by means of their Coxeter diagram Γ as follows.If H i , H j meet at ∂H n Example 1 Let k ≥ 3 and p 1 , . . ., p k ≥ 2 be integers.A compact Coxeter k-gon P = (p 1 , . . ., p k ) with interior angles π/p 1 , . . ., π/p k exists in H 2 if and only if its (normalised) angle sum satisfies 1 p 1 a hyperbolic Coxeter group G = (G, S) with set S of generating reflections in hyperplanes of H n as usual.By [6, Corollary, p. 376], if the Coxeter polyhedron P of G is compact, then the rational function f (t) = f S (t) is reciprocal (resp.anti-reciprocal) for n even (resp.n odd), that is, [2]8]econd smallest growth rate τ 2 of a compact Coxeter polygon in H 2 is realised in a unique way by the triangle with Coxeter symbol[3,8].The Salem number τ 2 ≈ 1.230391 has minimal polynomial t 10 − t 7 − t 5 − t 3 + 1 and is the seventh smallest Salem number in the list (L).That is, the vertex stabiliser G v ⊂ G is the dihedral group D kv 2 of order 2k v ≥ 4, with growth polynomial[2][k v ].As a consequence, [3,8]sition 1Proof The strategy of the proof is similar to the one given for τ 1 in [20, Section 4.1].By Steinberg's formula 4 (see also(8)), the growth function f[3,8](t) of the Coxeter triangle group[3,8]equals f [3,8] (t) = p(t) t 10 − t 7 − t 5 − t 3 + 1 , with a certain numerator polynomial p(t) ∈ Z[t].Let P ⊂ H 2 be a compact Coxeter polygon with number of vertices f 0 and associated Coxeter group G. Denote by π/k v the interior angle at the vertex v in P.
This polynomial equals the characteristic polynomial of the Coxeter transformation of the star graph with k ≥ 3 arms of length one, see Example 6.Let p 1 , . .., p k ≥ 2 be integers.We have the following equality in Z[t] :∆ p 1 ,...,p k (t) = Φ p 1 ,...,p k (t).

Table 1 :
Some approximate spectral radii of graphs.