1 Introduction

A lattice \(\Lambda \) is a discrete subgroup of \(\mathbb {R}^m\), for some positive integer m. Every lattice \(\Lambda \) has a basis \(B=\{\textbf{b}_1,\textbf{b}_2,\dots , \textbf{b}_n\}\), and each point of \(\Lambda \) may be represented as the linear sum of its basis vectors over \(\mathbb {Z}\), that is,

$$\begin{aligned} \Lambda =\Lambda (B)=\left\{ \sum _{i=1}^n x_i \textbf{b}_i: x_i \in \mathbb {Z}\right\} . \end{aligned}$$

We say that \(\Lambda \) is of rank n if \(\textbf{b}_1,\dots ,\textbf{b}_n\) are linearly independent over \(\mathbb {R}\) and form a basis for \(\mathbb {R}^n\), and we say that \(\Lambda \) is full-rank if \(n=m\).

The process of moving from a “bad” lattice basis (one in which the vectors are relatively long, non-orthogonal) to a “good” basis (relatively short and nearly orthogonal vectors) is known as lattice reduction. There are many different definitions of what constitutes a reduced lattice basis, and the focus of this paper will be on so-called HKZ reduced bases, named for mathematicians C. Hermite, A. Korkin and G. Zolotarev [1, 2]. Denote by

$$\begin{aligned}&\textbf{b}_i(j)=\textbf{b}_i-\sum _{k=1}^{j-1}\mu _{i,k}\textbf{b}_k(k), \hspace{2mm} \forall 1\le j \le i \le n, \\&\textbf{b}_1(1)=\textbf{b}_1, \\&\mu _{i,j}=\frac{\langle \textbf{b}_i,\textbf{b}_j(j)\rangle }{\Vert \textbf{b}_j(j)\Vert ^2}, \hspace{2mm} \forall 1 \le j <i \le n. \end{aligned}$$

Then \(B=\{\textbf{b}_1,\textbf{b}_2,\dots ,\textbf{b}_n\}\) is said to be HKZ reduced if the following properties hold:

  • \(|\mu _{i,j}| \le 1/2\), for all \(1 \le j < i \le n\),

  • \(\textbf{b}_1\) is the shortest nonzero lattice vector of \(\Lambda (B)\),

  • \(\{\textbf{b}_2(2),\textbf{b}_3(2),\dots ,\textbf{b}_n(2)\}\) is HKZ reduced.

HKZ reduction is very useful both in theory and in practice. For example, it is used in the study of quadratic forms [10] (see also [9] for the application of a variation of HKZ). HKZ reduction is also widely used in coding theory and lattice-based cryptography to solve the closest vector problem (see, e.g., [5, 8]). HKZ reduction also has applications in the investigation of lattice packings of spheres, see e.g. [11].

The ith successive minimum of a lattice \(\Lambda \), denoted by \(\lambda _i\), is the smallest real number such that there are i linearly independent vectors in \(\Lambda \) of length at most \(\lambda _i\).

Proposition 1

([2]) If \(B=\{\textbf{b}_1,\textbf{b}_2,\dots ,\textbf{b}_n\}\) is HKZ reduced, then for all \(1 \le i \le n-1\),

$$\begin{aligned} \Vert \textbf{b}_i(i)\Vert ^2 \le \frac{4}{3}\Vert \textbf{b}_{i+1}(i+1)\Vert ^2, \end{aligned}$$

and for all \(1 \le i \le n-2\),

$$\begin{aligned} \Vert \textbf{b}_i(i)\Vert ^2 \le \frac{3}{2}\Vert \textbf{b}_{i+2}(i+2)\Vert ^2. \end{aligned}$$

Proposition 2

([3]) If \(B=\{\textbf{b}_1,\textbf{b}_2,\dots ,\textbf{b}_n\}\) is HKZ reduced, then for all \(1 \le i \le n\),

$$\begin{aligned} \frac{4}{i+3} \lambda _i^2 \le \Vert \textbf{b}_i\Vert ^2 \le \frac{i+3}{4} \lambda _i^2. \end{aligned}$$

We define the Hermite invariant of a lattice \(\gamma (B)\) by

$$\begin{aligned} \gamma (B)=\frac{\lambda _1^2}{|\det (B)|^{\frac{2}{n}}}, \end{aligned}$$

and the Hermite constant of rank n, denoted by \(\gamma _n\), is the supremum of all Hermite invariants for rank n lattice bases.

As mentioned, the process of reduction is attaining a basis for a lattice that has “desirable” properties. One way of measuring the quality of a basis is by evaluating its orthogonality defect, which is defined by

$$\begin{aligned} \Delta (B)=\frac{\prod _{i=1}^n \Vert \textbf{b}_i\Vert ^2}{\det (B)^2}. \end{aligned}$$

It is easily seen that the orthogonality defect is not unique to a lattice, and a basis can be transformed so that the orthogonality defect becomes arbitrarily large. However, it is known that the orthogonality defect can be bounded if the basis is HKZ reduced.

Theorem 1

([3]) If \(B=\{\textbf{b}_1,\textbf{b}_2,\dots ,\textbf{b}_n\}\) is HKZ reduced, then

$$\begin{aligned} \Delta (B) \le \gamma _n^n \prod _{i=1}^n \frac{i+3}{4}. \end{aligned}$$

Whilst Lagarias et. al. managed to show that the orthogonality defect of HKZ reduced bases must be bounded, little is known about the exact value of the supremum of the orthogonality defect for HKZ reduced basis of a given rank. In this work, we calculate this supremum up to dimension 3 HKZ reduced bases, and use this result to improve the bound in Theorem 1.

Theorem 2

Let \(H_n\) denote the space of rank n HKZ reduced lattice bases. Set

$$\begin{aligned} \Delta _n=\max _{B \in H_n} \Delta (B). \end{aligned}$$

Then \(\Delta _1=1\), \(\Delta _2=\frac{4}{3}\), \(\Delta _3=\frac{25}{12}\).

Theorem 3

Using the notation as before, for all \(n \ge 4\),

$$\begin{aligned} \Delta _n \le \frac{25}{12} \gamma _{n-3}^{n-3} \prod _{i=4}^n \left( \frac{i}{4}+\frac{29}{24}\right) . \end{aligned}$$

The orthogonality defect is a quantity that is of significant importance in both the fields of post-quantum cryptography and coding theory [5]. Many post-quantum cryptosystems make use of lattices, and achieving a basis of a lattice that is “relatively orthogonal” makes breaking the cryptosystem significantly easier than being given a relatively non-orthogonal basis. Roughly, the orthogonality defect of a basis is an indication of how orthogonal the basis is as a whole - the lower the orthogonality defect, the closer the basis vectors are to being orthogonal. Moreover, it has been shown that the complexity of quantum lattice enumeration algorithms is directly linked to the orthogonality defect [6, 7]. Achieving bases with small orthogonality defect is therefore of vital importance in these fields.

2 Proof of Theorem 2

The value of \(\Delta _1\) is trivial. For \(\Delta _2\), for any basis \(B=\{\textbf{b}_1,\textbf{b}_2\}\),

$$\begin{aligned} \Delta (B)=\frac{\Vert \textbf{b}_2\Vert ^2}{\Vert \textbf{b}_2(2)\Vert ^2}=\frac{1}{1-\frac{\langle \textbf{b}_2,\textbf{b}_1\rangle ^2}{\Vert \textbf{b}_1\Vert ^2\Vert \textbf{b}_2\Vert ^2}} \le \frac{1}{1-\mu _{2,1}^2} \le \frac{4}{3}, \end{aligned}$$

and this upper bound is attained by the \(A_2\) lattice.

We now draw our attention to the case where \(n=3\). We will assume without loss of generality that \(\Vert \textbf{b}_1\Vert ^2=1\) (the value of \(\Vert \textbf{b}_1\Vert ^2\) will not affect the orthogonality defect of the lattice), and use the following notations:

$$\begin{aligned} \Vert \textbf{b}_2(2)\Vert ^2=k, \Vert \textbf{b}_3(3)\Vert ^2=l, \mu _{21}=\lambda , \mu _{31}=\mu , \mu _{32}=\sigma . \end{aligned}$$

Then,

$$\begin{aligned} \Vert x\textbf{b}_1+y\textbf{b}_2+z\textbf{b}_3\Vert ^2=(x+\lambda y +\mu z)^2+k(y+\sigma z)^2 +lz^2. \end{aligned}$$

By switching the signs of y, z we may assume without loss of generality that \(\lambda ,\mu \ge 0\). It is easily seen that we have,

$$\begin{aligned}&\frac{\Vert \textbf{b}_2\Vert ^2}{\Vert \textbf{b}_2(2)\Vert ^2}=1+\frac{\lambda ^2}{k}, \\&\frac{\Vert \textbf{b}_3\Vert ^2}{\Vert \textbf{b}_3(3)\Vert ^2}=1+\frac{1}{l}(\mu ^2+k\sigma ^2). \end{aligned}$$

Since the basis is HKZ reduced, the following inequalities hold:

$$\begin{aligned}&k+\lambda ^2 \ge 1, \end{aligned}$$
(1)
$$\begin{aligned}&l+k\sigma ^2+\mu ^2 \ge 1, \end{aligned}$$
(2)
$$\begin{aligned}&l+k(1+\sigma )^2+(1-\lambda -\mu )^2 \ge 1, \end{aligned}$$
(3)
$$\begin{aligned}&l+k(1-\sigma )^2+(\lambda -\mu )^2 \ge 1 \end{aligned}$$
(4)
$$\begin{aligned}&l+k\sigma ^2 \ge k. \end{aligned}$$
(5)

From this, we may ascertain the following inequalities:

$$\begin{aligned}&k \le \frac{l}{1-\sigma ^2}, \end{aligned}$$
(6)
$$\begin{aligned}&\lambda ^2 \ge 1-\frac{l}{1-\sigma ^2}, \end{aligned}$$
(7)
$$\begin{aligned}&\mu ^2 \ge 1-\frac{l}{1-\sigma ^2}. \end{aligned}$$
(8)

Now, using the inequalities \(l \ge \frac{2}{3},k \ge \frac{3}{4}\), if \(|\sigma | \le 1/3\) we have

$$\begin{aligned} \Delta (B)&=\left( 1+\frac{\lambda ^2}{k}\right) \left( 1+\frac{1}{l}(\mu ^2+k\sigma ^2)\right) \le \left( 1+\frac{1}{4k}\right) \left( 1+\frac{\frac{1}{4}+\frac{9}{8}\frac{1}{9}l}{l}\right) \\&=\left( 1+\frac{1}{4k}\right) \left( \frac{9}{8}+\frac{1}{4l}\right) \le \frac{4}{3}\frac{3}{2}=\gamma _3^3, \end{aligned}$$

so we may assume without loss of generality that \(1/3 \le |\sigma | \le 1/2\). Assume first that \(\sigma \le -1/3\). By inequalities 3 and 6, we have

$$\begin{aligned} l \ge \max \{1-(1-\lambda -\mu )^2-k(1+\sigma )^2,k(1-\sigma ^2)\}. \end{aligned}$$

Assume first that

$$\begin{aligned} 1-(1-\lambda -\mu )^2-k(1+\sigma )^2 \ge k(1-\sigma ^2) \iff k \le \frac{1}{2(1+\sigma )}\left( 1-(1-\lambda -\mu )^2\right) . \end{aligned}$$
(9)

Then

$$\begin{aligned} \Delta (B) \le \left( 1+\frac{\lambda ^2}{k}\right) \left( 1+\frac{\mu ^2+k\sigma ^2}{1-(1-\lambda -\mu )^2-k(1+\sigma )^2}\right) . \end{aligned}$$

Let

$$\begin{aligned} f(k)=\left( 1+\frac{\lambda ^2}{k}\right) \left( 1+\frac{\mu ^2+k\sigma ^2}{1-(1-\lambda -\mu )^2-k(1+\sigma )^2}\right) , \end{aligned}$$

so

$$\begin{aligned} f^{\prime \prime }(k)=\frac{\alpha }{k^3(1-(1-\lambda -\mu )^2-k(1+\sigma )^2)^3}, \end{aligned}$$

where

$$\begin{aligned} \alpha&=2\lambda ^2(\mu ^2+k\sigma ^2)(1-(1-\lambda -\mu )^2-k(1+\sigma )^2)^2+2\lambda ^2(1-(1-\lambda -\mu )^2-k(1+\sigma )^2)^3\\&\quad -2\lambda ^2k(1+\sigma )^2(\mu ^2+k\sigma ^2)(1-(1-\lambda -\mu )^2-k(1+\sigma )^2)\\&\quad -2k\lambda ^2\sigma ^2(1-(1-\lambda -\mu )^2-k(1+\sigma )^2)^2\\&\quad +2\lambda ^2k^2(1+\sigma )^4(\mu ^2+k\sigma ^2)+2k^3(1+\sigma )^4(\mu ^2+k\sigma ^2)\\&\quad +2\lambda ^2k^2\sigma ^2(1+\sigma )^2(1-(1-\lambda -\mu )^2-k(1+\sigma )^2)\\&\quad +2k^3\sigma ^2(1+\sigma )^2 \\&=\lambda ^2(\mu ^2+k\sigma ^2)(1-(1-\lambda -\mu )^2-2k(1+\sigma )^2)^2\\&\quad +\lambda ^2(1-(1-\lambda -\mu )^2-k(1+\sigma )^2)(1-(1-\lambda -\mu )^2-k((1+\sigma )^2+\sigma ^2))^2 \\&\quad +\lambda ^2(\mu ^2+k\sigma ^2)(1-(1-\lambda -\mu )^2-k(1+\sigma )^2)^2+\lambda ^2(1-(1-\lambda -\mu )^2-k(1+\sigma )^2)^3 \\&\quad +\lambda ^2k^2(1+\sigma )^4(\mu ^2+k\sigma ^2)+2k^3(1+\sigma )^4(\mu ^2+k\sigma ^2)\\&\quad +\lambda ^2k^2(2\sigma ^2(1+\sigma )^2-\sigma ^4)(1-(1-\lambda -\mu )^2-k(1+\sigma )^2)+2k^3\sigma ^2(1+\sigma )^2 \ge 0, \end{aligned}$$

since

$$\begin{aligned} 2\sigma ^2(1+\sigma )^2-\sigma ^4=\sigma ^2(\sigma ^2+4\sigma +2)>0 \end{aligned}$$

for all \(|\sigma |\le 1/2\), and so \(f^{\prime \prime }(k)\) is non-negative for all values of \(\lambda ,k,\sigma , \mu \) over their respective regions, so the maximum value of f(k) occurs either when k is maximum or when it is minimum. Suppose first that f(k) is maximum when k is minimum. Then

$$\begin{aligned} \Delta (B)&\le \left( 1+\frac{\lambda ^2}{1-\lambda ^2}\right) \left( 1+\frac{\mu ^2+\sigma ^2(1-\lambda ^2)}{1-(1-\lambda -\mu )^2-(1-\lambda ^2)(1+\sigma )^2}\right) . \end{aligned}$$

Let’s assume a contradiction, so

$$\begin{aligned} \left( 1+\frac{\lambda ^2}{1-\lambda ^2}\right) \left( 1+\frac{\mu ^2+\sigma ^2(1-\lambda ^2)}{1-(1-\lambda -\mu )^2-(1-\lambda ^2)(1+\sigma )^2}\right) \ge \frac{25}{12}, \end{aligned}$$
(10)

for some values of \(\mu ,\sigma ,\lambda \). Rearranging gives us the quadratic inequality in \(\sigma \):

$$\begin{aligned}&\frac{25}{12}(1-\lambda ^2)^2\sigma ^2+\left( -2(1-\lambda ^2)+\frac{25}{6}(1-\lambda ^2)^2\right) \sigma \\&\quad +1-(1-\lambda -\mu )^2-\frac{37}{12}(1-\lambda ^2)+\mu ^2\\&\quad +\frac{25}{12}(1-\lambda ^2)(1-\lambda -\mu )^2+\frac{25}{12}(1-\lambda ^2)^2 \ge 0 \end{aligned}$$

Denote by \(r^+,r^-\) respectively the greater and lesser roots of the equation above. Then we need either

$$\begin{aligned} \sigma \ge r^+, \hspace{2mm} \sigma \le r^-. \end{aligned}$$

However, we can show that the solutions to this equation yield contradictions. Using 9, we have

$$\begin{aligned} 1-\lambda ^2 \le 1-(1-\lambda -\mu )^2, \end{aligned}$$

and so the function is only valid under the boundaries

$$\begin{aligned}&0 \le \lambda \le 1/2, \end{aligned}$$
(11)
$$\begin{aligned}&1-2\lambda \le \mu \le 1/2, \end{aligned}$$
(12)

Using these boundaries, we can numerically show that there cannot exist a valid solution as we would require \(\sigma >-1/3\) for \(\sigma \ge r^+\), or \(\sigma <-1/2\) for \(\sigma \le r^-\). Hence, the inequality 10 cannot exist under the assumptions made.

Now assume that the maximum of f(k) occurs when k is maximum, so

$$\begin{aligned} \Delta (B) \le \left( 1+\frac{2\lambda ^2(1+\sigma )}{1-(1-\lambda -\mu )^2}\right) \left( 1+\frac{\mu ^2+\frac{\sigma ^2}{2(1+\sigma )}(1-(1-\lambda -\mu )^2)}{\frac{1}{2}(1-\sigma )(1-(1-\lambda -\mu )^2)}\right) . \end{aligned}$$
(13)

Once again, we assume that we have a contradiction so

$$\begin{aligned} \left( 1+\frac{2\lambda ^2(1+\sigma )}{1-(1-\lambda -\mu )^2}\right) \left( 1+\frac{\mu ^2+\frac{\sigma ^2}{2(1+\sigma )}(1-(1-\lambda -\mu )^2)}{\frac{1}{2}(1-\sigma )(1-(1-\lambda -\mu )^2)}\right) \ge \frac{25}{12}, \end{aligned}$$

which yields the quadratic inequality

$$\begin{aligned}&\frac{25\lambda ^4+100\lambda ^3\mu +198\lambda ^2\mu ^2+100\lambda \mu ^3+25\mu ^4-100\lambda ^3-300\lambda ^2\mu -300\lambda \mu ^2-100\mu ^3+100\lambda ^2+200\lambda \mu +100\mu ^2}{12}\sigma ^2\\&\quad -2\left( \lambda ^4+2\lambda ^3\mu -2\lambda ^2\mu ^2+2\lambda \mu ^3+\mu ^4-2\lambda ^3-2\lambda ^2\mu -2\lambda \mu ^2-2\mu ^3\right) \sigma \\&\quad -\frac{37}{12}\lambda ^4-\frac{25}{3}\lambda ^3\mu -\frac{13}{2}\lambda ^2\mu ^2-\frac{25}{3}\lambda \mu ^3-\frac{37}{12}\mu ^4\\&\quad +\frac{25}{3}\lambda ^3+17\lambda ^2\mu +17\lambda \mu ^2+\frac{25}{3}\mu ^3-\frac{13}{3}\lambda ^2-\frac{26}{3}\lambda \mu -\frac{13}{3}\mu ^2 \ge 0 \end{aligned}$$

Once again, labelling the lesser and greater roots of the above quadratic inequality by \(r^{-},r^+\), we are able to numerically verify that, under the boundary conditions \(0 \le \lambda ,\mu \le 1/2\), there exists no solution to \(\sigma \ge r^{+}\) (under the assumption that \(|\sigma | \ge 1/3\)), and there exists a single exact solution for \(\sigma \le r^{-}\), namely \(\sigma =-1/2\) which is attained when \(\mu =\lambda =1/2\). However, this point attains the value \(\Delta (B)=\frac{25}{12}\) exactly, so this does not constitute a contradiction to our claim.

We may now assume instead that

$$\begin{aligned} 1-(1-\lambda -\mu )^2-k(1+\sigma )^2 \le k(1-\sigma ^2) \iff k \ge \frac{1}{2(1+\sigma )}(1-(1-\lambda -\mu )^2). \end{aligned}$$
(14)

Then

$$\begin{aligned} \Delta (B) \le \left( 1+\frac{\lambda ^2}{k}\right) \left( 1+\frac{\mu ^2+k\sigma ^2}{k(1-\sigma ^2)}\right) =\left( 1+\frac{\lambda ^2}{k}\right) \left( \frac{1}{1-\sigma ^2}+\frac{\mu ^2}{k(1-\sigma ^2)}\right) . \end{aligned}$$

Clearly the upper bound attains its maximum value when k is minimal, so by inequality 14,

$$\begin{aligned} \Delta (B) \le \left( 1+\frac{2\lambda ^2(1+\sigma )}{1-(1-\lambda -\mu )^2}\right) \left( \frac{1}{1-\sigma ^2}+\frac{2\mu ^2}{(1-\sigma )(1-(1-\lambda -\mu )^2)}\right) . \end{aligned}$$

However, this inequality is identical to the inequality in 13, and since we have already shown that this inequality does not rise above \(\frac{25}{12}\) in value for \(0 \le \lambda ,\mu \le 1/2\), we are also done for this case.

We have confirmed our claim for all \(-1/2 \le \sigma \le 1/3\), so now we assume that \(1/3 \le \sigma \le 1/2\). By inequalities 4, 6, we have

$$\begin{aligned} l \ge \max \{1-(\lambda -\mu )^2-k(1-\sigma )^2, k(1-\sigma ^2)\}. \end{aligned}$$

Assume first that

$$\begin{aligned} 1-(\lambda -\mu )^2-k(1-\sigma )^2 \ge k(1-\sigma ^2) \iff k \le \frac{1}{2(1-\sigma )}\left( 1-(\lambda -\mu )^2\right) . \end{aligned}$$
(15)

Then

$$\begin{aligned} \Delta (B) \le \left( 1+\frac{\lambda ^2}{k}\right) \left( 1+\frac{\mu ^2+k\sigma ^2}{1-(\lambda -\mu )^2-k(1-\sigma )^2}\right) . \end{aligned}$$

Let

$$\begin{aligned} F(k)=\left( 1+\frac{\lambda ^2}{k}\right) \left( 1+\frac{\mu ^2+k\sigma ^2}{1-(\lambda -\mu )^2-k(1-\sigma )^2}\right) . \end{aligned}$$

Then

$$\begin{aligned} F^{\prime \prime }(k)=\frac{\epsilon }{k^3(1-(\lambda -\mu )^2-k(1-\sigma )^2)^3}, \end{aligned}$$

where

$$\begin{aligned} \epsilon&=2\lambda ^2(\mu ^2+k\sigma ^2)(1-(\lambda -\mu )^2-k(1-\sigma )^2)^2+2\lambda ^2(1-(\lambda -\mu )^2-k(1-\sigma )^2)^3 \\&\quad -2\lambda ^2k(1-\sigma )^2(\mu ^2+k\sigma ^2)(1-(\lambda -\mu )^2-k(1-\sigma )^2)\\&\quad -2\lambda ^2k\sigma ^2(1-(\lambda -\mu )^2-k(1-\sigma )^2)^2\\&\quad +2\lambda ^2k^2(1-\sigma )^4(\mu ^2+k\sigma ^2)+2k^3(1-\sigma )^4(\mu ^2+k\sigma ^2)\\&\quad +2\lambda ^2k^2\sigma ^2(1-\sigma )^2(1-(\lambda -\mu )^2-k(1-\sigma )^2)\\&\quad +2k^3\sigma ^2(1-\sigma )^2(1-(\lambda -\mu )^2-k(1-\sigma )^2) \\&\quad =\lambda (\mu ^2+k\sigma ^2)(1-(\lambda -\mu )^2-2k(1-\sigma )^2)^2 \\&\quad +\lambda ^2(1-(\lambda -\mu )^2-k(1-\sigma )^2)(1-(\lambda -\mu )^2-k((1-\sigma )^2+\sigma ^2))^2 \\&\quad +\lambda ^2(\mu ^2+k\sigma ^2)(1-(\lambda -\mu )^2-k(1-\sigma )^2)^2+\lambda ^2(1-(\lambda -\mu )^2-k(1-\sigma )^2)^3 \\&\quad +\lambda ^2k^2(1-\sigma )^4(\mu ^2+k\sigma ^2)+2k^3(1-\sigma )^4(\mu ^2+k\sigma ^2) \\&\quad +\lambda ^2k^2\sigma ^2(1-\sigma )^2(1-(\lambda -\mu )^2-k(1-\sigma )^2)\\&\quad +k^3(2\sigma ^2(1-\sigma )^2-\sigma ^4)(1-(\lambda -\mu )^2-k(1-\sigma )^2) \\&\quad +2k^3\sigma ^2(1-\sigma )^2(1-(\lambda -\mu )^2-k(1-\sigma )^2) \ge 0, \end{aligned}$$

since

$$\begin{aligned} 2\sigma ^2(1-\sigma )^2-\sigma ^4=\sigma ^2(2-4\sigma +\sigma ^4) > 0, \end{aligned}$$

for all \(1/3 \le \sigma \le 1/2\), and so \(F^{\prime \prime }(k)\) is non-negative for all values of \(\lambda ,k,\sigma ,\mu \) over their respective regions, so the maximum value of F(k) occurs either when k is maximum or when it is minimum. Suppose first that F(k) is maximum when k is minimum. Then

$$\begin{aligned} \Delta (B) \le \left( 1+\frac{\lambda ^2}{1-\lambda ^2}\right) \left( 1+\frac{\mu ^2+(1-\lambda ^2)\sigma ^2}{1-(\lambda -\mu )^2-(1-\lambda ^2)(1-\sigma )^2}\right) . \end{aligned}$$

Assume that

$$\begin{aligned} \left( 1+\frac{\lambda ^2}{1-\lambda ^2}\right) \left( 1+\frac{\mu ^2+(1-\lambda ^2)\sigma ^2}{1-(\lambda -\mu )^2-(1-\lambda ^2)(1-\sigma )^2}\right) \ge \frac{25}{12}, \end{aligned}$$

which gives us the quadratic inequality

$$\begin{aligned}&\frac{25}{12}(1-\lambda ^2)^2\sigma ^2+\left( 2(1-\lambda ^2)-\frac{25}{6}(1-\lambda ^2)^2\right) \sigma \\&\quad +1-(\lambda -\mu )^2-\frac{37}{12}(1-\lambda ^2)+\mu ^2+\frac{25}{12}(1-\lambda ^2)(\lambda -\mu )^2+\frac{25}{12}(1-\lambda ^2)^2 \ge 0. \end{aligned}$$

Let \(r^{-},r^+\) respectively denote the lesser and greater roots of the quadratic ienquality above. By inequality 15,

$$\begin{aligned} 1-\lambda ^2 \le 1-(\lambda -\mu )^2, \end{aligned}$$

and so we obtain the boundary conditions

$$\begin{aligned}&0 \le \lambda \le 1/2, \end{aligned}$$
(16)
$$\begin{aligned}&0 \le \mu \le 2\lambda . \end{aligned}$$
(17)

Again, we need either \(\sigma \le r^{-}\) or \(\sigma \ge r^+\). However, we can numerically verify that under boundary conditions 16,17, the solution \(\sigma \le r^{-}\) requires \(\sigma <1/3\) and the solution \(\sigma \ge r^+\) requires that \(\sigma >1/2\), both of which are clearly contradictions under the assumptions made.

Now, assume that F(k) attains its maximum when k is maximum. Then

$$\begin{aligned} \Delta (B) \le \left( 1+\frac{2\lambda ^2(1-\sigma )}{1-(\lambda -\mu )^2}\right) \left( 1+\frac{\mu ^2+\frac{\sigma ^2}{2(1-\sigma )}(1-(\lambda -\mu )^2}{\frac{1}{2}(1+\sigma )(1-(\lambda -\mu )^2)}\right) . \end{aligned}$$
(18)

Assume that

$$\begin{aligned} \left( 1+\frac{2\lambda ^2(1-\sigma )}{1-(\lambda -\mu )^2}\right) \left( 1+\frac{\mu ^2+\frac{\sigma ^2}{2(1-\sigma )}(1-(\lambda -\mu )^2}{\frac{1}{2}(1+\sigma )(1-(\lambda -\mu )^2)}\right) \ge \frac{25}{12}. \end{aligned}$$

Rearranging gives us the quadratic inequality

$$\begin{aligned}&\frac{25\lambda ^4-100\lambda ^3\mu +198\lambda ^2\mu ^2-100\lambda \mu ^3+25\mu ^4-50\lambda ^2+100\lambda \mu -50\mu ^2+25}{12}\sigma ^2\\&\quad +2\left( \lambda ^4-2\lambda ^3\mu -2\lambda ^2\mu ^2-2\lambda \mu ^3+\mu ^4-\lambda ^2-\mu ^2\right) \sigma \\&\quad -\frac{37}{12}\lambda ^4+\frac{25}{3}\lambda ^3\mu -\frac{13}{2}\lambda ^2\mu ^2+\frac{25}{3}\lambda \mu ^3\\&\quad -\frac{37}{12}\mu ^4+\frac{25}{6}\lambda ^2-\frac{13}{3}\lambda \mu +\frac{25}{6}\mu ^2-\frac{13}{12} \ge 0. \end{aligned}$$

Again, letting \(r^{-},r^+\) denote the lesser and greater roots of the quadratic inequality above, we verify numerically that \(r^{-} < 1/3\) and \(r^+ \ge 1/2\) for all \(0 \le \mu , \lambda \le 1/2\), with equality \(\sigma =1/2\) only when \(\lambda =\mu =1/2\). We have \(\Delta (B)=\frac{25}{12}\) at the point \(\sigma =\mu =\lambda =1/2\) but does not exceed \(\frac{25}{12}\) at any other point, so this does not constitute a contradiction to our claim.

Finally, assume that

$$\begin{aligned} 1-(\lambda -\mu )^2-k(1-\sigma )^2 \le k(1-\sigma ^2) \iff k \ge \frac{1}{2(1-\sigma )}\left( 1-(\lambda -\mu )^2\right) , \end{aligned}$$

so

$$\begin{aligned} \Delta (B) \le \left( 1+\frac{\lambda ^2}{k}\right) \left( 1+\frac{\mu ^2+k\sigma ^2}{k(1-\sigma ^2)}\right) =\left( 1+\frac{\lambda ^2}{k}\right) \left( \frac{1}{1-\sigma ^2}+\frac{\mu ^2}{k(1-\sigma ^2)}\right) . \end{aligned}$$

Clearly the upper bound attains its maximum when k is minimal, so

$$\begin{aligned} \Delta (B) \le \left( 1+\frac{2\lambda ^2(1-\sigma )}{1-(\lambda -\mu )^2}\right) \left( \frac{1}{1-\sigma ^2}+\frac{2\mu ^2}{(1+\sigma )(1-(\lambda -\mu )^2)}\right) . \end{aligned}$$

However, the function determining the upper bound for \(\Delta (B)\) here is identical to that in 18, for which we have verified is less than or equal to \(\frac{25}{12}\) over all values of \(\sigma , \mu , \lambda \) in their relative intervals, so we do not need to deal with this case. Since this exhausts all possible values for all the variables, we have verified that \(\Delta _3=\frac{25}{12}\), and the quadratic form

$$\begin{aligned} \left( x+\frac{1}{2}(y+z)\right) ^2+\left( y \pm \frac{1}{2}z\right) ^2+\frac{3}{4}z^2 \end{aligned}$$

attains this upper bound, so this is the best possible upper bound on the orthogonality defect for three dimensional HKZ reduced bases.

3 Proof of Theorem 3

Suppose that \(B=\{\textbf{b}_1,\textbf{b}_2,\dots ,\textbf{b}_n\}\) is the basis for a rank \(n \ge 4\) lattice, and assume that B is HKZ reduced. Then note that the subbasis \(\{\textbf{b}_1,\textbf{b}_2,\textbf{b}_3\}\) must also be HKZ reduced, and so

$$\begin{aligned} \frac{\Vert \textbf{b}_1\Vert ^2\Vert \textbf{b}_2\Vert ^2\Vert \textbf{b}_3\Vert ^2}{\Vert \textbf{b}_1\Vert ^2\Vert \textbf{b}_2(2)\Vert ^2\Vert \textbf{b}_3(3)\Vert ^2} \le \frac{25}{12}. \end{aligned}$$

Then

$$\begin{aligned} \Delta (B)=\prod _{i=1}^n \frac{\Vert \textbf{b}_i\Vert ^2}{\Vert \textbf{b}_i(i)\Vert ^2} \le \frac{25}{12}\prod _{i=4}^n \frac{\Vert \textbf{b}_i\Vert ^2}{\Vert \textbf{b}_i(i)\Vert ^2}. \end{aligned}$$

Also note that the lattice generated by the basis \(\{\textbf{b}_4(4),\textbf{b}_5(4),\dots ,\textbf{b}_n(4)\}\) is HKZ reduced, by definition. For any \(i \ge 4\), we have

$$\begin{aligned} \Vert \textbf{b}_i\Vert ^2=\Vert \textbf{b}_i(4)\Vert ^2+\sum _{j=1}^3|\mu _{i,j}|^2\Vert \textbf{b}_j(j)\Vert ^2 \le \Vert \textbf{b}_i(4)\Vert ^2+\frac{1}{4}\sum _{j=1}^3\Vert \textbf{b}_j(j)\Vert ^2. \end{aligned}$$

Using Proposition 1,

$$\begin{aligned} \Vert \textbf{b}_1\Vert ^2 \le 2\Vert \textbf{b}_4(4)\Vert ^2, \Vert \textbf{b}_2(2)\Vert ^2 \le \frac{3}{2}\Vert \textbf{b}_4(4)\Vert ^2, \Vert \textbf{b}_3(3)\Vert ^2 \le \frac{4}{3}\Vert \textbf{b}_4(4)\Vert ^2, \end{aligned}$$

so

$$\begin{aligned} \Vert \textbf{b}_i\Vert ^2 \le \Vert \textbf{b}_i(4)\Vert ^2 +\frac{1}{4}\left( 2+\frac{3}{2}+\frac{4}{3}\right) \Vert \textbf{b}_4(4)\Vert ^2=\Vert \textbf{b}_i(4)\Vert ^2+\frac{29}{24}\Vert \textbf{b}_4(4)\Vert ^2. \end{aligned}$$

Lagarias et. al. proved that if B is HKZ reduced, then

$$\begin{aligned} \Vert \textbf{b}_i(i)\Vert ^2 \le \lambda _i^2, \end{aligned}$$

for all \(1 \le i \le n\), and so for all \(i \ge 4\), letting \(\lambda _k(4)\) denote the kth successive minima in the lattice generated by \(\{\textbf{b}_4(4),\textbf{b}_5(4),\dots ,\textbf{b}_n(4)\}\),

$$\begin{aligned} \Vert \textbf{b}_i\Vert ^2&\le \Vert \textbf{b}_i(4)\Vert ^2+\frac{29}{24}\Vert \textbf{b}_4(4)\Vert ^2 =\Vert \textbf{b}_i(i)\Vert ^2+\sum _{j=4}^{i-1} |\mu _{i,j}|^2\Vert \textbf{b}_j(j)\Vert ^2+\frac{29}{24}\Vert \textbf{b}_4(4)\Vert ^2 \\&\le \lambda _{i-3}(4)^2+\frac{1}{4}\sum _{j=4}^{i-1} \lambda _{j-3}(4)^2+\frac{29}{24}\lambda _1(4)^2 \le \left( \frac{i}{4}+\frac{29}{24}\right) \lambda _{i-3}(4)^2. \end{aligned}$$

Then, using Minkowski’s second theorem we get

$$\begin{aligned} \Delta (B) \le \frac{25}{12}\prod _{i=4}^n\frac{\Vert \textbf{b}_i\Vert ^2}{\Vert \textbf{b}_i(i)\Vert ^2} \le \frac{25}{12}\prod _{i=4}^n\left( \frac{i}{4}+\frac{29}{24}\right) \frac{\lambda _{i-3}(4)^2}{\Vert \textbf{b}_i(i)\Vert ^2} \le \frac{25}{12}\gamma _{n-3}^{n-3} \prod _{i=4}^n \left( \frac{i}{4}+\frac{29}{24}\right) , \end{aligned}$$

as required.

4 Summary and future work

In this work, we determined the exact maximal value that the orthogonality defect can take for HKZ reduced bases up to dimension 3, and used this result to calculate an upper bound on the orthogonality defect of HKZ bases of arbitrary rank. The latter result seems to be sharper than any other bounds on the orthogonality defect for HKZ reduced bases in existing literature, according to the authors’ knowledge.

In the future, we will investigate whether the techniques applied in this paper can be used to sharpen the bounds on the orthogonality defect further, and to determine the value of \(\Delta _n\) exactly for \(n \ge 4\). We conjecture that \(\Delta _4=\gamma _4^4\), and preliminary computational results seem to agree with this conjecture, which would be an interesting result, given that \(\Delta _3=\frac{25}{24}\gamma _3^3\).