A Formalization of the LLL Basis Reduction Algorithm
Abstract
The LLL basis reduction algorithm was the first polynomialtime algorithm to compute a reduced basis of a given lattice, and hence also a short vector in the lattice. It thereby approximates an NPhard problem where the approximation quality solely depends on the dimension of the lattice, but not the lattice itself. The algorithm has several applications in number theory, computer algebra and cryptography.
In this paper, we develop the first mechanized soundness proof of the LLL algorithm using Isabelle/HOL. We additionally integrate one application of LLL, namely a verified factorization algorithm for univariate integer polynomials which runs in polynomial time.
1 Introduction
The LLL basis reduction algorithm by Lenstra, Lenstra and Lovász [8] is a remarkable algorithm with numerous applications. There even exists a 500page book solely about the LLL algorithm [10]. It lists applications in number theory and cryptology, and also contains the best known polynomial factorization algorithm that is used in today’s computer algebra systems.
The LLL algorithm plays an important role in finding short vectors in lattices: Given some list of linearly independent integer vectors Open image in new window , the corresponding lattice L is the set of integer linear combinations of the \(f_i\); and the shortest vector problem is to find some nonzero element in L which has the minimum norm.
Example 1
Consider \(f_1 = (1,1\,894\,885\,908,0)\), \(f_2 = (0,1,1\,894\,885\,908)\), and \(f_3 = (0,0,2\,147\,483\,648)\). The lattice of \(f_1,f_2,f_3\) has a shortest vector \((3,17,4)\). It is the linear combination \((3,17,4) = 3f_1 + 5\,684\,657\,741f_2 + 5\,015\,999\,938f_3\).
Whereas finding a shortest vector is NPhard [9], the LLL algorithm is a polynomial time algorithm for approximating a shortest vector: The algorithm is parametric by some Open image in new window and computes a short vector, i.e., a vector whose norm is at most \(\alpha ^{\frac{m1}{2}}\) times as large than the norm of any shortest vector.
In this paper, we provide the first mechanized soundness proof of the LLL algorithm: the functional correctness is formulated as a theorem in the proof assistant Isabelle/HOL [11]. Regarding the complexity of the LLL algorithm, we did not include a formal statement which would have required an instrumentation of the algorithm by some instruction counter. However, from the termination proof of our Isabelle implementation of the LLL algorithm, one can easily infer a polynomial bound on the number of arithmetic operations.
In addition to the LLL algorithm, we also verify one application, namely a polynomialtime^{1} algorithm for the factorization of univariate integer polynomials, that is: factorization into the content and a product of irreducible integer polynomials. It reuses most parts of the formalization of the Berlekamp–Zassenhaus factorization algorithm, where the main difference is the replacement of the exponentialtime reconstruction phase [1, Sect. 8] by a polynomialtime one based on the LLL algorithm.
The whole formalization is based on definitions and proofs from a textbook on computer algebra [16, Chap. 16]. Thanks to the formalization work, we figured out that the factorization algorithm in the textbook has a serious flaw.
The paper is structured as follows. We present preliminaries in Sect. 2. In Sect. 3 we describe an extended formalization about the Gram–Schmidt orthogonalization procedure. This procedure is a crucial subroutine of the LLL algorithm whose correctness is verified in Sect. 4. Since our formalization of the LLL algorithm is also executable, in Sect. 5 we compare it against a commercial implementation. We present our verified polynomialtime algorithm to factor integer polynomials in Sect. 6, and describe the flaw in the textbook. Finally, we give a summary in Sect. 7.
Our formalization is available in the archive of formal proofs (AFP) [2, 3].
2 Preliminaries
We assume some basic knowledge of linear algebra, but recall some notions and notations. The inner product of two real vectors \(v=(c_0,\dots ,c_n)\) and \(w=(d_0,\dots ,d_n)\) is Open image in new window . Two real vectors are orthogonal if their inner product is zero. The Euclidean norm of a real vector v is Open image in new window . A linear combination of vectors \(v_0,\dots ,v_m\) is \(\sum _{i=0}^{m} c_i v_i\) with \(c_0,\dots ,c_m \in \mathbb {R}\), and we say it is an integer linear combination if \(c_0,\dots ,c_m \in \mathbb {Z}\). Vectors are linearly independent if none of them is a linear combination of the others. The span – the set of all linear combinations – of linearly independent vectors \(v_0,\dots ,v_{m1}\) forms a vector space of dimension m, and \(v_0,\dots ,v_{m1}\) are its basis. The lattice generated by linearly independent vectors Open image in new window is the set of integer linear combinations of Open image in new window .
The degree of a polynomial \(f(x) = \sum _{i=0}^n c_i x^i\) is Open image in new window , the leading coefficient is Open image in new window , the content is the GCD of coefficients \(\{c_0,\dots ,c_n\}\), and the norm Open image in new window is the norm of its corresponding coefficient vector, i.e., Open image in new window .
If \(f = f_0 \cdot \ldots \cdot f_m\), then each \(f_i\) is a factor of f, and is a proper factor if f is not a factor of \(f_i\). Units are the factors of 1, i.e., \(\pm 1\) in integer polynomials and nonzero constants in field polynomials. By a factorization of polynomial f we mean a decomposition \(f = c \cdot f_0 \cdot \ldots \cdot f_m\) into the content c and irreducible factors \(f_0,\dots ,f_m\); here irreducibility means that each \(f_i\) is not a unit and admits only units as proper factors.
Our formalization has been carried out using Isabelle/HOL, and we follow its syntax to state theorems, functions and definitions. Isabelle’s keywords are written in Open image in new window . Throughout Sects. 3 and 4, we present Isabelle sources in a way where we are inside some context which fixes a parameter Open image in new window , the dimension of the vector space.
3 Gram–Schmidt Orthogonalization
The Gram–Schmidt orthogonalization (GSO) procedure takes a list of linearly independent vectors Open image in new window from \(\mathbb {R}^n\) or \(\mathbb {Q}^n\) as input, and returns an orthogonal basis Open image in new window for the space that is spanned by the input vectors. In this case, we also write that Open image in new window is the GSO of Open image in new window .
We already formalized this procedure in Isabelle as a function Open image in new window when proving the existence of Jordan normal forms [15]. That formalization uses an explicit carrier set to enforce that all vectors are of the same dimension. For the current formalization task, the usage of a carrierbased vector and matrix library is important: Harrison’s encoding of dimensions via types [5] is not expressive enough for our application; for instance for a given square matrix of dimension n we need to multiply the determinants of all submatrices that only consider the first i rows and columns for all \(1 \le i \le n\).
Whereas there is no conceptual problem in expressing these definitions and proving the equalities in Isabelle/HOL, there still is some overhead because of the conversion of types. For instance in lemma Open image in new window , Open image in new window is a list of vectors; in (1), g is a recursively defined function from natural numbers to vectors; and in (3), the list of \(g_i\)’s is seen as a matrix.
A nice property of ocprojections is that they are unique up to v and the span of S. Back to GSO, since \(g_i\) is the ocprojection of \(f_i\) and \(\{f_0,\dots ,f_{i1}\}\), we conclude that \(g_i\) is uniquely determined in terms of \(f_i\) and the span of \(\{f_0,\dots ,f_{i1}\}\). Put differently, we obtain the same \(g_i\) even if we modify some of the first i input vectors of the GSO: only the span of these vectors must be preserved.
Whereas this shortvector lemma requires only a half of a page in the textbook, in the formalization we had to expand the condensed paperproof into 170 lines of more detailed Isabelle source, plus several auxiliary lemmas.
For instance, on papers one easily multiplies two sums ( Open image in new window ) and directly omits quadratically many neutral elements by referring to orthogonality, whereas we first had to prove this auxiliary fact in 34 lines.
The shortvector lemma is the key to obtaining a short vector in the lattice. It tells us that the minimum value of Open image in new window is a lower bound for the norms of the nonzero vectors in the lattice. If Open image in new window for some \(\alpha \in \mathbb {R}\), then the basis is weakly reduced w.r.t. \(\alpha \). If moreover \(\alpha \ge 1\), then \(f_0 = g_0\) is a short vector in the lattice generated by Open image in new window : Open image in new window for any nonzero vector h in the lattice.
The GSO of some basis Open image in new window will not generally be (weakly) reduced, but this problem can be solved with the LLL algorithm.
4 The LLL Basis Reduction Algorithm
A readable, but inefficient, implementation of the LLL algorithm is given by Algorithm 1, which mainly corresponds to the algorithm in the textbook [16, Chap. 16.2–16.3]: the textbook fixes \(\alpha = 2\) and \(m = n\). Here, it is important to know that whenever the algorithm mentions \(\mu _{i,j}\), it is referring to \(\mu \) as defined in (2) for the current values of Open image in new window and Open image in new window . In the algorithm, \(\lfloor x \rceil \) denotes the integer closest to x, i.e., \(\lfloor x \rceil := \lfloor x + \frac{1}{2}\rfloor \).
Let us have a short informal view on the properties of the LLL algorithm. The first required property is maintaining the lattice of the original input Open image in new window . This is obvious, since the basis is only changed in lines (4) and (6), and since swapping two basis elements, and adding a multiple of one basis element to a different basis element will not change the lattice. Still, the formalization of these facts required 190 lines of Isabelle code.
The second property is that the resulting GSO should be weakly reduced. This requires a little more argumentation, but is also not too hard: the algorithm maintains the invariant of the whileloop that the first i elements of the GSO are weakly reduced w.r.t. approximation factor \(\alpha \). This invariant is trivially maintained in line (7), since the condition in line (5) precisely checks whether the weakly reduced property holds between elements \(g_{i1}\) and \(g_i\). Moreover, being weakly reduced up to the first i vectors is not affected by changing the value of \(f_i\), since the first i elements of the GSO only depend on \(f_0,\dots ,f_{i1}\). So, the modification of \(f_i\) in line (4) can be ignored w.r.t. being weakly reduced.
Hence, formalizing the partial correctness of Algorithm 1 w.r.t. being weakly reduced is not too difficult. What makes it interesting, are the remaining properties we did not yet discuss. The most difficult part is the termination of the algorithm, and it is also nontrivial that the final basis is reduced. Both of these properties require equations which precisely determine how the GSO will change by the modification of Open image in new window in lines 4 and 6.
Concerning the complexity, each \(\mu _{i,j}\) can be computed with \(\mathcal{O}(n)\) arithmetic operations using its defining Eq. (2). Also, the updates of the GSO after swapping basis elements require \(\mathcal{O}(n)\) arithmetic operations. Since there are at most m iterations in the forloop, each iteration of the whileloop in Algorithm 2 requires \(\mathcal{O}(n \cdot m)\) arithmetic operations. As a consequence, if \(n = m\) and I is the number of iterations of the whileloop, then Algorithm 2 requires \(\mathcal{O}(n^3 + n^2 \cdot I)\) many arithmetic operations, where the cubic number stems from the initial computation of the GSO in line 1.
To verify that Algorithm 2 computes a weakly reduced basis, it “only” remains to verify its termination, and the invariant that the updates of \(g_i\)’s indeed correspond to the recomputation of the GSOs. These parts are closely related, because the termination argument depends on the explicit formula for the new value of \(g_{i1}\) as defined in line 6 as well as on the fact that the GSO remains unchanged in lines 3–4.
Since the termination of the algorithm is not at all obvious, and since it depends on valid inputs (e.g., it does not terminate if \(\alpha \le 0\)) we actually modeled the whileloop as a partial function in Isabelle [6]. Then in the soundness proof we only consider valid inputs and use induction via some measure which in turn gives us an upper bound on the number of loop iterations.
In the lemma, Open image in new window is a function which implements lines 3–4 of the algorithm. The lemma shows that the invariant is maintained and the GSO is unchanged, and moreover expresses the sole purpose of lines 3–4: they make the values \(\mu _{i,j}\) small.
Our correctness proofs for Open image in new window and Open image in new window closely follows the description in the textbook, and we mainly refer to the formalization and the textbook for more details: the presented lemmas are based on a sequence of technical lemmas that we do not expose at this point.
Here, we only sketch the termination proof: The value of the Gramian determinant for parameter \(k \ne i\) stays identical when swapping \(f_i\) and \(f_{i1}\), since it just corresponds to an exchange of two rows which will not modify the absolute value of the determinant. The Gramian determinant for parameter \(k = i\) will decrease by using the first statement of lemma Open image in new window , the explicit formula for the updated \(g_{i1}\) in line 6, the condition Open image in new window , and the fact that \(\mu _{i,i1} \le \frac{1}{2}\).
We did not formally prove the polynomialtime complexity in Isabelle. This task would at least require two further steps: since Isabelle/HOL functions are mathematical functions, we would need to instrument them by an instruction counter and hence make its usage more cumbersome; and we would need to formally prove that each arithmetic operation can be performed in polynomial time by giving bounds for the numerators and denominators in \(f_i\), \(g_i\), and \(\mu _{i,j}\).
Note that the reasoning on the number bounds should not be underestimated. To illustrate this, consider the following modification to the algorithm, which we described in the submitted version of this paper: Since the termination proof only requires that \(\mu _{i,i1}\) must be small, for obtaining a weakly reduced basis one may replace the forloop in lines 3–4 by a single update \(f_i := f_i  \lfloor \mu _{i,i1} \rceil \cdot f_{i1}\). Then the total number of arithmetic operations will reduce to \(\mathcal{O}(n^3 \cdot \log A)\). However, we figured out experimentally that this change is a bad idea, since then the bitlengths of the norms of \(f_i\) are no longer polynomially bounded: some input lattice of dimension 20 with 20 digit numbers led to the consumption of more than 64 GB of memory so that we had to abort the computation.
This indicates that formally verified bounds would be valuable. And indeed, the textbook contains informal proofs for bounds, provided that each \(\mu _{i,j}\) is small after executing lines 3–4. Here, a weakly reduced basis does not suffice.
With the help of Open image in new window it is now trivial to formally verify the correctness of the LLL algorithm, which is defined as Open image in new window in the sources.
5 Experimental Evaluation of the Verified LLL Algorithm
We formalized Open image in new window in a way that permits code generation [4]. Hence, we can evaluate the efficiency of our verified LLL algorithm. Here, we use a fixed approximation factor \(\alpha = 2\), we map Isabelle’s integer operations onto the unbounded integer operations of the target language Haskell, and we compile the code with ghc version 8.2.1 using the O2 parameter.
We consider the LatticeReduce procedure of Mathematica version 11.2 as an alternative way to compute short vectors. The documentation does not specify the value of \(\alpha \), but mentions that Storjohann’s variant [12] of the LLL basis reduction algorithm is implemented.
For the input, we use lattices of dimension n where each of the n basis vectors has ndigit random numbers as coefficients. So, the size of the input basis is cubic in n; for instance the bases for \(n = 1\), 10, and 100 are stored in files measured in bytes, kilobytes, and megabytes, respectively.
Figure 1 displays the execution times in seconds for increasing n. The experiments have all been run on an iMacPro with a 3.2 GHz Intel Xeon W running macOS 10.13.4. The execution times of both algorithms can both be approximated by a polynomial \(c \cdot n^6\) – the gray lines behind the dots – where the ratio between the constant factors c is 306.5, which is also reflected in the diagrams by using different scales for the timeaxis.
Besides efficiency, it is worth mentioning that we did not find bugs in fplll’s or Mathematica’s implementation: the short vectors that are generated by both tools have always been as short as our verified ones, but not much shorter: the average ratio between the norms is 0.74 for fplll and 0.93 for Mathemathica.
Under http://clinformatik.uibk.ac.at/isafor/LLL_src.tgz one can access the sources and the input lattices for the experiments.
6 Factorization of Polynomials in Polynomial Time
In this section we first describe how the LLL algorithm helps to factor integer polynomials, by following the textbook [16, Chap. 16.4–16.5].
We only summarize how we tried to verify the corresponding factorization Algorithm 16.22 of the textbook. Indeed, we almost finished it: after 1 500 lines of code we had only one remaining goal to prove. However, we were unable to figure out how to discharge this goal and then also started to search for inputs where the algorithm delivers wrong results. After a while we realized that Algorithm 16.22 indeed has a serious flaw, with details provided in Sect. 6.2.
Therefore, we derive another algorithm based on ideas from the textbook, which also runs in polynomialtime, and prove its soundness in Isabelle/HOL.
6.1 Short Vectors for Polynomial Factorization
In order to factor an integer polynomial f, we may assume a modular factorization of f into several monic factors \(u_i\): Open image in new window modulo m where \(m = p^l\) is some prime power for userspecified l. In Isabelle, we just reuse our verified modular factorization algorithm [1] to obtain the modular factorization of f.
We briefly explain how to compute nontrivial integer polynomial factors h of f based on Lemma 1, as also informally described in the textbook.
Lemma 1
([16, Lemma 16.20]). Let f, g, u be nonconstant integer polynomials. Let u be monic. If u divides f modulo m, u divides g modulo m, and Open image in new window , then Open image in new window is nonconstant.
Let f be a polynomial of degree n. Let u be any degreed factor of f modulo m. Now assume that f is reducible, so \(f = f_1 \cdot f_2\) where w.l.o.g., we assume that u divides \(f_1\) modulo m and that Open image in new window . Let the lattice \(L_{u,k}\) be the set of all polynomials of degree below \(d+k\) which are divisible by u modulo m. As Open image in new window , clearly \(f_1 \in L_{u, n  d}\).
Hence, if l is chosen large enough so that Open image in new window then all preconditions of Lemma 1 are satisfied, and Open image in new window will be a nonconstant factor of f. Since \(f_1\) divides f, also Open image in new window will be a nonconstant factor of f. Moreover, the degree of h will be strictly less than n, and so h is a proper factor of f.
6.2 Bug in Modern Computer Algebra
In the previous section we have chosen the lattice \(L_{u,k}\) for \(k = nd\) to find a polynomial h that is a proper factor of f. This has the disadvantage that h is not necessarily irreducible. In contrast, Algorithm 16.22 tries to directly find irreducible factors by iteratively searching for factors w.r.t. the lattices \(L_{u,k}\) for increasing k from 1 up to \(n  d\).
We do not have the space to present Algorithm 16.22 in detail, but just state that the arguments in the textbook and the provided invariants all look reasonable. Luckily, Isabelle was not so convinced: We got stuck with the goal that the content of the polynomial g corresponding to the short vector is not divisible by the chosen prime p, and this is not necessarily true.
The first problem occurs if the content of g is divisible by p. Consider \(f_1 = x^{12}+x^{10}+x^8+x^5+x^4+1\) and \(f_2 = x\). When trying to factor \(f = f_1 \cdot f_2\), then \(p = 2\) is chosen, and at a certain point the short vector computation is invoked for a modular factor u of degree 9 where \(L_{u,4}\) contains \(f_1\). Since \(f_1\) itself is a shortest vector, \(g = p \cdot f_1\) is a short vector: the approximation quality permits any vector of \(L_{u,4}\) of norm at most Open image in new window . For this valid choice of g, the result of Algorithm 16.22 will be the nonfactorization \(f = f_1 \cdot 1\).
We informed the authors of the textbook about this first problem. They admitted the flaw and it is easy to fix.
There is however a second potential problem. If g is even divisible by \(p^l\), then Algorithm 16.22 will again return wrong results. In the formalization we therefore integrate the check Open image in new window into the factorization algorithm^{4}, and then this modified version of Algorithm 16.22 is correct.
We could not conclude the question whether the additional check is really required, i.e., whether Open image in new window can ever happen, and just give some indication that the question is nontrivial. For instance, when factoring \(f_1\) above, then \(p^l\) is a number with 124 digits, Open image in new window , so in particular all basis elements of \(L_{u,1}\) will have a norm of at least \(p^l\). Note that \(L_{u,1}\) also does not have quite short vectors: any vector in \(L_{u,1}\) will have norm of at least 111 digits. However, since the approximation factor in this example is only two digits long, the short vector computation must result in a vector whose norm has at most 113 digits, which is not enough for permitting \(p^l\) with its 124 digits as leading coefficient of g.
6.3 A Verified Factorization Algorithm
To verify the factorization algorithm of Sect. 6.1, we formalize the two key facts to relate lattices and factors of polynomials: Lemma 1 and the lattice \(L_{u,k}\).
To prove Lemma 1, we partially follow the textbook, although we do the final reasoning by means of some properties of resultants which were already proved in the previous development of algebraic numbers [14]. We also formalize Hadamard’s inequality, which states that for any square matrix A having rows \(v_i\), then Open image in new window . Essentially, the proof of Lemma 1 consists of showing that the resultant of f and g is 0, and then deduce Open image in new window . We omit the fulldetailed proof, the interested reader can see it in the sources.
To define the lattice \(L_{u,k}\) for a degree d polynomial u and integer k, we define the basis \(v_0,\dots ,v_{k+d1}\) of the lattice \(L_{u,k}\) such that each \(v_i\) is the \((k+d)\)dimensional vector corresponding to polynomial \(u(x) \cdot x^i\) if \(i<k\), and to monomial \(m\cdot x^{k+di}\) if \(k \le i < k+d\).
There are some important facts that we must prove about Open image in new window .

Open image in new window is a list of linearly independent vectors as required for applying the LLL algorithm to find a short vector in \(L_{u,k}\).
 \(L_{u,k}\) characterizes the polynomials which have u as a factor modulo m: That is, any polynomial that satisfies the right hand side can be transformed into a vector that can be expressed as integer linear combination of the vectors of Open image in new window . Similarly, any vector in the lattice \(L_{u,k}\) can be expressed as integer linear combination of Open image in new window and corresponds to a polynomial of degree \(<k+d\) which is divisible by u modulo m.
The first property is a consequence of obvious facts that the matrix S is upper triangular, and its diagonal entries are nonzero if both u and m are nonzero. Thus, the vectors in Open image in new window are linearly independent.
Open image in new window is a recursive function which receives two parameters: the polynomial Open image in new window that has to be factored and Open image in new window , which is the list of modular factors of the polynomial Open image in new window . Open image in new window computes a short vector (and transforms it into a polynomial) in the lattice generated by a basis for \(L_{u,k}\) and suitable k, that is, Open image in new window . Open image in new window is the list of elements of Open image in new window that divide Open image in new window modulo Open image in new window , and Open image in new window contains the rest of elements of Open image in new window Open image in new window returns the list of irreducible factors of Open image in new window Termination follows from the fact that the degree decreases, that is, in each step the degree of both Open image in new window and Open image in new window is strictly less than the degree of Open image in new window
 1.
 2.
 3.
Open image in new window is the unique modular factorization of Open image in new window modulo Open image in new window
 4.
Open image in new window and Open image in new window are coprime, and Open image in new window is squarefree in Open image in new window
 5.
Open image in new window is sufficently large: Open image in new window where Open image in new window
Concerning complexity, it is easy to see that if Open image in new window splits into i factors, then Open image in new window invokes the short vector computation for exactly \(i + (i1)\) times: \(i1\) invocations are used to split Open image in new window into the i irreducible factors, and for each of these factors one invocation is required to finally detect irreducibility.
We further combine this algorithm with a preprocessing algorithm of earlier work [1]. This preprocessing splits a polynomial f into \(c \cdot f_1^1 \cdot \ldots \cdot f_k^k\) where c is the content of f which is not further factored. Each \(f_i\) is squarefree and contentfree, and will then be passed to Open image in new window . The combined algorithm factors arbitrary univariate integer polynomials into its content and a list of irreducible polynomials.
When experimentally comparing our verified LLLbased factorization algorithm with the verified Berlekamp–Zassenhaus factorization algorithm [1] we see no surprises. On the random polynomials from the experiments in [1], Berlekamp–Zassenhaus’s algorithm performs much better: it can factor each polynomial within a minute, whereas the LLLbased algorithm already fails on the smallest example. It is an irreducible polynomial with 100 coefficients where the LLL algorithm was aborted after a day when trying to compute a reduced basis for a lattice of dimension 99 with coefficients having up to 7 763 digits.
7 Summary
We formalized the LLL algorithm for finding a basis with short, nearly orthogonal vectors of an integer lattice, as well as its most famous application to get a verified factorization algorithm for integer polynomials which runs in polynomial time. The work is based on our previous formalization of the Berlekamp–Zassenhaus factorization algorithm, where the exponential reconstruction phase is replaced by the polynomialtime latticereduction algorithm. The whole formalization consists of about 10 000 lines of code, including a 2 200line theory which contains generalizations and theorems that are not exclusive to our development. This theory can extend the Isabelle standard library and up to six different AFP entries. As far as we know, this is the first formalization of the LLL algorithm and its application to factor polynomials in any theorem prover. This formalization led us to find a major flaw in a textbook.
There are some possibilities to extend the current formalization, e.g., by verifying faster variants of the LLL algorithm or by integrating other applications like the more efficient factorization algorithm of van Hoeij [10, Chap. 8]: it uses simpler lattices to factor polynomials, but its verification is much more intricate.
Footnotes
 1.
Again, we only mechanized the correctness proof and not the proof of polynomial complexity.
 2.
The formalization also shows soundness for \(\alpha = \frac{4}{3}\), but then the polynomial runtime is not guaranteed.
 3.
\(\frac{4\alpha }{4 + \alpha } = 1\) for \(\alpha = \frac{4}{3}\) and in that case one has to drop the logarithm from the measure.
 4.
When discussing the second problem with the authors, they proposed an even more restrictive check.
 5.
The corresponding Isabelle/HOL implementation contains some sanity checks which are solely used to ensure termination. We present here a simplified version.
Notes
Acknowledgments
This research was supported by the Austrian Science Fund (FWF) project Y757. Jose Divasón is partially funded by the Spanish projects MTM201454151P and MTM201788804P. Akihisa Yamada is supported by ERATO HASUO Metamathematics for Systems Design Project (No. JPMJER1603), JST. Some of the research was conducted while Sebastiaan Joosten and Akihisa Yamada were working in the University of Innsbruck. We thank Jürgen Gerhard and Joachim von zur Gathen for discussions on the problems described in Sect. 6.2, and Bertram Felgenhauer for discussions on gaps in the paper proofs. The authors are listed in alphabetical order regardless of individual contributions or seniority.
References
 1.Divasón, J., Joosten, S.J.C., Thiemann, R., Yamada, A.: A formalization of the Berlekamp–Zassenhaus factorization algorithm. In: CPP 2017, pp. 17–29. ACM (2017)Google Scholar
 2.Divasón, J., Joosten, S., Thiemann, R., Yamada, A.: A verified factorization algorithm for integer polynomials with polynomial complexity. Archive of Formal Proofs, Formal proof development, February 2018. http://isaafp.org/entries/LLL_Factorization.html
 3.Divasón, J., Joosten, S., Thiemann, R., Yamada, A.: A verified LLL algorithm. Archive of Formal Proofs, Formal proof development, February 2018. http://isaafp.org/entries/LLL_Basis_Reduction.html
 4.Haftmann, F., Nipkow, T.: Code generation via higherorder rewrite systems. In: Blume, M., Kobayashi, N., Vidal, G. (eds.) FLOPS 2010. LNCS, vol. 6009, pp. 103–117. Springer, Heidelberg (2010). https://doi.org/10.1007/9783642122514_9CrossRefGoogle Scholar
 5.Harrison, J.: The HOL light theory of Euclidean space. J. Autom. Reason. 50(2), 173–190 (2013)MathSciNetCrossRefGoogle Scholar
 6.Krauss, A.: Recursive definitions of monadic functions. In: PAR 2010. EPTCS, vol. 43, pp. 1–13 (2010)Google Scholar
 7.Lee, H.: Vector spaces. Archive of Formal Proofs, Formal proof development, August 2014. http://isaafp.org/entries/VectorSpace.html
 8.Lenstra, A.K., Lenstra, H.W., Lovász, L.: Factoring polynomials with rational coefficients. Math. Ann. 261, 515–534 (1982)MathSciNetCrossRefGoogle Scholar
 9.Micciancio, D.: The shortest vector in a lattice is hard to approximate to within some constant. SIAM J. Comput. 30(6), 2008–2035 (2000)MathSciNetCrossRefGoogle Scholar
 10.Nguyen, P.Q., Vallée, B. (eds.): The LLL Algorithm  Survey and Applications. Information Security and Cryptography. Springer, Heidelberg (2010). https://doi.org/10.1007/9783642022951CrossRefzbMATHGoogle Scholar
 11.Nipkow, T., Wenzel, M., Paulson, L.C. (eds.): Isabelle/HOL. LNCS, vol. 2283. Springer, Heidelberg (2002). https://doi.org/10.1007/3540459499CrossRefzbMATHGoogle Scholar
 12.Storjohann, A.: Faster algorithms for integer lattice basis reduction. Technical report 249, Department of Computer Science, ETH Zurich (1996)Google Scholar
 13.The FPLLL development team. fplll, a lattice reduction library (2016). https://github.com/fplll/fplll
 14.Thiemann, R., Yamada, A.: Algebraic numbers in Isabelle/HOL. In: Blanchette, J.C., Merz, S. (eds.) ITP 2016. LNCS, vol. 9807, pp. 391–408. Springer, Cham (2016). https://doi.org/10.1007/9783319431444_24CrossRefGoogle Scholar
 15.Thiemann, R., Yamada, A.: Formalizing Jordan normal forms in Isabelle/HOL. In: CPP 2016, pp. 88–99. ACM (2016)Google Scholar
 16.von zur Gathen, J., Gerhard, J.: Modern Computer Algebra, 3rd edn. Cambridge University Press, New York (2013)CrossRefGoogle Scholar
Copyright information
Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.
The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.