Cryptanalysis of a public key cryptosystem based on Diophantine equations via weighted LLL reduction

Abstract

Researching post-quantum cryptography is now an important task in cryptography. Although various candidates of post-quantum cryptosystems (PQC) have been constructed, sizes of their public keys are large. Okumura constructed a candidate of PQC whose security is expected to be based on certain Diophantine equations (DEC). Okumura analysis suggests that DEC achieves the high security with small public key sizes. This paper proposes a polynomial time-attack on the one-way property of DEC. We reduce the security of DEC to finding special short lattice points of some low-rank lattices derived from public data. The usual LLL algorithm could not find the most important lattice point in our experiments because of certain properties of the lattice point. Our heuristic analysis leads us to using a variant of the LLL algorithm, called a weighted LLL algorithm by us. Our experiments suggest that DEC with 128 bit security becomes insecure by our attack.

1 Introduction

Researching post-quantum cryptography is now an important task in cryptography. In fact, National Institute of Standards and Technology published a draft of the report on post-quantum cryptography NISTIR 8105 [23] (see also their announcement at PQCrypto 2016 [24]). Although various cryptosystems expected to be post-quantum cryptosystems (PQC) have been already constructed, see [7, 11] for details, sizes of their public keys are large. Thus finding computationally-hard problems which allow us to construct PQC with public keys of small sizes is a very important task in cryptography.

A Diophantine problem is well-known to be a computationally-hard problem in mathematics [12], and there are some cryptographic schemes based on the problem [6, 17, 20, 31], which are expected to have resistance to quantum algorithms. (Note that Diophantine problem here means a problem to find integral or rational zeros of a given multivariate polynomial with integer coefficients and high degree.) However, a polynomial time-attack on the one-way property of the cryptosystem [20] is proposed [10], and Proposition 2 in [17] suggests that the protocols [6, 17, 31] are impractical.

We can also consider the Diophantine problems over other rings. The Algebraic Surface Cryptosystem (ASC) [4] is based on the difficulty of the section finding problem, which can be viewed as the Diophantine problem over global function fields. Such the Diophantine problem is shown to be unsolvable in general [26, 29]. The security analysis suggests that ASC with public keys of sizes of about 500 bits achieves high security, see [4]. However, the ideal decomposition attack [15] breaks the one-way property of ASC.

Okumura [25] constructed a candidate of PQC of which the security is expected to be based on the difficulty of solving a special class of Diophantine equations, called Diophantine equations of degree increasing type, over $\mathbb {Z}$ (we will recall a definition of a polynomial of degree increasing type in Sect. 3). We call this cryptosystem DEC for short. Okumura shows that the solvability of Diophantine equations of degree increasing type is undecidable in general, see Remark 3.2 of [25]. DEC is a number field analogue of ASC and use the twisted plaintext, obtained from a plaintext by using RSA-like modular arithmetic, and some random polynomials with large coefficients in the encryption process. These are the main ideas of DEC to resist the analogues of all attacks [15, 18, 28, 30] on ASC and cryptosystems [1,2,3], which are proposed previously as ASC. In Sect. 4 of [25], Okumura points out that the above ideas increase the number of possible parameters in DEC, and that breaking the one-way property of DEC will become infeasible. Okumura also points out that one can decode a plaintext correctly from the twisted plaintext by using polynomials of degree increasing type as public keys. We will review DEC and its recommended parameters briefly in Sect. 3.

Another important property of DEC in post-quantum cryptography is that we may use public keys with small sizes, e.g., about 1, 200 bits with 128 bit security (see Remark 9). The size (1, 200 bits) is about 10 times smaller than sizes of public keys used in cryptosystems [21, 22, 27], which are well-known to be efficient among the candidates of PQC, with 128 bit security. Thus we consider that the security analysis of DEC is an important task in cryptography.

1.1 Our contribution

In this paper, we propose a polynomial time-attack on DEC. We show a linearization technique to transform the one-way property of DEC to finding appropriate solutions of linear systems obtained from public data. The use of three polynomials as a ciphertext enables us to use the linearization technique which constructs linear systems. This is the first weakness of DEC. Our attack consists of three steps. In each step, we have a linear system and need to find its appropriate solution, i.e., we need to find an appropriate lattice point in the lattice which is the solution space of the linear system. We use a solution obtained in the first (resp. second) step to construct a linear system in the second (resp. third) step. After finding appropriate solutions of the linear systems in all the steps, it is possible to recover a plaintext with sufficiently high probability by applying the Babai nearest plane algorithm [5] and some modular arithmetic.

Our various experiments on our attack in Sect. 6 suggest that finding a correct solution results in breaking DEC with sufficiently high probability. More precisely, after we find a correct solution in the first step, we can solve the linear systems in the second and third steps (note that in the third step, we may use an incorrect solution obtained in the second step). Thus the success of the first step is most important for our attack.

The rank of the lattice occurring in the first step is low, e.g., 3-rank in almost all cases, and a target lattice point in the first step is relatively short in the lattice. The quality of basis reduction algorithms such as the LLL algorithm [19] depends heavily on the rank of a lattice, and the LLL algorithm outputs a shortest lattice point in many cases for 3-rank lattices, see [19]. Thus it seems that one can succeed in the first step by using the LLL algorithm (or other basis reduction algorithms). However, as we will see in Sect. 4.3, the usual LLL algorithm does not seem to work well for finding the target lattice point in the first step, where by the “usual LLL algorithm”, we mean the LLL algorithm in terms of p-norms ($1 \le p \le \infty$) $\Vert {\varvec{ a}} \Vert _{p} := \left( | a_1 |^p + \ldots + |a_n|^p \right) ^{\frac{1}{p}}$. We heuristically analyse a reason why the usual LLL algorithm is not useful in our attack as follows: the target lattice point in the first step is not shortest, in terms of p-norms ($1 \le p \le \infty )$, with high probability, but some of its entries are comparatively small. In other words, the target lattice point is a comparatively short (not neccessarily shortest) in terms of well-known norms and has entries of unbalanced sizes.

1.2 Weighted LLL

In order to find lattice points having such properties, we find a special norm which makes the target lattice point in the first step (nearly) shortest by a heuristic way and apply a special LLL algorithm in terms of the special norm. We call the special norm and the special LLL algorithm the weighted norm and the weighted LLL algorithm, respectively. By a weighted norm for a vector ${\varvec{a}}=(a_1,...,a_n)$, we mean the norm:

$$\begin{aligned} \Vert {\varvec{a}} \Vert = \sqrt{\left( a_1w_1 \right) ^2 + \cdots + \left( a_nw_n \right) ^2}, \end{aligned}$$

where $w_i$’s are positive real numbers, which we call the weight factors. Note that as we already mentioned above, using other well-known norms, e.g., the p-norms ($1 \le p \le \infty$), in the LLL algorithm does not seem to be effective in finding the target lattice point.

We also note that using the weighted LLL algorithm can be also considered as using a re-scaling of a lattice to find lattice points with entries of unbalanced sizes in an LLL reduced basis of the lattice. Such a method can be also found in Coppersmith’s method [9] (see also Chapter 19 of [16]) and in Faugére et al.’s method [14]. In our method, each entries of the weighted norm are 2-power integers to use the knowledge of the bit length of entries of our target lattice point as in Faugére et al.’s method [14] (the possibility of knowing the bit length of entries of our target lattice point is the second weakness of DEC).

1.3 Experimental verification of our attack

Our many experiments in Sect. 6 suggest that the weighted LLL algorithm can find target lattice points in the first step of our attack with high probability (the probability being about from 70 to 90%) for the recommended parameters in Sect. 3. These results suggest that the weighted LLL algorithm is effective in cryptanalysis of cryptosystems whose security are reduced to finding lattice points with special properties: they are not shortest, but the bit length of their entries are almost known and comparatively small among entries of lattice points in certain lattices. In addition, our experiments also suggest that our attack breaks the one-way property of DEC with probability being about from 20 to 40% (this probability is sufficient in practical cryptanalysis). Our detailed complexity analysis on our attack and our experiments show that our attack is performed in polynomial time, and thus we conclude that our attack via the weighted LLL algorithm is practical and makes DEC insecure.

This paper is organized as follows: In Sect. 2, we give a definition of a weighted norm and describe the weighted LLL algorithm. In Sect. 3, we give a brief review of DEC. In Sect. 4, we describe the outline and some assumptions of our attack, and we also give an algorithm of our attack and a toy example to illustrate our attack. In Sect. 5, we analyse the complexity on our attack. In Sect. 6, we give some experimental results on our attack.

1.4 Notation

Throughout this paper, we denote by $R[\underline{x}]:= R[x_1, \ldots , x_n]$ the polynomial ring with n variables over a ring R. For every $\underline{{\varvec{i}}} = \left( i_1, \ldots , i_n \right) \in {\left( {\mathbb {Z}}_{\ge 0} \right) }^{n}$ and $\underline{{\varvec{a}}} = \left( a_1, \ldots , a_n \right) \in R^n$, we denote the element $a_1^{i_1} \cdots a_n^{i_n} \in R$, the monomial $x_1^{i_1} \cdots x_n^{i_n} \in R[\underline{x}]$ and the value $\sum _{k=1}^{n} i_k$ by ${\underline{{\varvec{a}}}}^{\underline{{\varvec{i}}}}$, ${\underline{x}}^{\underline{{\varvec{i}}}}$ and $\sum \underline{{\varvec{i}}}$, respectively. We can write any element $f \left( \underline{x} \right) = f \left( x_1, \ldots , x_n \right) \in R[\underline{x}] \smallsetminus \{ 0 \}$ (sometimes we also write f simply) in a unique way as a sum of terms:

$$\begin{aligned} f \left( \underline{x} \right) = \sum _{\underline{{\varvec{i}}} \in \varLambda } c_{\underline{{\varvec{i}}}} \underline{x}^{\underline{{\varvec{i}}}}, \end{aligned}$$

where $\varLambda$ is the finite subset of $\left( \mathbb {Z}_{\ge 0} \right) ^n$ and $c_{\underline{{\varvec{i}}}} \in R \smallsetminus \{ 0 \}$ for $\underline{{\varvec{i}}} \in \varLambda$. We then write $c_{\underline{{\varvec{i}}}} \left( f \right) := c_{\underline{{\varvec{i}}}}$ for $\underline{{\varvec{i}}} \in \varLambda _f := \varLambda$. We call $\varLambda _f$ the support of f. The total degree of f is denoted by $w_f$. For every element $\underline{{\varvec{a}}} = \left( a_1, \ldots , a_n \right) \in R^n$ and invertible element $d \in R^{\times }$, we denote the element $\left( a_1 / d, \ldots , a_n / d \right) \in R^n$ by $\underline{{\varvec{a}}}/d$. Then we denote the value of $f \left( \underline{x} \right)$ at $\underline{{\varvec{a}}}/d$ by $f \left( a_1 /d , \ldots , a_n / d \right)$ or $f \left( \underline{{\varvec{a}}}/d \right)$. In addition, if $R =$ $\mathbb {Z}$ or $\mathbb {Q}$, then we use the following notation:

$$\begin{aligned} \varGamma _f:= & {} \{ \left( \underline{{\varvec{i}}}, b_{\underline{{\varvec{i}}}} \right) \in \varLambda _f \times \mathbb {Z}_{>0} \ ; \ 2^{{b_{\underline{{\varvec{i}}}}} - 1} \le | c_{\underline{{\varvec{i}}}} \left( f \right) | < 2^{b_{\underline{{\varvec{i}}}}} \}, \\ H \left( f \right):= & {} \max \{ | c_{\underline{{\varvec{i}}}} \left( f \right) | \ ; \ \underline{{\varvec{i}}} \in \varLambda _f \}. \end{aligned}$$

We call $H \left( f \right)$ the height of f. In addition, if for a polynomial $f \in \mathbb {Z}[\underline{x}]$, the support $\varLambda _f = \{ \underline{{\varvec{i}}}_1, \ldots , \underline{{\varvec{i}}}_q \}$ is ordered by the order coming from the lexicographical order on the monomials of f, then we denote by ${\varvec{f}} = \left( c_{\underline{{\varvec{i}}}_1}(f), \ldots , c_{\underline{{\varvec{i}}}_q}(f) \right)$ the sequence of the ordered coefficients of f.

An m-dimensional lattice is defined as a discrete additive subgroup of an m-dimensional vector space over $\mathbb {R}$. It is well-known that for any lattice $\mathcal{L}$, there exist $\mathbb {R}$-linearly independent vectors generating $\mathcal{L}$ as a $\mathbb {Z}$-module. The rank of $\mathcal{L}$ is its rank as a $\mathbb {Z}$-module. For any lattice in $\mathbb {R}^m$ and its basis $\{ {\varvec{b}}_1, \ldots , {\varvec{b}}_r \}$, let ${\varvec{U}}$ be an $r \times m$ matrix whose i-th row vector coincides with ${\varvec{b}}_i$ for each i. Then we call ${\varvec{U}}$ the basis matrix of the lattice. Let $\langle \cdot , \cdot \rangle : \mathbb {R}^n \times \mathbb {R}^n \rightarrow \mathbb {R}$ be the natural inner product for some $n \in \mathbb {Z}_{>0}$. For a vector ${\varvec{v}} \in \mathbb {R}^n$, we denote the Euclidean norm of ${\varvec{v}}$ by $\Vert {\varvec{v}} \Vert$. We define the rounding function $\lfloor \cdot \rceil : \mathbb {R} \rightarrow \mathbb {Z}$ as $\lfloor c \rceil := \lfloor c + \frac{1}{2} \rfloor$ for any $c \in \mathbb {R}$. Let ${\varvec{M}}$ be an $m \times n$ matrix over $\mathbb {Z}$ and $\varphi _{{\varvec{M}}}$ the homomorphism as additive groups between $\mathbb {Z}^{m} \rightarrow \mathbb {Z}^{n}$ defined by ${\varvec{v}} \mapsto {\varvec{v}} {\varvec{M}}$. Then the kernel of $\varphi _{{\varvec{M}}}$ is a lattice in $\mathbb {R}^m$, and we call it the kernel lattice of ${\varvec{M}}$.

2 The weighted LLL algorithm

In this section, we explain the weighted LLL algorithm, which is a key of our attack in Sect. 4, briefly. First, we define a weighted norm and a weighted lattice. They are useful for describing the weighted LLL algorithm.

Definition 1

Given a vector ${\varvec{w}} = \left( w_1, \ldots , w_m \right) \in {\left( \mathbb {R}_{>0} \right) }^m$, the weighted norm $\Vert \cdot \Vert _{\varvec{w}} : \mathbb {R}^m \rightarrow \mathbb {R}$ for ${\varvec{w}}$ is defined as follows:

$$\begin{aligned} \Vert {\varvec{a}} \Vert _{\varvec{w}} := \sqrt{ \left( a_1 w_1 \right) ^2 + \cdots + \left( a_m w_m \right) ^2 } \quad \text{, } \text{ where } {\varvec{a}} = \left( a_1, \ldots , a_m \right) \in \mathbb {R}^m. \end{aligned}$$

A weighted lattice for ${\varvec{w}}$ in $\mathbb {R}^m$ is defined as a lattice endowed with the weighted norm for ${\varvec{w}}$ (this means that we always mean the weighted norm on the weighted lattice when we consider a norm on the weighted lattice). Given a lattice $\mathcal{L} \subset \mathbb {R}^m$ and a vector ${\varvec{w}} \in (\mathbb {R}_{>0})^m$, we denote $\mathcal{L}$ by $\mathcal{L}^{\varvec{w}}$ whenever we endow $\mathcal{L}$ with the structure of a weighted lattice for ${\varvec{w}}$.

For a lattice $\mathcal{L} \subset \mathbb {R}^m$ and a vector ${\varvec{w}}= \left( w_1, \ldots , w_m \right) \in {\left( \mathbb {R}_{>0} \right) }^m$, set a diagonal matrix ${\varvec{W}}$ whose (i, i)-entry is $w_i$ for $1 \le i \le m$. We consider the isomorphism $f_{{\varvec{W}}} : \mathbb {R}^m \longrightarrow \mathbb {R}^m$ by ${\varvec{x}} \mapsto {\varvec{x}} {\varvec{W}}$. Then, it is easy to show the equivalence of finding shortest lattice points, related with each other, in two lattices $\mathcal{L}^{\varvec{w}}$ and $f_{{\varvec{W}}} \left( \mathcal{L} \right)$.

The weighted LLL algorithm for ${\varvec{w}}$ is an algorithm to compute an LLL reduced basis (with respect to $\Vert \cdot \Vert _{\varvec{w}}$) of $\mathcal{L}^{\varvec{w}}$ (we call such a basis a weighted LLL reduced basis for ${\varvec{w}}$ in this paper).

The most important lattice point in our attack is not necessarily shortest in a low-rank lattice, but only some of its entries are comparatively small. This property leads us to applying the weighted LLL algorithm to find such a lattice point by carefully controlling the entries of a weighted LLL reduced basis, see Sect. 4.3.

Remark 2

Controlling the entries of a basis output by the LLL algorithm is used in Coppersmith’s method [9] and Faugére et al.’s method [14], see also Chapter 19 of [16]. In their method, the scale of a lattice (or equivalently an inner product used in the LLL algorithm) is changed by heuristic ways. One can conduct such changes by changing a norm from the Euclidean norm to a weighted norm for some weight. In particular, our method for choosing a weighted norm is the same as the method in [14], see Step 1-2 of our algorithm in Sect. 4.2.

3 Brief review of DEC

In this section, we review DEC briefly, see Sect. 3 in [25] for details. As we mentioned in Sect. 1, DEC is constructed as a candidate of PQC and has the property, which is strongly desired in post-quantum cryptography, that sizes of public keys in DEC is small, e.g., about 1, 200 bits with 128 bit security, see Remark 9. Note that sizes of public keys in cryptosystems [21, 22, 27], which are well-known to be efficient among the candidates of PQC, are about 10 times larger than 1, 200 bits.

3.1 Definiton of polynomials of degree increasing type

Definition 3

Let $X \left( \underline{x} \right) \in \mathbb {Z}[\underline{x}]$ be a non-zero polynomial and define a map

$$\begin{aligned} \sigma : \mathbb {Z}^n \longrightarrow \mathbb {Z}_{\ge 0} \ ; \ \underline{{\varvec{i}}} \mapsto \sum \underline{{\varvec{i}}}, \end{aligned}$$

where we recall that $\sum \underline{{\varvec{i}}} = \sum _{1 \le k \le n}i_k$ for $\underline{{\varvec{i}}} = \left( i_1, \ldots , i_n \right)$. The polynomial X is of degree increasing type if $\sigma |_{\varLambda _X}$ is injective.

Remark 4

Let $X \left( \underline{x} \right)$ be a non-zero polynomial of $\mathbb {Z}[\underline{x}]$.

• (1) From Definition 3, it is easy to see that $X \left( \underline{x} \right)$ is of degree increasing type if and only if the total degrees of the monomials of $X \left( \underline{x} \right)$ are different each other.

• (2) Let X be a polynomial of degree increasing type. By the following order $\succ$, the support $\varLambda _X$ becomes a totally ordered set: for two elements $\left( i_1, \ldots , i_n \right)$ and $\left( j_1, \ldots , j_n \right)$ in $\varLambda _X$, we have $\left( i_1, \ldots , i_n \right) \succ \left( j_1, \ldots , j_n \right)$ if and only if $i_1 + \cdots + i_n > j_1 + \cdots + j_n$.

Throughout this paper, whenever a polynomial X is of degree increasing type, we endow $\varLambda _X$ with the total order given in Remark 4 (2).

Example 5

The polynomial $X \left( x, y, z \right) := 3 x^3 y^2 z - 4 x^2 y^2 - x y z +5 y z + y + 11 \in \mathbb {Z}[x, y, z]$ is of degree increasing type.

Now, we describe DEC according to [25]. Note that Okumura did not suggest the security parameter because his purpose was to design the encryption scheme with 128 bit security. However, we here set the security parameter $\lambda$ to analyse the complexity of our attack for each security level.

In accordance with [25], we regard the total degree of a public key polynomial as a parameter, which we denote by $w_X$. Note that the parameter $w_X$ is taken to be an integer independent of the security parameter $\lambda$. In Remark 7 below, we will describe the reason why DEC has the two independent parameters $\lambda$ and $w_X$.

3.2 Key generation process

• Secret Key: A vector $\underline{{\varvec{a}}} := \left( a_1, \ldots , a_n \right) \in \mathbb {Z}^n$.

• Public Key:

  • (1) A positive integer d with $\mathrm{gcd} \left( a_i, d \right) = 1$ for all $1 \le i \le n$.

  • (2) A positive integer e with $\mathrm{gcd} \left( e, \varphi \left( d \right) \right) = 1$, where $\varphi$ is the Euler function.

  • (3) A polynomial $X \left( \underline{x} \right) \in \mathbb {Z}[ \underline{x} ]$ of degree increasing type such that X is irreducible, $X \left( \underline{{\varvec{a}}}/d \right) = 0$ and $\# \varLambda _X \le w_X$, where $\varLambda _X$ and $w_X$ denote the support and the total degree of X, respectively.

• Construction of $X \left( \underline{x} \right)$:

  • (1) Choose $\varLambda \subset {\left( \mathbb {Z}_{\ge 0} \right) }^n$ such that $3 \le \# \{ \sum \underline{{\varvec{i}}} \ ; \ \underline{{\varvec{i}}} \in \varLambda \} = \# \varLambda < \infty$ and $\underline{{\varvec{0}}} \in \varLambda$, where $\underline{{\varvec{0}}}:= \left( 0, \ldots , 0 \right) \in {\left( \mathbb {Z}_{\ge 0} \right) }^n$.

  • (2) Let $\underline{{\varvec{k}}}$ denote the maximal element of $\varLambda$ (note that $\varLambda$ is a totally ordered set in terms of the order given in Remark 4 (2)). Choose a random non-zero integer $c_{\underline{{\varvec{i}}}}$ for each $\underline{{\varvec{i}}} \in \varLambda \smallsetminus \{ \underline{{\varvec{k}}}, \underline{{\varvec{0}}} \}$. For a choice of $c_{\underline{{\varvec{i}}}}$, see Remark 9 (2).

  • (3) Choose random integers $c_{\underline{{\varvec{k}}}}$ and $c_{\underline{{\varvec{0}}}}$ such that

    $$\begin{aligned} c_{\underline{{\varvec{k}}}} {\underline{{\varvec{a}}}}^{\underline{{\varvec{k}}}} + c_{\underline{{\varvec{0}}}} d^{w}= & {} - \sum _{{\varvec{k}} \in \varLambda \smallsetminus \{ \underline{{\varvec{k}}}, \underline{{\varvec{0}}} \}} c_{\underline{{\varvec{i}}}} {\underline{{\varvec{a}}}}^{\underline{{\varvec{i}}}} d^{w - \sum \underline{{\varvec{i}}}}, \end{aligned}$$

    (1)

    where $w:= \max \{ \sum \underline{{\varvec{i}}} \ ; \ \underline{{\varvec{i}}} \in \varLambda \}$.

  • (4) Set $\varLambda _X := \varLambda$ and $X \left( \underline{x} \right) := \sum _{\underline{{\varvec{i}}} \in \varLambda _X} c_{\underline{{\varvec{i}}}} \underline{x}^{\underline{\varvec{i}}}$.

See Sect. 3.5 for a choice of a public key X and the sizes of the integers e, d and $a_i$’s.

Remark 6

There exist integers $c_{\underline{{\varvec{k}}}}$ and $c_{\underline{{\varvec{0}}}}$ such that the equality (1) is satisfied because $a_i$ and d are mutually prime for each $i \in \{ 1, \ldots , n \}$ from the assumption.

Remark 7

DEC has two parameters $\lambda$ and $w_X$ for the following reason: The public key of DEC is a Diophantine equation X of degree increasing type, and the secret key is its solution. Since there is no algorithm for solving Diophantine equations of degree increasing type, we set the security parameter, denoted by $\lambda$, which determines the security level against the key recovery attack by the brute force search (note that $\lambda$ also determines the security level against some attacks on the one-way property of DEC, see [25]). On the other hand, $w_x$ is an important parameter which complicates public diophantine equations and makes solving them difficult (by any method other than the brute force search), see also Remark 9.

3.3 Encryption process

• Plaintext: A polynomial $m \in \mathbb {Z}[ x_1, \ldots , x_n ]$ such that

  • (a) $\varLambda _m = \varLambda _X$,

  • (b) $1< c_{i_1, \ldots , i_n} \left( m \right) < d$ for all $\left( i_1, \ldots , i_n \right) \in \varLambda _m$,

  • (c) $\gcd \left( c_{i_1, \ldots , i_n} \left( m \right) , d \right) = 1$ for all $\left( i_1, \ldots , i_n \right) \in \varLambda _m$.

• Encryption Process:

  • (1) Choose a positive integer $N \in \mathbb {Z}_{>0}$ uniformly so that we have $N d > 2^{\lambda } H \left( X \right)$. For a size of N, see Section 3.5 below.

  • (2) Construct $\widetilde{m} \left( \underline{x} \right) \in \mathbb {Z}[\underline{x}]$, called the twisted plaintext, by setting $\varLambda _{\widetilde{m}} := \varLambda _{m}$ and $c_{\underline{{\varvec{i}}}} \left( \widetilde{m} \right) := {c_{\underline{{\varvec{i}}}} \left( m \right) }^e \left( \mathrm{mod} \ N d \right)$, where $0< c_{\underline{{\varvec{i}}}} \left( \widetilde{m} \right) < N d$ for $\underline{{\varvec{i}}} \in \varLambda _{\widetilde{m}}$.

  • (3) Choose $f \left( \underline{x} \right) \in \mathbb {Z}[ \underline{x} ]$ uniformly at random such that

    • (a) $\varLambda _f = \varLambda _X$,

    • (b) $H \left( \widetilde{m} \right)< c_{\underline{{\varvec{k}}}} \left( f \right) < N d$ and $\mathrm{gcd} \left( c_{\underline{{\varvec{k}}}} \left( f \right) , d \right) = 1$, where $\underline{{\varvec{k}}}$ denotes the maximal element of $\varLambda _f$.

  • (4) Choose $s_{j} \left( \underline{x} \right) , r_{j} \left( \underline{x} \right) \in \mathbb {Z}[\underline{x}]$ uniformly at random so that we have $\varGamma _{s_j} = \varGamma _{X}$ and $\varGamma _{r_j} = \varGamma _f$ for $1 \le j \le 3$.

  • (5) Put $F_j \left( \underline{x} \right) := \widetilde{m} \left( \underline{x} \right) + s_j \left( \underline{x} \right) f \left( \underline{x} \right) + r_j \left( \underline{x} \right) X \left( \underline{x} \right)$ for $1 \le j \le 3$. Send $\left( F_1, F_2, F_3, N \right)$ as a ciphertext.

3.4 Decryption process

• Decryption Process:

  • (1) By substituting $\underline{{\varvec{a}}} / d$, a zero of $X \left( \underline{x} \right)$, into $F_j \left( \underline{x} \right)$, we obtain

    $$\begin{aligned} h_j := F_j \left( \underline{{\varvec{a}}}/d \right) = \widetilde{m} \left( \underline{{\varvec{a}}}/d \right) + s_j \left( \underline{{\varvec{a}}}/d \right) f \left( \underline{{\varvec{a}}}/d \right) \text{ for } 1 \le j \le 3. \end{aligned}$$

    Compute

    $$\begin{aligned} H_1 := \left( h_1 - h_2 \right) d^{2 w_X}= & {} \left( s_1 \left( \underline{{\varvec{a}}}/d \right) - s_2 \left( \underline{{\varvec{a}}}/d \right) \right) f \left( \underline{{\varvec{a}}}/d \right) d^{2 w_X}, \\ H_2 := \left( h_1 - h_3 \right) d^{2 w_X}= & {} \left( s_1 \left( \underline{{\varvec{a}}}/d \right) - s_3 \left( \underline{{\varvec{a}}}/d \right) \right) f \left( \underline{{\varvec{a}}}/d \right) d^{2 w_X}. \end{aligned}$$

  • (2) Compute $g := \mathrm{gcd} \left( H_1, H_2 \right)$. If $\mathrm{gcd} \left( g, d \right) > 1$, then let $d'$ be the smallest factor of g satisfying $\mathrm{gcd} \left( d, g/{d'} \right) = 1$ and replace g by $g/{d'}$.

  • (3) Compute $H := h_1 d^{2 w_X} \left( \mathrm{mod} \ g \right)$ and $\mu := H d^{- w_X} \left( \mathrm{mod} \ g \right)$.

  • (4) Obtain the plaintext polynomial $m \left( \underline{x} \right)$ from $\mu$ or $\mu - g$ by using an algorithm described in Sects.  3.4 and 3.5 of [25].

Remark 8

In the algorithm in Sects. 3.4 and 3.5 of [25], we need to compute $\varphi (d)$ efficiently. From this, we should choose a prime number as d.

3.5 Parameter size

In Sect. 5 of [25], sizes of public/secret keys and ciphertexts are estimated so that DEC can be expected to have 128 bit security under some assumptions. In the following, we give their sizes under the same assumptions as [25] to analyse the complexity of our attack.

• (1) The sizes of $\underline{{\varvec{a}}}$, d, e and N:

  $$\begin{aligned}&2^{\frac{\lambda }{2}} \le d< 2^{\frac{\lambda }{2}+1}, \ (\lambda + 1) + \left( \frac{\lambda }{2} + 1\right) w_X \le e< 2 \left( (\lambda + 1) + \left( \frac{\lambda }{2} + 1\right) w_X \right) , \\&\frac{2^{\left\lceil \frac{\lambda }{n-1} \right\rceil }}{\varphi \left( d \right) } d \le | a_i |< \frac{2^{\left\lceil \frac{\lambda }{n-1} \right\rceil +1}}{\varphi \left( d \right) } d \left( 1 \le i \le n \right) , \\&2^{\lambda + \left( \frac{\lambda }{2} + 1\right) \left( w_X - 1 \right) } \le N < 2^{\lambda + 1 + \left( \frac{\lambda }{2} + 1\right) \left( w_X - 1 \right) }. \end{aligned}$$

We assume that $| c_{\underline{{\varvec{i}}}} \left( X \right) | < 2^{b}$ for any $\underline{{\varvec{i}}} \in \varLambda _X \smallsetminus \{ \underline{{\varvec{k}}}, \underline{{\varvec{0}}} \}$, where $\underline{{\varvec{k}}}$ denotes the maximal element of $\varLambda _X$, see Sect. 5 of [25].

• (2) The size of a secret key is at most

  $$\begin{aligned} \left( \left\lceil \frac{\lambda }{n-1} \right\rceil + 1 \right) n + \lceil \mathrm{log}_2 \ d - \mathrm{log}_2 \ \varphi \left( d \right) \rceil \end{aligned}$$

  bits.

• (3) The size of a public key is at most

  $$\begin{aligned} \left( \left\lceil \frac{\lambda }{n-1} \right\rceil + \left( \frac{\lambda }{2} + 2 + b\right) + \lceil \mathrm{log}_2 \ d - \mathrm{log}_2 \ \varphi \left( d \right) \rceil \right) w_X + (\lambda + 1) + \lceil \mathrm{log}_2 \ e \rceil \end{aligned}$$

  bits.

• (4) The size of a ciphertext is at most

  $$\begin{aligned} \frac{3}{2} \left( w_X^2 + w_X \right) \left( \lambda + 1 + (\lambda + 2) w_X + \lceil \mathrm{log}_2 \ w_X \rceil \right) + \lambda + 1 + \left( \frac{\lambda }{2} + 1\right) \left( w_X - 1 \right) \end{aligned}$$

  bits. Note that the size of each coefficient of $F_i$ is at most

  $$\begin{aligned} \lambda + 1 + (\lambda + 2) w_X + \lceil \mathrm{log}_2 \ w_X \rceil \end{aligned}$$

  bits for $i = 1$, 2, and 3.

Remark 9

• (1) In Sect. 4.5 of [25], it is pointed out that we should use a polynomial X satisfying $w_X \ge 5$, $n \ge 3$ and some conditions as a public key in order to avoid finding rational solutions to $X = 0$. However, polynomials of degree increasing type are in a special class of polynomials, and finding rational zeros of such polynomials may be easier than finding those of general polynomials. Moreover, although finding rational zeros of polynomials of higher degree seems to be difficult in general, we should consider sizes of public keys and ciphertexts. Thus we recommend to use X of degree 10 as a public key.

• (2) In Sect. 5 of [25], it is pointed out that for a public key X and $\underline{{\varvec{i}}} \in \varLambda _X \smallsetminus \{ \underline{{\varvec{k}}}, \underline{{\varvec{0}}} \}$, we may choose $c_{\underline{{\varvec{i}}}}(X) \le 2^{10}$, where $\underline{{\varvec{k}}}$ denotes the maximal element of $\varLambda _X$. However, since solving Diophantine equations of degree increasing type may be easier than solving more general Diophantine equations as we mentioned above, we should also consider using larger $c_{\underline{{\varvec{i}}}}(X)$ for $\underline{{\varvec{i}}} \in \varLambda _X \smallsetminus \{ \underline{{\varvec{k}}}, \underline{{\varvec{0}}} \}$ to deal with a wide class of polynomials of degree increasing type. In our experiments of Sect. 6, we choose $c_{\underline{{\varvec{i}}}}(X)$ so that the sizes of $\left| c_{\underline{{\varvec{i}}}}(X) \right|$ are b bits for $b = 10, 50$ and 100.

• (3) When $\lambda = 128$, $w_X = \# \varLambda _X$ and $b = 10$, we generated 100 public keys X randomly and measured their sizes. As a result, their average size is about 1, 200 bits. This size (1, 200 bits) is about 10 times smaller than sizes of public keys in cryptosystems [21, 22, 27], which are well-known to be efficient among the candidates of PQC.

3.6 Toy example of DEC

In the following, we give a toy example of DEC in the case of $n = 2$.

• Secret Key: $\underline{{\varvec{a}}} = \left( a, b \right) = \left( 47, 49 \right) \in \mathbb {Z}^2$.

• Public Key: $(d, e, X) = (5, 17, 125 x^3 + 675 y - 110438)$.

  ($\varLambda _X = \{ \left( 3, 0 \right) , \left( 0,1 \right) , \left( 0, 0 \right) \}$, $\underline{{\varvec{k}}} = \left( 3, 0 \right)$, $H \left( X \right) =110438$.)

• Plaintext: $m \left( \underline{x} \right) = m \left( x, y \right) = 3 x^3 + 3 y + 2$.

• Objects for Encryption:

  • (1) $N = 353408$ ($N d =1767040$).

  • (2) $\widetilde{m} \left( \underline{x} \right) = \widetilde{m} \left( x, y \right) = 146243 x^3 + 146243 y + 131072$ ($H \left( \widetilde{m} \right) = 146243$).

  • (3) $f \left( \underline{x} \right) = f \left( x, y \right) = 949843 x^3 + 1324952 y + 1109775$.

    ($c_{\underline{{\varvec{k}}}} \left( f \right) = 949843$, $H \left( \widetilde{m} \right) = 146243< c_{\underline{{\varvec{k}}}} \left( f \right) = 949843 < 1767040 = N d$.)

  • (4) $s_j$ and $r_j$:

    $$\begin{aligned} s_1= & {} 115 x^3 + 924 y + 126337, \ \ s_2 = 82 x^3 + 962 y + 89939, \\ s_3= & {} 67 x^3 + 977 y + 121816, \ \ r_1 = 691019 x^3 + 1363650 y + 1329029, \\ r_2= & {} 852655 x^3 + 1584164 y + 2007688, \\ r_3= & {} 940020 x^3 + 2016302 y + 1144882. \end{aligned}$$

  • (5) Cipher Polynomials: $F_j := \widetilde{m} + s_j f + r_j X$.

    $$\begin{aligned} F_1= & {} 195609320 x^6 + 1666918487 x^3 y + 43979457762 x^3 + 2144719398 y^2 \\&+ 18714355042 y - 6569529455, \\ F_2= & {} 184469001 x^6 + 1795957655 x^3 y - 8395474520 x^3 + 2343914524 y^2 \\&- 53364106711 y - 121912862547, \\ F_3= & {} 181141981 x^6 + 1903319645 x^3 y + 12109757546 x^3 + 2655481954 y^2 \\&- 59418815676 y + 8750004156. \end{aligned}$$

4 Weighted LLL-based polynomial time-attack for DEC

We give in this section our attack algorithm against DEC, based on the weighted LLL. We use the following notation described in Notation of Sect. 1: for a polynomial $h = \sum _{\underline{{\varvec{i}}} \in \varLambda _h}c_{\underline{{\varvec{i}}}}(h)x^{\underline{{\varvec{i}}}} \in \mathbb {Z}[ \underline{x} ]$, let ${\varvec{h}} := \left( c_{\underline{{\varvec{i}}}_1}(h), \ldots , c_{\underline{{\varvec{i}}}_{\# \varLambda _h}}(h) \right)$, where $\varLambda _h = \{ \underline{{\varvec{i}}}_1,\ldots , \underline{{\varvec{i}}}_{\# \varLambda _h} \}$ is the support of h. Note that $\varLambda _h$ is an ordered set (see Notation in Sect. 1).

Let $(d, e, X(\underline{x}) ) \in \mathbb {Z}^2 \times ( \mathbb {Z}[ \underline{x} ] )$ and $( F_1(\underline{x}), F_2(\underline{x}), F_3(\underline{x}), N ) \in ( \mathbb {Z}[ \underline{x} ] )^3 \times \mathbb {Z}$ be a public key and a ciphertext, as described in Sects. 3.2 and 3.3. Let $m(\underline{x}) \in \mathbb {Z}[ \underline{x} ]$ be a plaintext. Each cipher polynomial is of the form $F_j (\underline{x}) = \widetilde{m}(\underline{x}) + s_j(\underline{x}) f(\underline{x}) + r_j(\underline{x}) X(\underline{x})$ for the twisted plaintext $\widetilde{m}(\underline{x})$ and some random polynomials $f (\underline{x})$, $s_j (\underline{x})$ and $r_j (\underline{x})$. For the choice of $f (\underline{x})$, $s_j (\underline{x})$ and $r_j (\underline{x})$, see Sect. 3.3 for details. We write $\varLambda _X = \{ \underline{{\varvec{i}}}_1, \ldots , \underline{{\varvec{i}}}_q \}$ with $\underline{{\varvec{i}}}_1 \succ \cdots \succ \underline{{\varvec{i}}}_q$, where the total order $\succ$ on $\varLambda _X$ is given in Remark 4 (2). Recall that the supports of m, $\widetilde{m}$, $s_j$, $r_j$ and f ($1 \le j \le 3$) are the same as $\varLambda _X$, which allows attackers to suppose $\varLambda _{F_1} = \varLambda _{F_2} = \varLambda _{F_3}$. Let $\underline{{\varvec{k}}}$ denote the maximal element of $\varLambda _X$. To simplify the notation, put $q := \# \varLambda _X$ throughout this section. For recovering m, it suffices to get the correct $\widetilde{m}$.

4.1 Idea of our attack

Before we give an algorithm of our attack, we describe the idea of our attack. Recall from Sect. 3 that in DEC, we use the cipher polynomials of the form

$$\begin{aligned} F_j := \widetilde{m} + s_j f + r_j X \quad \text{ for } j =1, 2 \text{ and } 3. \end{aligned}$$

We reduce recovering $\widetilde{m}$ to finding special solutions to certain linear systems derived from X and $(F_1, F_2, F_3, N)$, the public key and the ciphertext, by linearization techniques described below.

We have the following equalities for $j=1$ and 2 from the way to construct the cipher polynomials:

$$\begin{aligned} F_j - F_{j+1}= & {} ( s_j - s_{j+1} ) f + ( r_{j} - r_{j+1} ) X. \end{aligned}$$

Since the cipher polynomials $F_1 \left( \underline{x} \right)$, $F_2 \left( \underline{x} \right)$, $F_3 \left( \underline{x} \right)$ and the public key $X \left( \underline{x} \right)$ are known, we may obtain $f \left( \underline{x} \right)$ if we determine $s_1 \left( \underline{x} \right) - s_2 \left( \underline{x} \right)$ and $s_2 \left( \underline{x} \right) - s_3 \left( \underline{x} \right)$. We set

$$\begin{aligned} s^{\prime }_j:= & {} s_j - s_{j+1},\quad r^{\prime }_j := r_j - r_{j+1}, \\ F^{\prime }_j:= & {} F_j - F_{j+1} = s^{\prime }_j f + r^{\prime }_j X \quad \text{ for } j=1 \text{ and } 2, \\ g:= & {} s^{\prime }_2 r^{\prime }_1 - s^{\prime }_1 r^{\prime }_2. \end{aligned}$$

We then have the following equalities:

$$\begin{aligned} F^{\prime }_1 \left( \underline{x} \right)= & {} s^{\prime }_1 \left( \underline{x} \right) f \left( \underline{x} \right) + r^{\prime }_1 \left( \underline{x} \right) X \left( \underline{x} \right) , \end{aligned}$$

(2)

$$\begin{aligned} F^{\prime }_2 \left( \underline{x} \right)= & {} s^{\prime }_2 \left( \underline{x} \right) f \left( \underline{x} \right) + r^{\prime }_2 \left( \underline{x} \right) X \left( \underline{x} \right) , \end{aligned}$$

(3)

$$\begin{aligned} g \left( \underline{x} \right) X \left( \underline{x} \right)= & {} s^{\prime }_2 \left( \underline{x} \right) F^{\prime }_1 \left( \underline{x} \right) - s^{\prime }_1 \left( \underline{x} \right) F^{\prime }_2 \left( \underline{x} \right) . \end{aligned}$$

(4)

4.1.1 First step: determination of $s^{\prime }_j$ for $j = 1$ and 2

Here, we describe how to determine ${\varvec{s}}^{\prime }_j$ for $j = 1$ and 2. (As we mentioned in Sect. 1, the vectors ${\varvec{s}}^{\prime }_j$ ($j = 1$ and 2) are the most important target vectors). In the equality (4), we regard the coefficients of $s^{\prime }_j \left( \underline{x} \right)$ and $g \left( \underline{x} \right)$ as indeterminates. We then obtain the linear system ${\varvec{u}} {\varvec{A}} = {\varvec{0}}$, where ${\varvec{A}}$ is the $\left( \left( 2 q + \# \varLambda _{X^2} \right) \times \# \varLambda _{X^3} \right)$ coefficient matrix of the linear system. We denote by $\mathcal{L}^{\prime }_1$ the kernel lattice of ${\varvec{A}}$, where the kernel lattice of ${\varvec{A}}$ is defined as the nullspace of ${\varvec{A}}$, see Notation in Sect. 1. Let $\mathcal{L}_1$ be the lattice spanned by the vectors consisting of the 1-(2q)th entries of the elements in $\mathcal{L}^{\prime }_1$. Experimentally, the rank of $\mathcal{L}_1$ is equal to 3 in many cases, see Remark  27 in Sect. 6. Thus, we assume the following condition:

Assumption 10

The rank of $\mathcal{L}_1$ is equal to 3.

Moreover, as we will see in Sect. 4.3, the correct $\left( {\varvec{s}}^{\prime }_1, {\varvec{s}}^{\prime }_2 \right)$ has the property described in Sects. 1 and 2 so that the usual LLL reduction does not work well to find $\left( {\varvec{s}}^{\prime }_1, {\varvec{s}}^{\prime }_2 \right)$. Note that this is true in many cases because of the construction of X (cf. Sect. 3.2). Thus, we use the weighted LLL reduction for a weight ${\varvec{w}}$ described below. Put ${\varvec{w}}^{\prime } = \left( w^{\prime }_1, \ldots , w^{\prime }_q \right)$ as follows:

$$\begin{aligned} w^{\prime }_j := 2^{\left\lfloor \log _2 \left( \frac{H \left( X \right) }{c_{\underline{{\varvec{i}}}_j}} \right) \right\rfloor }, \end{aligned}$$

where ${\varvec{X}} := ( c_{\underline{{\varvec{i}}}_1} (X), \ldots , c_{\underline{{\varvec{i}}}_q} (X) )$ denotes the vector of the coefficients of $X \left( \underline{x} \right)$. We set ${\varvec{w}}:= \left( w^{\prime }_1, \ldots , w^{\prime }_q , w^{\prime }_1 , \ldots , w^{\prime }_q \right)$. Assume the following condition.

Assumption 11

The $\left( {\varvec{s}}^{\prime }_1, {\varvec{s}}^{\prime }_2 \right)$ is a shortest vector in $\mathcal{L}_1^{\varvec{w}}$.

Let $f_{{\varvec{W}}}$ be the isomorphism described in Sect. 2 from $\mathbb {R}^{2q}$ to $\mathbb {R}^{2q}$ as $\mathbb {R}$-vector spaces. From Assumption 10, the rank of $f_{{\varvec{W}}}(\mathcal{L}_1)$ is equal to 3. This means that we can expect the weighted LLL reduction for the weight ${\varvec{w}}$ to output a shortest vector in $\mathcal{L}_1^{\varvec{w}}$ with high probability. Thus it is expected to find the correct $\left( {\varvec{s}}^{\prime }_1, {\varvec{s}}^{\prime }_2 \right)$ via the weighted LLL reduction for the weight ${\varvec{w}}$, see Sect. 2 and Assumption  11.

Remark 12

As we will see in Sect. 4.3, one may fail in determining $( {\varvec{s}}^{\prime }_1, {\varvec{s}}^{\prime }_2 )$ even if one adopts the LLL reduction in terms of the p-norm ($1 \le p \le \infty$) as a lattice reduction for $\mathcal{L}_1$. Thus, the above assumptions and applying the weighted LLL reduction to $\mathcal{L}_1$ are crucial for our attack.

4.1.2 Second step: obtaining a candidate of f

Here, we describe how to determine a candidate of f. We substitute $s^{\prime }_1 \left( \underline{x} \right)$ and $s^{\prime }_2 \left( \underline{x} \right)$ obtained in Step 1 into (2) and (3). In a similar way to Step 1, by regarding the coefficients of $f \left( \underline{x} \right)$ and $r^{\prime }_j \left( \underline{x} \right)$ for $j=1$ and 2 as indeterminates, we have the linear system. We then fix $f^{\prime } \left( \underline{x} \right)$ such that (2) and (3) hold and that $f^{\prime } \left( \underline{x} \right)$ is close to the correct $f \left( \underline{x} \right)$, i.e., the absolute values of all coefficients of the polynomial $f^{\prime } \left( \underline{x} \right) - f \left( \underline{x} \right)$ are small. Note that $f^{\prime } \left( \underline{x} \right)$ does not necessarily coincide with the correct $f \left( \underline{x} \right)$ to recover $\widetilde{m}$ (cf. Remark 18 and Steps 3-3 and 3-4 in Sect. 4.2).

Remark 13

In Step 2, any solution $\left( f^{\prime }, r_1^{\prime \prime } \right)$ to the linear system can be written as $f^{\prime } = f + a X$ and $r_1^{\prime \prime } = r^{\prime }_1 - a s^{\prime }_1$, respectively ($a \in \mathbb {Z}$) if $\mathrm{gcd} \left( X, s^{\prime }_1 \right) = 1$ and if the solution in Step 1 is the correct $\left( {\varvec{s}}^{\prime }_1, {\varvec{s}}^{\prime }_2 \right)$. In fact, by putting $p := f^{\prime } - f$ and $q := r^{\prime \prime }_1 - r^{\prime }_1$, we have

$$\begin{aligned} F^{\prime }_1= & {} s^{\prime }_1 f^{\prime } + r^{\prime \prime }_1 X \\= & {} s^{\prime }_1 \left( f + p \right) + \left( r^{\prime }_1 + q \right) X \\= & {} \left( s^{\prime }_1 f + r^{\prime }_1 X \right) + \left( s^{\prime }_1 p + q X \right) \\= & {} F^{\prime }_1 + \left( s^{\prime }_1 p + q X \right) . \end{aligned}$$

It follows that $s^{\prime }_1 p = - q X$. Thus if $\mathrm{gcd} \left( X, s^{\prime }_1 \right) = 1$, there exists an integer $a \in \mathbb {Z}$ such that $p = a X$ and $q = -a s^{\prime }_1$ since $\mathrm{deg}\ p \le \mathrm{deg}\ X$ and $\mathrm{deg}\ q \le \mathrm{deg}\ s^{\prime }_1$. This fact implies that the rank of the kernel lattice in Step 2 is equal to 1 if $\mathrm{gcd} \left( X, s^{\prime }_1 \right) = 1$. If the solution obtained in Step 1 is $\left( -s^{\prime }_1, -s^{\prime }_2 \right)$, then $\left( f^{\prime }, r^{\prime \prime }_1 \right)$ can be written as $f^{\prime } = - f + a X$ and $r^{\prime \prime }_1 = r^{\prime }_1 + a s^{\prime }_1$, respectively ($a \in \mathbb {Z}$) by the same argument. Note that since X is irreducible from the construction of X in Sect. 3.2, we have $\gcd (X, s^{\prime }_1) = 1$ with high probability.

4.1.3 Third step: recovery of $\widetilde{m}$

Here, we describe how to recover $\widetilde{m}$. It is sufficient for recovering $\widetilde{m} \left( \underline{x} \right)$ to find ${\varvec{s}}_1$, see Remark 18 and Steps 3-3 and 3-4 in Sect. 4.2. From the form of the ciphertext (see Sect.  3.3), consider the following equality:

$$\begin{aligned} F_1= & {} \widetilde{m} + s_1 f^{\prime } + r_1 X, \end{aligned}$$

(5)

where $f^{\prime } \left( \underline{x} \right)$ is the polynomial obtained in Step 2 and other polynomials $\widetilde{m} \left( \underline{x} \right)$, $s_1 \left( \underline{x} \right)$ and $r_1 \left( \underline{x} \right)$ are unknown. Note that if we have the correct solution in Step 1 and $\mathrm{gcd} \left( X, s^{\prime }_1 \right) = 1$, then there exists a unique polynomial $r \left( \underline{x} \right)$ such that the correct $\widetilde{m} \left( \underline{x} \right)$, $s_1 \left( \underline{x} \right)$ and $f^{\prime } \left( \underline{x} \right)$ (not necessarily $f \left( \underline{x} \right) )$ satisfy the equality $F_1 = \widetilde{m} + s_1 f' + r X$, see Remark 18. In a similar way to Steps 1 and 2, by regarding the coefficients of $\widetilde{m} \left( \underline{x} \right)$, ${s_1} \left( \underline{x} \right)$ and ${r_1} \left( \underline{x} \right)$ as indeterminates, we have the linear system ${\varvec{w}} {\varvec{C}} = {\varvec{c}}$, where ${\varvec{C}}$ is the $\left( 3 q \times \# \varLambda _{X^2} \right)$ coefficient matrix of the linear system and ${\varvec{c}} \in {\mathbb {Z}}^{\# \varLambda _{X^2}}$. We denote by $\mathcal{L}_3$ the kernel lattice of ${\varvec{C}}$. The rank of $\mathcal{L}_3$ is equal to 3 with high probability, see Remark  27. From this, we assume the following:

Assumption 14

The rank of $\mathcal{L}_3$ is equal to 3.

Let ${\varvec{w}}_0$ be one solution to ${\varvec{w}} {\varvec{C}} = {\varvec{c}}$ and $\{ {\varvec{w}}_1, {\varvec{w}}_2, {\varvec{w}}_3 \}$ a basis of $\mathcal{L}_3$. Note that every integral solution to the system is represented as ${\varvec{w}}_0 + a_1 {\varvec{w}}_1 + a_2 {\varvec{w}}_2 + a_3 {\varvec{w}}_3$ ($a_i \in \mathbb {Z}$, $i=1,2$ and 3). The 1-$\# \varLambda _X$-th entries of ${\varvec{w}}_0$, ${\varvec{w}}_1$, ${\varvec{w}}_2$ and ${\varvec{w}}_3$ correspond to the coefficients of $\widetilde{m}$. As we will see in Remark  17, the system ${\varvec{w}} {\varvec{C}} = {\varvec{0}}$ has a solution ${\varvec{w}}^{\prime }$ whose 1-$\# \varLambda _X$-th entries equal zero. We choose such a solution as ${\varvec{w}}_3$. Assume the following condition:

Assumption 15

The entries in ${\varvec{s}}_1$ coincide with the $\left( \# \varLambda _X + 1 \right)$-$2 \# \varLambda _X$-th entries in ${\varvec{w}}_0 + {\varvec{w}}_3 - {\varvec{z}}$, where ${\varvec{z}}$ is a closest lattice point in $\mathcal{L}^{\prime }_3 := \langle {\varvec{w}}_1, {\varvec{w}}_2 \rangle _{\mathbb {Z}}$ to ${\varvec{w}}_0 + {\varvec{w}}_3$. In other words, ${\varvec{s}}_1$ is embedded in ${\varvec{w}}_0 + {\varvec{w}}_3 - {\varvec{z}}$ as its $\left( \# \varLambda _X + 1 \right)$-$2 \# \varLambda _X$-th entries.

The lattice $\mathcal{L}^{\prime }_3$ has rank 2, and thus we can expect to find ${\varvec{s}}_1$ in polynomial time by the Babai nearest plane algorithm [5] for solving CVP with sufficiently high probability under Assumption 15.

Remark 16

The reason why we assume Assumption 15 is the following: From the choice of ${\varvec{s}}_1$, the absolute values of the entries of ${\varvec{s}}_1$ are sufficiently smaller than those of $\widetilde{{\varvec{m}}}$ and ${\varvec{r}}_1$. Thus we can expect that the value of $\Vert {\varvec{w}}_0 + {\varvec{w}}_3 - \left( a_1 {\varvec{w}}_1 + a_2 {\varvec{w}}_2 \right) \Vert$ is sufficiently small if certain entries of the vector ${\varvec{w}}_0 + {\varvec{w}}_3 - \left( a_1 {\varvec{w}}_1 + a_2 {\varvec{w}}_2 \right)$ coincide with those of ${\varvec{s}}_1$.

Remark 17

In Step 3, the linear system ${\varvec{w}} {\varvec{C}} = {\varvec{0}}$ has a solution ${\varvec{w}}^{\prime }$ whose 1-$\# \varLambda _X$-th entries equal zero. Let $\left( {\varvec{m}}^{\prime } , {\varvec{s}}^{\prime }, {\varvec{r}}^{\prime } \right)$ be one solution to ${\varvec{w}} {\varvec{C}} = {\varvec{c}}$, i.e., $F_1 = m^{\prime } + s^{\prime } f^{\prime } + r^{\prime } X$. The vector $\left( {\varvec{m}}^{\prime } , {\varvec{s}}^{\prime }, {\varvec{r}}^{\prime } \right) + \left( {\varvec{0}} , {\varvec{X}}, - {\varvec{f}}^{\prime } \right)$ is also a solution to ${\varvec{w}} {\varvec{C}} = {\varvec{c}}$. In fact, we have

$$\begin{aligned} \left( m^{\prime } + 0 \right) + \left( s^{\prime } + X \right) f^{\prime } + \left( r^{\prime } - f^{\prime } \right) X = \left( m^{\prime } + s^{\prime } f^{\prime } + r^{\prime } X \right) + X f^{\prime } - f^{\prime } X = F_1. \end{aligned}$$

Thus $\left( {\varvec{0}} , {\varvec{X}}, - {\varvec{f}}^{\prime } \right)$ is an element of $\mathcal {L}_3$.

Remark 18

If we succeed in finding the correct $s_1$ in Step 3 and $\gcd (X, s^{\prime }_1) = 1$, there exists r satisfying the equality $F_1 - s_1 f^{\prime } = \widetilde{m} + r X$. In fact, $f^{\prime }$ obtained in Step 2 can be written as $f^{\prime }= f + a X$ or $f^{\prime } = - f + a X$ ($a \in \mathbb {Z}$) from Remark  13. We may assume that $f' = f + a X$. Then we have

$$\begin{aligned} F_1 - \widetilde{m}- s_1 f'= & {} s_1 f + r_1 X - s_1 f' \\= & {} s_1 \left( f' - a X \right) + r_1 X - s_1 f' \\= & {} \left( r_1 - a s_1 \right) X. \end{aligned}$$

Thus we have $F_1 - s_1 f^{\prime } = \widetilde{m} + r X$ by putting $r := r_1 - a s_1$.

4.2 Algorithm of our attack

Based on the idea in Sect. 4.1, we write down our attack algorithm against DEC in what follows. Let $(d, e, X(\underline{x}) ) \in \mathbb {Z}^2 \times \mathbb {Z}[ \underline{x} ]$ and $( F_1(\underline{x}), F_2(\underline{x}), F_3(\underline{x}), N )$ $\in ( \mathbb {Z}[ \underline{x} ] )^3 \times \mathbb {Z}$ be a public key and a ciphertext, as described in Sects  3.2 and 3.3. Let $m(\underline{x}) \in \mathbb {Z}[ \underline{x} ]$ be a plaintext. Each cipher polynomial is of the form $F_j (\underline{x}) = \widetilde{m}(\underline{x}) + s_j(\underline{x}) f(\underline{x}) + r_j(\underline{x}) X(\underline{x})$ for the twisted plaintext $\widetilde{m}(\underline{x})$ and some random polynomials $f (\underline{x})$, $s_j (\underline{x})$ and $r_j (\underline{x})$. We also recall that $\varLambda _X$ and $w_X$ denote the support of X and the total degree of X, respectively, see Notation in Sect. 1. Let $\underline{{\varvec{k}}}$ be the maximal element of $\varLambda _X$, see Remark  4 (2) for the ordering.

Attack Algorithm

• Input: $\left( d, e, X (\underline{x} ) \right)$ and $\left( F_1 (\underline{x}), F_2 (\underline{x}), F_3 (\underline{x}), N \right)$, a public key and a ciphertext.

• Output: $\widetilde{m} \left( \underline{x} \right)$, a twisted plaintext.

• Step 1: Determination of $s_j^{\prime }:= s_j - s_{j+1}$ for $j = 1$ and 2

  • Step 1-1: Put $F^{\prime }_j := F_j - F_{j+1}$, $r^{\prime }_j := r_j - r_{j+ 1}$ ($1 \le j \le 2$) and put $g := s^{\prime }_2r^{\prime }_1 - s^{\prime }_1r^{\prime }_2$. Compute a basis of the kernel lattice of ${\varvec{A}}$, i.e., solve ${\varvec{u}} {\varvec{A}} = {\varvec{0}}$. This system is derived from unknown coefficients in

    $$\begin{aligned} s^{\prime }_2 F^{\prime }_1 - s^{\prime }_1F^{\prime }_2= & {} g X, \end{aligned}$$

    (6)

    where ${\varvec{A}}$ is the $\left( 2 \# \varLambda _{X} + \# \varLambda _{X^2} \right) \times \# \varLambda _{X^3}$ coefficient matrix of the linear system obtained from the Eq. (6). Let $\{ {\varvec{u}}^{\prime }_1, {\varvec{u}}^{\prime }_2, {\varvec{u}}^{\prime }_3 \}$ be the set of basis vectors for the kernel lattice.

  • Step 1-2: We denote by ${\varvec{u}}_i$ the vector embedded in ${\varvec{u}}^{\prime }_i$ as its 1-$\left( 2 \# \varLambda _X \right)$-th entries for $i = 1$, 2 and 3. Execute the weighted LLL reduction for the weight described in Sect.  4.1 to the lattice $\mathcal{L}_1 := \langle {\varvec{u}}_1, {\varvec{u}}_2, {\varvec{u}}_3 \rangle$, and then get $({\varvec{s}}^{\prime }_1, {\varvec{s}}^{\prime }_2)$.

• Step 2: Obtaining a candidate of f

  • Step 2-1: Compute a solution to ${\varvec{v}} {\varvec{B}} = {\varvec{b}}$. This system is derived from unknown coefficients in

    $$\begin{aligned} F^{\prime }_1 = s^{\prime }_1 f + r^{\prime }_1 X, \quad F^{\prime }_2 = s^{\prime }_2 f + r^{\prime }_2X, \end{aligned}$$

    (7)

    where ${\varvec{B}}$ is the $\left( 3 \# \varLambda _{X} \times \# \varLambda _{X^2} \right)$ coefficient matrix obtained from the Eq.  (7). Let ${\varvec{v}}_0$ be a solution, and let $\{ {\varvec{v}}_1 \}$ be a basis of the kernel lattice $\mathcal{L}_2$ of ${\varvec{B}}$. If $\gcd (X, s^{\prime }_1) = 1$ in $\mathbb {Z}[\underline{x}]$, then the lattice $\mathcal{L}_2$ always has rank 1, see Remark  13.

  • Step 2-2: Compute ${\varvec{v}}^{\prime }_0 := {\varvec{v}}_0 - \lfloor \langle {\varvec{v}}_0, {\varvec{v}}_1 \rangle / \langle {\varvec{v}}_1, {\varvec{v}}_1 \rangle \rceil {\varvec{v}}_1$, another solution to ${\varvec{v}} {\varvec{B}} = {\varvec{b}}$. Let ${\varvec{v}}_0^{\prime \prime }$ be the vector embedded in ${\varvec{v}}^{\prime }_0$ as its 1-$(\# \varLambda _{X})$-th entries. Let $f^{\prime } \left( \underline{x} \right) \in \mathbb {Z}[\underline{x}]$ be a polynomial with ${\varvec{f}^{\prime }} = {\varvec{v}}_0^{\prime \prime }$. Experimentally ${\varvec{v}}_0^{\prime }$ gives in many cases a polynomial closer to f than ${\varvec{v}}_0$, see Step 2 in Sect.  4.3.

• Step 3: Recovery of $\widetilde{m}$

  • Step 3-1: Compute a solution to ${\varvec{w}} {\varvec{C}} = {\varvec{c}}$ and a basis of the kernel lattice of ${\varvec{C}}$. This system is derived from unknown coefficients in

    $$\begin{aligned} F_1= & {} \widetilde{m} + s_1 f^{\prime } + r_1 X, \end{aligned}$$

    (8)

    where ${\varvec{C}}$ is the $( 3 \# \varLambda _{X} \times \# \varLambda _{X^2})$ coefficient matrix oftained from the Eq. (8) and $f^{\prime }$ is the polynomial obtained in Step 2-2. Let ${\varvec{w}}_0$ be a solution and $\{ {\varvec{w}}_1, {\varvec{w}}_2, {\varvec{w}}_3 \}$ a basis of the kernel lattice, denoted by $\mathcal{L}_3$.

  • Step 3-2: Apply the Babai nearest plane algorithm to compute a closest lattice point ${\varvec{z}}$ in $\mathcal{L}^{\prime }_3:= \langle {\varvec{w}}_1, {\varvec{w}}_2 \rangle _{\mathbb {Z}}$ to ${\varvec{w}}_0 + {\varvec{w}}_3$. Let ${\varvec{s}}_1$ be the vector embedded in ${\varvec{w}}_0 + {\varvec{w}}_3 - {\varvec{z}}$ as its $\left( \# \varLambda _X + 1 \right)$-$2 \# \varLambda _X$-th entries.

  • Step 3-3: Compute a solution to ${\varvec{x}} {\varvec{H}} = {\varvec{h}}$, the linear system derived from unknown coefficients in

    $$\begin{aligned} F_1 - \widetilde{m} - s_1 f^{\prime }= & {} r X, \end{aligned}$$

    (9)

    where ${\varvec{H}}$ is the $\left( 2 \# \varLambda _X \times \# \varLambda _{X^2} \right)$ coefficient matrix obtained from the Eq.  (9) and the coefficients of $\widetilde{m}$ and r are indeterminates. Let ${\varvec{x}}$ be a solution to ${\varvec{x}} {\varvec{H}} = {\varvec{h}}$. Let ${\varvec{r}}^{\prime }$ be the vector consisting of the entries corresponding to r of ${\varvec{x}}$. Then we obtain a polynomial $r^{\prime }$ whose coefficients coincide with those of r except the constant part, i.e., $r = r^{\prime } + t$ for some $t \in \mathbb {Z}$.

  • Step 3-4: Compute

    $$\begin{aligned} e^{\prime }:= & {} e^{-1} \ (\mathrm{mod} \ \varphi \left( d \right) ), \\ H_1:= & {} F_1 - s_1 f^{\prime } - r^{\prime } X, \\ \mu:= & {} c_{\underline{{\varvec{k}}}} \left( H_1 \right) , \\ c_{\underline{{\varvec{k}}}} \left( m^{\prime } \right):= & {} \mu ^{e^{\prime }} \ \left( \mathrm{mod} \ d \right) \quad \left( 0< {c_{\underline{{\varvec{k}}}} \left( m^{\prime } \right) }< d \right) , \\ c_{\underline{{\varvec{k}}}} \left( \widetilde{m} \right):= & {} \left( c_{\underline{{\varvec{k}}}} \left( m^{\prime } \right) \right) ^{e} \ \left( \mathrm{mod} \ N d \right) \quad \left( 0< c_{\underline{{\varvec{k}}}} \left( \widetilde{m} \right) < N d \right) , \\ t:= & {} \left( \mu - c_{\underline{{\varvec{k}}}} \left( \widetilde{m} \right) \right) / c_{\underline{{\varvec{k}}}} \left( X \right) , \\ \widetilde{m}:= & {} F_1 - s_1 f^{\prime } - \left( r^{\prime } + t \right) X. \end{aligned}$$

    Output $\widetilde{m} \left( \underline{x} \right)$.

Remark 19

One may consider that applying the Babai nearest plane algorithm in terms of a weighted norm, or searching a desired vector ${\varvec{s}}_1$ by adding some elements in $\mathcal{L}_3^{\prime }$ are effective. However, the one-way property of DEC can be broken with sufficiently high probability without such operations. We will see the details in Sect. 6. Hence in our attack let us omit these procedures.

Remark 20

In Step 3-4 of the above algorithm, we use the fact that $c_{\underline{{\varvec{k}}}}(X)$ is divisible by d to compute an integer t, see (1) for the divisibility of $c_{\underline{{\varvec{k}}}}(X)$.

4.3 Cryptanalysis of toy example

We break the one-way property of the instance in Sect. 3.6 of DEC. We use the same notations as in Sect. 3.6. In this case, we have $\varLambda _g = \varLambda _{X^2} = \{ \left( 6, 0 \right) , \left( 3, 1 \right) , \left( 3, 0 \right) , \left( 0, 2 \right) , \left( 0, 1 \right) , \left( 0, 0 \right) \}$.

4.3.1 First step: determination of $s^{\prime }_j = s_j - s_{j+1}$

Here, we determine $s^{\prime }_j = s_j - s_{j+1}$ for $j = 1$ and 2. Compute

$$\begin{aligned} F^{\prime }_1 := F_1 - F_2= & {} 11140319 x^6 - 129039168 x^3 y + 52374932282 x^3 - 199195126 y^2 \\&+ 72078461753 y + 115343333092, \\ F^{\prime }_2 := F_2 - F_3= & {} 3327020 x^6 - 107361990 x^3 y - 20505232066 x^3 - 311567430 y^2 \\&+ 6054708965 y - 130662866703. \end{aligned}$$

We put

$$\begin{aligned} s^{\prime }_j:= & {} c_1^{\left( j \right) } x^3 + c_2^{\left( j \right) } y + c_3^{\left( j \right) } \quad \text{ for } j=1 \text{ and } 2, \\ g:= & {} c_1^{\left( g \right) } x^6 + c_2^{\left( g \right) } x^3 y + c_3^{\left( g \right) } x^3 + c_4^{\left( g \right) } y^2 + c_5^{\left( g \right) } y + c_6^{\left( g \right) }, \end{aligned}$$

where $c_{i}^{\left( j \right) }$’s and $c_i^{\left( g \right) }$’s are indeterminates. By comparing the coefficient of $\underline{x}^{\underline{{\varvec{i}}}}$ for each $\underline{{\varvec{i}}} \in \varLambda _{X^3}$ in the Eq. (4), we have the linear system ${\varvec{u}} {\varvec{A}}^{\prime } = {\varvec{0}}$, where ${\varvec{A}}^{\prime }$ is a $\left( 9 \times 9 \right)$ matrix.

The rank of the kernel lattice $\mathcal{L}^{\prime }_1$ of ${\varvec{A}}^{\prime }$ is equal to 3. Compute a basis $\{ {\varvec{u}}^{\prime }_1, {\varvec{u}}^{\prime }_2, {\varvec{u}}^{\prime }_3 \}$ of $\mathcal{L}^{\prime }_1$. Let ${\varvec{u}}_j$ be the vector of the 1-6th entries consisting of ${\varvec{u}}^{\prime }_j$ for $j=1$, 2 and 3. We then have

$$\begin{aligned} \left( \begin{array}{c} {\varvec{u}}_1\\ {\varvec{u}}_2\\ {\varvec{u}}_3 \end{array}\right)= & {} \left( \begin{array}{cccccc} 1 &{} 11464 &{} -3475226 &{} 80 &{} 5520 &{} 916415\\ 0 &{} 27025 &{} -8194204 &{} 0 &{} 12000 &{} 2328055\\ 0 &{} 0 &{} 0 &{} 125 &{} 675 &{} -110438 \end{array} \right) . \end{aligned}$$

By applying the LLL reduction to the lattice $\mathcal{L}_1$ spanned by ${\varvec{u}}_1, {\varvec{u}}_2, {\varvec{u}}_3$, we have an LLL reduced basis

$$\begin{aligned} \left( \begin{array}{c} {{\varvec{a}}_1}\\ {{\varvec{a}}_2}\\ {{\varvec{a}}_3} \end{array} \right)= & {} \left( \begin{array}{cccccc} 1568 &{} 3927 &{} -8708 &{} -435 &{} -4365 &{} -6789\\ -1792 &{} -4488 &{} 9952 &{} 515 &{} 5085 &{} -8018\\ 3841 &{} 9499 &{} 15250 &{} -1095 &{} -10905 &{} -1034 \end{array} \right) . \end{aligned}$$

However, actually, the target vector $( {\varvec{s}}^{\prime }_1, {\varvec{s}}^{\prime }_2 )$ defined by the coefficients of $s^{\prime }_1$ and $s^{\prime }_2$ is

$$\begin{aligned} \left( {\varvec{s}}^{\prime }_1, {\varvec{s}}^{\prime }_2 \right)= & {} \left( 33, - 38, 36398, 15, -15, -31877 \right) . \end{aligned}$$

Thus ${\varvec{a}}_i$ does not coincide with both of $( {\varvec{s}}^{\prime }_1, {\varvec{s}}^{\prime }_2 )$ and $- ( {\varvec{s}}^{\prime }_1, {\varvec{s}}^{\prime }_2 )$ for any $1 \le i \le 3$. Note that 1-2nd and 4-5th entries of the correct $( {\varvec{s}}^{\prime }_1, {\varvec{s}}^{\prime }_2 )$ are much smaller than its other entries. This is true in many cases from the constructions of X, $s^{\prime }_1$ and $s^{\prime }_2$ described in Sect. 3 of [25] and Sect. 4.2 in this paper. On the other hand, the absolute values of all entries of ${\varvec{a}}_i$ have almost the same sizes for $1 \le i \le 3$. Moreover, it is easy to see $\left\| ( {\varvec{s}}^{\prime }_1, {\varvec{s}}^{\prime }_2 ) \right\| _{p} > \max \{ \left\| {\varvec{a}}_1 \right\| _{p}, \left\| {\varvec{a}}_2 \right\| _{p}, \left\| {\varvec{a}}_3 \right\| _{p} \}$ for any $1 \le p \le \infty$, where $\Vert \cdot \Vert _p$ denotes the p-norm. For example, we have $\Vert ( {\varvec{s}}^{\prime }_1, {\varvec{s}}^{\prime }_2 ) \Vert _2 \approx 48383.47 > \max \{ \Vert {\varvec{a}}_1 \Vert _2, \Vert {\varvec{a}}_2 \Vert _2, \Vert {\varvec{a}}_3 \Vert _2 \} \approx 21418.08$. This means that our target vector $( {\varvec{s}}^{\prime }_1, {\varvec{s}}^{\prime }_2 )$ is not shortest in $\mathcal{L}_1$ of 3-rank in terms of $\Vert \cdot \Vert _p$ for any $1 \le p \le \infty$. Thus, it seems that the LLL lattice basis reduction in terms of well-known norms , e.g., $\Vert \cdot \Vert _p$ for $1 \le p \le \infty$, does not work well for finding $( {\varvec{s}}^{\prime }_1, {\varvec{s}}^{\prime }_2 )$.

To obtain $\left( {\varvec{s}}^{\prime }_1, {\varvec{s}}^{\prime }_2 \right)$, we apply the weighted LLL reduction for the weight ${\varvec{w}}$ described below to $\mathcal{L}_1$ since the above situation is good for the weighted LLL reduction, see Sect. 4 in [14]. Recall that ${\varvec{X}} = \left( 125, 675, -110438 \right)$. We have

$$\begin{aligned} \left( \frac{H \left( X \right) }{125}, \frac{H \left( X \right) }{675}, \frac{H \left( X \right) }{110438} \right)= & {} \left( \frac{110438}{125}, \frac{110438}{675}, 1 \right) . \end{aligned}$$

Put

$$\begin{aligned} {\varvec{w}}= & {} \left( 2^{\left\lfloor \log _2 \left( \frac{110438}{125} \right) \right\rfloor }, 2^{\left\lfloor \log _2 \left( \frac{110438}{675} \right) \right\rfloor }, 1, 2^{\left\lfloor \log _2 \left( \frac{110438}{125} \right) \right\rfloor }, 2^{\left\lfloor \log _2 \left( \frac{110438}{675} \right) \right\rfloor }, 1 \right) \\= & {} \left( 2^9, 2^7, 1, 2^9, 2^7, 1 \right) . \end{aligned}$$

We obtain the following weighted LLL reduced basis of $\mathcal{L}_1^{\varvec{w}}$:

$$\begin{aligned} \left( \begin{array}{c} {{\varvec{b}}_1}\\ {{\varvec{b}}_2}\\ {{\varvec{b}}_3} \end{array} \right)= & {} \left( \begin{array}{cccccc} 33 &{} -38 &{} 36398 &{} 15 &{} -15 &{} -31877\\ -33 &{} 38 &{} -36398 &{} 110 &{} 690 &{} -78561\\ -158 &{} -637 &{} 74040 &{} -15 &{} 15 &{} 31877 \end{array} \right) . \end{aligned}$$

Note that ${\varvec{b}}_1$ just coincides with $\left( {\varvec{s}}^{\prime }_1, {\varvec{s}}^{\prime }_2 \right)$.

4.3.2 Second step: obtaining a candidate of f

Here, we obtain a candidate of f. We set

$$\begin{aligned} f:= & {} c_1^{\left( f \right) } x^3 + c_2^{\left( f \right) } y + c_3^{\left( f \right) }, \\ r^{\prime }_j:= & {} c_1^{\left( j \right) } x^3 + c_2^{\left( j \right) } y + c_3^{\left( j \right) } \quad \left( j = 1 \text{ and } 2 \right) , \end{aligned}$$

where $c_{i}^{\left( f \right) }$’s and $c_{i}^{\left( j \right) }$’s are indeterminates. By substituting $s^{\prime }_1$ and $s^{\prime }_2$ obtained in Step 1 into the equalities (2) and (3), and by comparing the coefficient of $\underline{x}^{\underline{{\varvec{i}}}}$ for each $\underline{{\varvec{i}}} \in \varLambda _{X^2}$, we have the linear system ${\varvec{v}} {\varvec{B}} = {\varvec{b}}$, where B is a $\left( 9 \times 6 \right)$ matrix. The rank of the kernel lattice $\mathcal{L}_2$ of ${\varvec{B}}$ is equal to 1. We obtain a solution ${\varvec{v}}_0$ to ${\varvec{v}} {\varvec{B}} = {\varvec{b}}$ and a basis $\{ {\varvec{v}}_1 \}$ of $\mathcal{L}_2$ as follows:

$$\begin{aligned} \left( \begin{array}{c} {\varvec{v}}_0\\ {\varvec{v}}_1 \end{array}\right)= & {} \left( \begin{array}{ccccccccc} -32 &{} -3804373 &{} 840328137 &{} 89131 &{} -509276 &{} 275909743 &{} 26620 &{} -546123 &{} -241370517\\ 125 &{} 675 &{} -110438 &{} -33 &{} 38 &{} -36398 &{} -15 &{} 15 &{} 31877 \end{array}\right) . \end{aligned}$$

Compute another solution ${\varvec{v}}^{\prime }_0 := {\varvec{v}}_0 - \lfloor \langle {\varvec{v}}_0, {\varvec{v}}_1 \rangle / \langle {\varvec{v}}_1, {\varvec{v}}_1 \rangle \rceil {\varvec{v}}_1$ to ${\varvec{v}} {\varvec{B}} = {\varvec{b}}$. Let ${\varvec{v}}_0^{\prime \prime }$ be the vector consisting of the 1-3rd entries of ${\varvec{v}}^{\prime }_0$. We then have

$$\begin{aligned} {\varvec{v}}_0^{\prime \prime } = \left( 950468, 1328327, 557585 \right) , \end{aligned}$$

and set

$$\begin{aligned} f^{\prime }:= & {} 950468 x^3 + 1328327 y + 557585. \end{aligned}$$

Note that the polynomial $f^{\prime }$ obtained from ${\varvec{v}}_0^{\prime \prime }$ is closer to the correct f than the one obtained from ${\varvec{v}}_0$. We also note that it is possible to proceed to the next step even if $f^{\prime }$ does not coincide with f, see Remark 18.

4.3.3 Third Step: Recovery of $\widetilde{m}$

Finally, we recover $\widetilde{m} \left( x,y \right)$. We find $s_1 \left( x,y \right)$ before recovering $\widetilde{m} \left( x,y \right)$. Put

$$\begin{aligned} \widetilde{m}:= & {} c_1 x^3 + c_2 y + c_3, \\ s_1:= & {} c_4 x^3 + c_5 y + c_6, \\ r_1:= & {} c_7 x^3 + c_8 y + c_9, \end{aligned}$$

where $c_i$’s are indeterminates. By substituting $f^{\prime }$ obtained in Step 2 into the equalities (5), and by comparing the coefficient of $\underline{x}^{\underline{{\varvec{i}}}}$ for each $\underline{{\varvec{i}}} \in \varLambda _{X^2}$, we have the linear system ${\varvec{w}} {\varvec{C}} = {\varvec{c}}$, where ${\varvec{C}}$ is a $\left( 9 \times 6 \right)$ matrix. The rank of the kernel lattice $\mathcal{L}_3$ of ${\varvec{C}}$ is equal to 3. We fix a solution ${\varvec{w}}_0$ to the system and a basis $\{ {\varvec{w}}_1, {\varvec{w}}_2, {\varvec{w}}_3 \}$ of $\mathcal{L}_3$ as follows:

$$\begin{aligned} \left( \begin{array}{c} {\varvec{w}}_0\\ {\varvec{w}}_1\\ {\varvec{w}}_2\\ {\varvec{w}}_3 \end{array}\right)= & {} \left( \begin{array}{ccccccccc} 225204073068 &{} 315361848743 &{} -6569529455 &{} -10 &{} 249 &{} 0 &{} 1640912 &{} 2687357 &{} 0\\ 1 &{} 163580614 &{} -36132895073 &{} 0 &{} 0 &{} 43 &{} 0 &{} 0 &{} -326961\\ 0 &{} 475525025 &{} -105037483109 &{} 0 &{} 0 &{} 125 &{} 0 &{} 0 &{} -950468\\ 0 &{} 0 &{} 0 &{} 125 &{} 675 &{} -110438 &{} -950468 &{} -1328327 &{} -557585 \end{array} \right) . \end{aligned}$$

We find a vector ${\varvec{z}}$ in the lattice $\langle {\varvec{w}}_1, {\varvec{w}}_2 \rangle _{\mathbb {Z}}$ close to ${\varvec{w}}_0 + {\varvec{w}}_3$ by applying the Babai nearest plane algorithm. We then have the matrix

$$\begin{aligned} \left( \begin{array}{ccccccccc} 225203926700 &{} 315361701825 &{} -6569550089 &{} 0 &{} 0 &{} -236775 &{} 0 &{} 0 &{} -1254928\\ 146368 &{} 146918 &{} 20634 &{} 115 &{} 924 &{} 126337 &{} 690444 &{} 1359030 &{} 697343 \end{array} \right) , \end{aligned}$$

where 1st and 2nd rows are the vectors ${\varvec{z}}$ and ${\varvec{w}}_0 + {\varvec{w}}_3 - {\varvec{z}}$, respectively. The vector embedded in ${\varvec{w}}_0 + {\varvec{w}}_3 - {\varvec{z}}$ as its 4–6th entries is equal to the correct ${\varvec{s}}_1$.

Next, we compute r satisfying $F_1 - \widetilde{m} - s^{\prime }_1 = r X$. Note that there exists a polynomial r satisfying the above equality, and that we can recover $\widetilde{m}$ if we obtain such an r (cf. Remark 18 and Step 3-4 in Sect.  4.2). We set

$$\begin{aligned} r:= & {} {c_1} x^3 + {c_2} y + {c_3}, \\ \widetilde{m}:= & {} {c_4} x^3 + c_5 y + c_6, \end{aligned}$$

where $c_i$’s are indeterminates. In the equality $F_1 - s_1 f' = \widetilde{m} + r X$, by comparing the coefficient of $\underline{x}^{\underline{{\varvec{i}}}}$ for each $\underline{{\varvec{i}}} \in \varLambda _{X^2}$, we have the linear system ${\varvec{x}} {\varvec{H}} = {\varvec{h}}$, where ${\varvec{H}}$ is a $\left( 6 \times 6 \right)$ matrix. The rank of the kernel lattice $\mathcal{L}_4$ of ${\varvec{H}}$ is equal to 1. We fix a solution ${\varvec{x}}_0$ to ${\varvec{x}} {\varvec{H}} = {\varvec{h}}$ and a basis $\{ {\varvec{x}}_1 \}$ of $\mathcal{L}_4$ as follows:

$$\begin{aligned} \left( \begin{array}{c} {\varvec{x}}_0\\ {\varvec{x}}_1 \end{array} \right)= & {} \left( \begin{array}{cccccc} 2591380 &{} 4015684 &{} 0 &{} 226710493 &{} 1223593193 &{} -200170290060\\ 0 &{} 0 &{} 1 &{} -125 &{} -675 &{} 110438 \end{array}\right) . \end{aligned}$$

We set

$$\begin{aligned} r^{\prime }:= & {} 2591380 {x}^3 + 4015684 y + 1. \end{aligned}$$

There exists a unique $t \in \mathbb {Z}$ such that $r = r^{\prime } + t$. Our aim is to find such an integer t, see Steps 3-3 and 3-4 in Sect. 4.2. Let $\underline{{\varvec{k}}}$ be the maximal element in $\varLambda _X$. Put

$$\begin{aligned} e^{\prime }:= & {} e^{-1} \ (\mathrm{mod} \ \varphi \left( d \right) ) \\= & {} 1, \\ H_1:= & {} F_1- s_1 f^{\prime } - r^{\prime } X \\= & {} 226710368 x^3 + 1223592518 y - 200170179622, \\ \mu:= & {} {c_{\underline{{\varvec{k}}}} \left( H_1 \right) } \\= & {} - 200170179622, \\ c_{\underline{{\varvec{k}}}} \left( m^{\prime } \right):= & {} \mu ^{e^{\prime }} \ \left( \mathrm{mod} \ d \right) \quad \left( 0< c_{\underline{{\varvec{k}}}} \left( m^{\prime } \right)< d \right) \\= & {} 3, \\ c_{\underline{{\varvec{k}}}} \left( \widetilde{m} \right)= & {} \left( c_{\underline{{\varvec{k}}}} \left( m^{\prime } \right) \right) ^{e} \ \left( \mathrm{mod} \ N d \right) \quad \left( 0< {c_{\underline{{\varvec{k}}}} \left( \widetilde{m} \right) } < N d \right) \\= & {} 146243, \\ t= & {} \left( \mu - c_{\underline{{\varvec{k}}}} \left( \widetilde{m} \right) \right) / c_{\underline{{\varvec{k}}}} \left( X \right) \\= & {} 1812513, \\ \widetilde{m}= & {} F_1 - s_1 f^{\prime } - \left( r^{\prime } + t \right) X \\= & {} 146243 x^3 + 146243 y + 131072. \end{aligned}$$

We succeeded in recovering $\widetilde{m} \left( x, y \right)$ in Sect. 3.6.

5 Complexity analysis

In this section, we investigate the complexity of the algorithm in Sect. 4.2. We analyse our attack in accordance with the parameter sizes in Sect. 3.5 (cf. Sect.  5 in [25]). Let $X \in \mathbb {Z}[\underline{x}]$ be a public key of DEC. Let $w_X$ and $\varLambda _X$ denote the total degree and the support of X, respectively. To simplify the notations, we set $w := w_X$, assume $w = \# \varLambda _X$ and fix b, where b is the maximum of the bit length of the coefficients of X except its leading and constant terms. We show that the attack performs in polynomial time in terms of the parameters w and $\lambda$. Here note that w and $\lambda$ are independent of each other, see Remark 7 in Sect. 3. In our complexity analysis, we use the same notation as in Sect.  4.2. The parameters d and e are $O \left( 2^\lambda \right)$ and $O \left( w \lambda \right)$, respectively. Note that the size of each coefficient of $F_j$ is ${O \left( w \lambda \right) }$ bits for $j = 1$, 2 and 3, see Sect. 3.5 for the representation of the parameters by w and $\lambda$. Assume that the size of each coefficient of $s_1 f^{\prime }$ is bounded by $O ( w \lambda )$ bits.

Remark 21

First, let us determine the bit complexity of the computation of polynomials with integer coefficients in the algorithm. We suppose the arithmetic operations of addition and subtraction of two polynomials $F, G \in \mathbb {Z}[\underline{x}]$ are $O \left( \mathrm{min} \{ q_F, q_G \} \right)$ in $\mathbb {Z}$, where $q_F$ and $q_G$ are the number of the terms of F and G, respectively. Moreover, the arithmetic operations of multiplication of them are $O \left( \left( \max \{ q_F, q_G \} \right) ^2 \right)$ in $\mathbb {Z}$. We compute ${F_1}':=F_1 - F_2$ and ${F_2}':= F_2 - F_3$ at the beginning of the algorithm. Note that the number of the terms of $F_j$ is at most $w^2$ for each $1 \le j \le 3$. The sizes of the coefficients of $F_j$ are $O \left( w \lambda \right)$ for $j = 1$, 2 and 3. Thus the arithmetic complexity of computing $F^{\prime }_1$ and $F^{\prime }_2$ is $O \left( w^2 \right)$, and its bit complexity is

$$\begin{aligned} O \left( w^2 \left( w \lambda \right) \right)= & {} O \left( w^3 \lambda \right) . \end{aligned}$$

(10)

We do such computations in (6)–(9). The arithmetic complexity of (6)–(9) is $O (w^4)$ and thus the bit complexity is

$$\begin{aligned} O \left( w^4 \left( w \lambda \right) ^2 \right)= & {} O \left( w^6 \lambda ^2 \right) \end{aligned}$$

(11)

since the sizes of the coefficients of the polynomials appearing in (6)–(9) are $O \left( w \lambda \right)$ bits. Note that we regard the coefficients of certain polynomials as indeterminates. (For example, in (6), we regard the coefficients of ${s_1}'$, ${s_2}'$ and g as indeterminates.) On the other hand, we compute $H_1 := F_1 - s_1 f' - r X$ in Step 3-4. In this case, we do not regard any coefficient as indeterminates. Since for each of $s_1$, $f'$, r and X, the number of its terms is w, we require $O \left( w^2 \right)$ arithmetic operations for computing $s_1 f'$ and rX. In addition, for each of $F_1$, $s_1 f'$ and rX, the number of its terms is $O (w^2)$. Here recall that the size of each coefficient of the polynomials $F_1$, $s_1 f'$ and rX is $O \left( w \lambda \right)$ bits. Thus the bit complexity of computing $H_1$ is

$$\begin{aligned} O \left( w^2 \left( w \lambda \right) ^2 \right)= & {} O \left( w^4 \lambda ^2 \right) . \end{aligned}$$

(12)

Remark 22

Second, we solve one or two linear systems in each step of our attack. Then, we obtain one solution and the kernel lattice for each linear system. We assume that the bit complexity of solving a non-homogeneous linear system is equivalent to the bit complexity of computing the (row) Hermite Normal Form (HNF) of the augmented matrix of the system. According to Chapter 2 in [16], we assume that the computation of the HNF of an $n \times m$ matrix ${\varvec{M}} = (M_{i,j})_{i,j}$ requires $O(nm^4(\log (\Vert {\varvec{M}} \Vert _{\infty }))^2)$ bit operations, where $\Vert {\varvec{M}} \Vert _{\infty } := \max _{i,j} \{ |M_{i,j}| \}$. On the other hand, we assume that a homogeneous linear system is solved by the Gaussian elimination.

To simplify the notations, we assume the sizes of the entries of one solution and an output basis of the kernel lattice of each linear system are $O \left( \ell \right)$ bits if the sizes of the entries of its augmented matrix are $O \left( \ell \right)$ bits.

Remark 23

Third, we discuss the size of the norm of a vector with integer entries. Let ${\varvec{a}} = \left( a_1, \ldots , a_k \right) \in \mathbb {Z}^k$ be a vector with $| a_i | \le 2^l$ for $1 \le i \le k$. Since $\Vert {\varvec{a}} \Vert \le \sqrt{ k 2^{2l} }$, the size of $\Vert {\varvec{a}} \Vert$ is bounded by $\mathrm{log} \left( \sqrt{ k 2^{2l} } \right) = \mathrm{log} \left( k^{1/2} \right) + l = O \left( \mathrm{log} \left( k \right) + l \right)$ bits. Similarly, the size of $\Vert {\varvec{a}} \Vert ^2$ is $O \left( \mathrm{log} \left( k \right) + l \right)$ bits.

5.1 The complexity of first step

Step 1-1 We estimate the bit complexity for solving the linear system ${\varvec{u}} {\varvec{A}} = {\varvec{0}}$ with at most $2 w + w^2$ indeterminates and $w^3$ equations. Since this linear system is homogeneous, the arithmetic complexity in $\mathbb {Z}$ of solving the linear system is $O \left( w^6 \right)$, see Remark 22. The size of each entry of ${\varvec{A}}$ is $O \left( w \lambda \right)$ bits, and thus Step 1-1 requires

$$\begin{aligned} O \left( w^{8} \lambda ^2 \right) \end{aligned}$$

(13)

bit operations. In addition, we note that the sizes of the entries of ${\varvec{u}}^{\prime }_1, {\varvec{u}}^{\prime }_2$ and ${\varvec{u}}^{\prime }_3$, that are basis vectors of the kernel lattice $\mathcal{L}^{\prime }_1$ of ${\varvec{A}}$, are $O \left( w \lambda \right)$ bits from Remark 22.

Step 1-2 In the beginning of this step, we compute ${\varvec{U}} {\varvec{W}}$, where ${\varvec{U}}$ is a basis matrix of $\mathcal{L}_1$ with $3 \times 2 w$ entries and ${\varvec{W}}$ is a $\left( 2 w \times 2 w \right)$ diagonal matrix. The arithmetic complexity of multiplying these matrices is $3 \times \left( 2 w \right) = O \left( w \right)$. Since the size of each entry of ${\varvec{U}}$ and ${\varvec{W}}$ is $O \left( w \lambda \right)$ bits, the multiplying runs in

$$\begin{aligned} O \left( w \times \left( w \lambda \right) ^2 \right)= & {} O \left( w^3 \lambda ^2 \right) \end{aligned}$$

(14)

bit operations. We note that the size of each entry of ${\varvec{U}} {\varvec{W}}$ is $O \left( w \lambda \right)$ bits. After the multiplying, we execute the LLL reduction to the 2w-dimensional lattice $f_{\varvec{W}} \left( \mathcal{L}_1 \right)$ of 3-rank with the basis matrix ${\varvec{U}} {\varvec{W}}$. According to [19], the computation of the LLL reduction requires

$$\begin{aligned} O \left( 3^5 \left( 2 w \right) \left( \mathrm{log} \left( 2 w \times 2^{2 w \lambda } \right) \right) ^3 \right) \end{aligned}$$

bit operations in this case because the norms of the row vectors of ${\varvec{U}} {\varvec{W}}$ are $O \left( \sqrt{2 w \times 2^{2 w \lambda }} \right)$. Thus the LLL reduction of this step runs in

$$\begin{aligned} O \left( w^4 \lambda ^3 \right) \end{aligned}$$

(15)

bit operations. Th size of any entry of the vectors of the LLL reduced basis is $O \left( \sqrt{3 \left( w \times 2^{2 w \lambda } \right) } \right)$ because the rank of $f_{\varvec{W}} \left( \mathcal{L}_1 \right)$ is equal to 3, and because $\Vert {\varvec{u}}_i {\varvec{W}} \Vert ^2 = O \left( w \times 2^{2 w \lambda } \right)$ for $i = 1$, 2 and 3, and row vectors ${\varvec{u}}_i$ of ${\varvec{U}}$. Thus the size of any entry of the LLL-reduced basis matrix of ${\varvec{U}} {\varvec{W}}$ is $O \left( w \lambda \right)$ bits. We multiple the diagonal matrix ${\varvec{W}}^{-1}$ by the LLL reduced basis matrix. The arithmetic complexity of the multiplying is $3 \times 2 w = O \left( w \right)$. Thus the multiplying runs in

$$\begin{aligned} O \left( w \left( w \lambda \right) ^2 \right)= & {} O \left( w^3 \lambda ^2 \right) \end{aligned}$$

(16)

bit operations.

5.2 The complexity of second step

Step 2-1 In this step, we solve the linear system ${\varvec{v}} {\varvec{B}} = {\varvec{b}}$ with 3w indeterminates and at most $w^2$ equations. From Remark 22, the bit complexity of this step can be estimated as

$$\begin{aligned} O \left( w^{11} \lambda ^2 \right) . \end{aligned}$$

(17)

Every entry of a solution and basis vectors of the kernel lattice $\mathcal{L}_2$ has the size of $O \left( w \lambda \right)$ bits from the same reason as Step 1-1. Note that $\mathcal{L}_2$ is a 3w-dimensional lattice of 1-rank. Hence the sizes of the norms of ${\varvec{v}}_0$ and ${\varvec{v}}_1$ are $O \left( \sqrt{ \left( 3 w \times 2^{2 w \lambda } \right) } \right)$.

Step 2-2 In this step, we compute ${\varvec{v}}^{\prime }_0 := {\varvec{v}}_0 - \lfloor \langle {\varvec{v}}_0, {\varvec{v}}_1 \rangle / \langle {\varvec{v}}_1, {\varvec{v}}_1 \rangle \rceil {\varvec{v}}_1$. This computation requires $O \left( 2^4 \left( 3 w \right) \left( \mathrm{log} \left( 3 w \times 2^{2 w \lambda } \right) \right) ^2 \right)$ bit operations in accordance with Chapter 17 in [16]. Hence Step 2-2 requires

$$\begin{aligned} O \left( w^3 \lambda ^2 \right) \end{aligned}$$

(18)

bit operations.

5.3 The complexity of third step and the total complexity of our attack

Step 3-1 In this step, we compute a solution to the system ${\varvec{w}} {\varvec{C}} = {\varvec{c}}$, and a basis of the kernel lattice of ${\varvec{C}}$ with 3w indeterminates and at most $w^2$ equations. In a similar way to Step 2-1, the computation requires

$$\begin{aligned} O \left( w^{11} \lambda ^2 \right) \end{aligned}$$

(19)

bit operations. Every entry of a solution ${\varvec{w}}_0$ and basis vectors ${\varvec{w}}_1, {\varvec{w}}_2$ and ${\varvec{w}}_3$ of the kernel lattice $\mathcal{L}_3$ has the size of $O \left( w \lambda \right)$ bits. Note that the norms of ${\varvec{w}}_0$, ${\varvec{w}}_1$, ${\varvec{w}}_2$ and ${\varvec{w}}_3$ are $O \left( \sqrt{ \left( 3 w \times 2^{2 w \lambda } \right) } \right)$.

Step 3-2 In this step, we apply the Babai nearest plane algorithm to the 3w-dimensional lattice $\mathcal{L}^{\prime }_3 := \langle {\varvec{w}}_1, {\varvec{w}}_2 \rangle$ of 2-rank and the vector ${\varvec{w}}_0 + {\varvec{w}}_3$. Before executing the Babai nearest plane algorithm, we execute the LLL reduction to $\mathcal{L}_3^{\prime }$. Since $\mathcal{L}_3^{\prime }$ has 2-rank and 3w-dimension, the LLL reduction requires

$$\begin{aligned} O \left( 2^5 \left( 3 w \right) \left( \mathrm{log} \left( 3 w \times 2^{2 w \lambda } \right) \right) ^3 \right) \end{aligned}$$

bit operations. Thus the LLL reduction in Step 3-2 requires $O \left( w^4 \lambda ^3 \right)$ bit operations. The norm of any vector of the LLL reduced basis is

$$\begin{aligned} O \left( \sqrt{2 \left( 3 w \times 2^{2 w \lambda } \right) } \right) \end{aligned}$$

(cf. Chapter 17 in [16]). In a similar way to deriving the bit complexity of Gram-Schmidt algorithm (see Theorem 17.3.4 in [16]), one can verify that the Babai nearest plane algorithm requires $O \left( ( 3 w )^5 2^3 \left( \mathrm{log} \left( \sqrt{2 \left( 3 w \times 2^{2 w \lambda } \right) } \right) \right) ^2 \right)$ bit operations. From this, the bit complexity of the Babai nearest plane algorithm is $O \left( w^7 \lambda ^2 \right)$ in this case. Hence Step 3-2 runs in

$$\begin{aligned} O \left( w^4 \lambda ^3 \right) + O \left( w^7 \lambda ^2 \right) \end{aligned}$$

(20)

bit operations.

Step 3-3 We compute a solution to ${\varvec{x}} {\varvec{H}} = {\varvec{h}}$ with 2w indeterminates and at most $w^2$ equations. The size of any entry of ${\varvec{H}}$ and ${\varvec{h}}$ is $O \left( w \lambda \right)$ bits. Hence Step 3-3 runs in

$$\begin{aligned} O \left( w^{11} \lambda ^2 \right) \end{aligned}$$

(21)

bit operations.

Step 3-4 At the beginning of this step, we compute $e':=e^{-1} \ \mathrm{mod} \ \varphi \left( d \right)$ by using the extended Euclid’s algorithm. According to Remark 3.5 in [25], the integer d should be chosen so that one can compute $\varphi \left( d \right)$ efficiently because the computation is needed in the decryption process (see [25], Sect. 3.4). In Remark 3.5 of [25], the integer d is expected to be a prime number as such an example. From this, we assume d is a prime number, and then we have $\varphi \left( d \right) = d - 1$.

Next, we compute $c_{\underline{{\varvec{k}}}} \left( m^{\prime } \right) := \mu ^{e^{\prime }} \ \left( \mathrm{mod} \ d \right)$ $\left( 0< c_{\underline{{\varvec{k}}}} \left( m' \right) < d \right)$, where $e^{\prime } := e^{-1}$ (mod $\varphi (d)$) and ${\mu }$ is a certain coefficient of $H_1 \left( \underline{x} \right)$ (cf. Step 3-4 in Sect. 4.2). Recall that the bit sizes of $e^{\prime }$, $\mu$ and d are $O \left( \lambda \right)$, $O \left( w \lambda \right)$ and $O \left( \lambda \right)$, respectively. Thus this computation can be done in $O \left( w \lambda ^2 + \lambda ^3 \right)$ bit operations by the square-and-multiply algorithm for modular exponentiation.

Third, we compute $c_{\underline{{\varvec{k}}}} \left( \widetilde{m} \right) := \left( c_{\underline{{\varvec{k}}}} \left( m^{\prime } \right) \right) ^{e} \ \left( \mathrm{mod} \ N d \right)$ $\left( 0< c_{\underline{{\varvec{k}}}} \left( \widetilde{m} \right) < N d \right)$. Recall from Sect. 3.5 that the size of N is $O ( w \lambda )$ bits. Note that the sizes of $c_{\underline{{\varvec{k}}}} \left( m^{\prime } \right)$, e and Nd are $O \left( \lambda \right)$, $O \left( \mathrm{log} \left( w \lambda \right) \right)$ and $O \left( w \lambda \right)$ bits, respectively. Thus, the square-and-multiply algorithm requires $O \left( \left( w \lambda \right) ^2 \mathrm{log} \left( w \lambda \right) \right)$ bit operations to compute $c_{\underline{{\varvec{k}}}} \left( \widetilde{m} \right)$. As a consequence, those modular exponential arithmetic can be performed in $O \left( \lambda ^3 + w^2 \lambda ^2 \ \mathrm{log} \left( w \lambda \right) \right)$ bit operations. Finally, the computation of $t := \left( \mu - c_{\underline{{\varvec{k}}}} \left( \widetilde{m} \right) \right) / c_{\underline{{\varvec{k}}}} \left( X \right)$ runs in $O \left( w^2 \lambda ^2 \right)$ bit operations. The total bit complexity of Step 3-4 is

$$\begin{aligned} O \left( \lambda ^3 + w^2 \lambda ^2 \mathrm{log} \left( w \lambda \right) \right) . \end{aligned}$$

(22)

Putting all the steps together, namely considering (10)–(22), we can determine the complexity of our attack.

Theorem 24

The total bit complexity of the attack in Sect. 4.2 is

$$\begin{aligned} O \left( {w}^{11} {\lambda }^2 \right) + O \left( {w}^4 \lambda ^3 \right) . \end{aligned}$$

Consequently, our attack performs in polynomial time for all the parameters $\lambda$ and $w_X$, where $\lambda$ and $w_X$ are independent of each other.

Remark 25

The estimated complexity in Theorem 24 shows that the computation of our attack may become expensive for large $w=w_X$ and $\# \varLambda _X \le w$. Thus, to secure DEC, one can think of increasing the parameters w and $\# \varLambda _X$. However, DEC is impractical for large $w_X$ and $\# \varLambda _X$ since ciphertexts of DEC have exceedingly large sizes. For example, when $w_X = \# \varLambda _X = 45$, $b = 10$ and $\lambda = 128$, we generated 100 ciphertexts $(F_1, F_2, F_3, N)$ in accordance with Sects.  3.2 and 3.3, and measured their sizes. As a result, their average size is about 10, 086, 237 bits.

Remark 26

From the above reason, the dominant term of the estimated complexity in Theorem 24 is $O (w^4 \lambda ^3)$ in practice.

6 Experimental verification

In this section, we demonstrate with experimental results that our attack algorithm enables one to break the one-way property of DEC in practical time. In our experiments, we generated DEC instances of $n=4$, where n is the number of indeterminates of a public key $X(\underline{x})$. The PC used in our experiments is as follows: The OS is Mac OS X, 64 bit. The processor is 2.60GHz CPU (Intel Corei5). The memory is 16GB. Authors implemented the attack algorithm over Magma V2.21-3 [8]. For the parameters, we adopted recommended ones in Remark 9 (such parameters shall make DEC instances $\lambda =128$ bit level secure).

Procedures of Our Experiments For three parameters $w_X$, $\# \varLambda _X$ and b, each of which is independent of the security parameter $\lambda$, we conduct the following procedure 100 times:

• 1. Construct secret/public keys in accordance with Section 3.2.

• 2. With the public key, we generate a ciphertext in accordance with Sect. 3.3.

• 3. For the above public key and the ciphertext, recover the twisted plaintext by Attack Algorithm given in Sect. 4.2.

In our experiments, we generated each public key X so that its coefficients have b bit sizes except the terms of its maximal degree and constant., i.e., $2^{b-1} \le | c_{\underline{{\varvec{i}}}} \left( X \right) | < 2^{b}$ for all $\underline{{\varvec{i}}} \in \varLambda _X$ with $\underline{{\varvec{i}}} \ne \underline{{\varvec{k}}}, \underline{{\varvec{0}}}$. Here $\underline{{\varvec{k}}}$ denotes the maximal element of $\varLambda _X$, see Remark  4 (2) for the ordering. For each DEC instance generated as above, we also apply a variant of Attack Algorithm in order to show the effectiveness of weighted LLL reduction for our cryptanalysis. Here the variant adopts the LLL reduction in terms of the Euclidean norm in the first step of the original attack instead of the weighted LLL reduction. We measure the number of successes and time performance only if our attack succeeds, i.e., $\widetilde{m}$ or $- \widetilde{m}$ is recovered in the final step.

Table 1 indicates results of our experiments on our cryptanalysis of DEC instances. In Step 1 of the table, the number of successes is shown only if the target lattice point $\left( {\varvec{s}}^{\prime }_1, {\varvec{s}}^{\prime }_2 \right)$ or $- \left( {\varvec{s}}^{\prime }_1, {\varvec{s}}^{\prime }_2 \right)$ is found. For the target lattice point, see Step 1 of Attack Algorithm in Sect. 4.2. In Step 3 of the table, the number of successes is shown only if we succeeded in finding a twisted plaintext $\widetilde{m}$ (or $- \widetilde{m}$).

We see from the results of Step 1 in Table 1 that the weighted LLL reduction recovered our target lattice point in Step 1 with high probability, being about from 70 to 90%. On the other hand, we could not find the target lattice point with the usual LLL reduction in any case of our experiments (we omit to show the experimental results on the attack with the usual LLL reduction). We see from the results of Step 3 in Table  1 that our attack algorithm with the weighted LLL reduction could find the twisted plaintext $\widetilde{m}$ (or $- \widetilde{m}$) with sufficiently high probability, being about from 20 to 40%. We, however, could not succeed in finding the twisted plaintext at all by another one with the usual LLL reduction. From this, we infer that to adopt the weighted LLL reduction is quit important for our attack to succeed, and that our attack with the weighted LLL reduction has sufficiently high success probability for practical cryptanalysis.

From the point of view on the efficiency of generating keys and encryption/decryption, we consider that the parameters of Table 1 are practical. We also refer to Tables 4, 5 and 6 given in Sect. 6 of the designer’s paper [25]. We conclude from these experimental results that the attack algorithm can break, with sufficiently high probability, the one-way property of DEC in practical time.

Table 1 Experimental results on Attack Algorithm given in Sect. 4.2 for DEC of 128 bit level security with 4 indeterminates

Remark 27

The ranks of lattices occurring in Step 1 are equal to 3 in many cases. In fact, this is true for 100 instances of DEC constructed in our experiments. The LLL reduction finds shortest vectors in such lattices of low rank with high probability. In Step 1, a weighted norm is determined so that the target vector becomes a (nearly) shortest vector in terms of the norm. Thus the most important vector for our attack (the target vector in Step 1) is found by the weighted LLL reduction with high probability.

Remark 28

The existence of some failures of our attack suggests that there may exist a method to resist our attack. We analyzed some failure cases and found a reason why our attack failed in finding target lattice points in Steps 1 and 3. In Step 1 of each failure case, the weighted LLL algorithm found a shortest vector, but our target lattice point was not shortest. Similarly, in Step 3 of each failure case, our target lattice point was not a closest vector, while the Babai nearest plane algorithm found a closest vector. Therefore one may resist our attack if it is possible to choose random polynomials or public/secret keys such that our target lattice points are not shortest or closest in lattices ocurring in Steps 1 and 3. However, special choices of polynomials may lead us to another attack, and adding brute force methods to our attack seems to find target lattices points in such cases (see below). In order to resist our attack, we conclude that a major improvement of DEC is required. For example, the number of ciphertexts (polynomials) should be reduced from 3 to 2 or 1 because using 3 ciphertexts is essential to our attack.

On the other hand, we consider whether there is room for improving our attack or not. A simple improvement is to add steps of brute force search (with small range) to Steps 1 and 3. Our analysis in Sect. 4.1 suggests that our target vectors in Steps 1 and 3 are nearly shortest and nearly closest vectors, respectively, and thus our target vectors seem to be found by brute force methods with small range. However, we omit to conduct experiments on our attack with brute force methods. As mentioned above, we believe that our attack has already provided a practical solution to a problem of breaking DEC which is a candidate of PQC with small key sizes.

7 Conclusion

We present in this paper a polynomial time-attack based on the weighted LLL reduction against the one-way property of a Diophantine Equation-based Cryptosystem (DEC), which was proposed in 2015 by the third author of this paper as one of the candidates of Post-Quantum Cryptosystems (PQC). Compared with other well-known candidates of PQC, sizes of public keys in DEC are much smaller, e.g., about 1, 200 bits for 128 bit level security. This is a strongly desired characteristic for candidates of PQC.

Diophantine equations are generally unsolvable, and thus it is expected to be a base of the security of PQC. However, we showed that DEC’s security does not rely on the computational hardness to solve Diophantine equations, and that moreover DEC is no longer secure. Concretely, with linearlization technique, one can reduce breaking the one-way property of DEC to computing certain (comparatively) shorter points in low rank-lattices. Our most crucial target lattice point has the following special property: it is not necessarily a shortest lattice point whereas most of the entries are comparatively small. In our attack, even with the LLL reduction in terms of well-known norms, e.g., p-norms for $1 \le p \le \infty$, one seems to fail in finding such lattice points.

The most (heuristically-)technical point in our attack is changing the norm in the LLL reduction from the Euclidean norm to an appropriate weighted one. One can see from our analysis that the most important target lattice point becomes a (nearly) shortest lattice point in terms of a weighted norm, where the weight is determined by our heuristic method. Furthermore, the most important target lattice point is embedded in a (weighted) lattice of 3-rank, which implies the weighted LLL reduction can output with high probability such a target point. From this, we applied the weighted LLL reduction, which is the LLL reduction in terms of a weighted norm to our cryptanalysis. Our experimental results and complexity analysis suggest that for all the recommended parameters, the one-way property of DEC can be broken with sufficiently high probability by our polynomial time-attack based on the weighted LLL reduction.

We also demonstrated with our experimental results that the weighted LLL reduction gives an effective computational tool to find lattice points of special characteristic: the sizes of entries are almost known and most of them are small. Hence the weighted LLL reduction can provide a tool to investigate the security of cryptosystems whose security are transformed to the problem of computing such lattice points.