1 Introduction

Public-key cryptography is the foundation for establishing secure communication between multiple parties. Traditional public-key algorithms such as RSA are based on the hardness of factoring large numbers or the discrete logarithm problem, but can be attacked in polynomial time once a capable quantum computer exists. Code-based public-key cryptosystems are considered to be post-quantum secure, but compared to RSA or elliptic curve cryptography their crucial drawback is the significantly larger key size. Recently, the National Institute of Standards and Technology (NIST) has initiated a standardization progress for post-quantum secure public-key algorithms [31]. The currently being evaluated Round 3 of the competition includes 9 code-based and lattice-based public-key encryption algorithms. The NIST competition and its systems attract a lot of attention and show the importance of designing post-quantum secure public-key encryption algorithms.

The Faure–Loidreau (FL) code-based cryptosystem [16, 28] is based on the problem of reconstructing linearized polynomials and can be seen as linearized equivalent of the (broken) Augot–Finiasz cryptosystem [5]. While the Augot–Finiasz cryptosystem is closely connected to (list) decoding Reed–Solomon codes, the FL cryptosystem is connected to (list) decoding Gabidulin codes, a special class of rank-metric codes [18]. In contrast to McEliece-type (or Niederreiter-type) cryptosystems, where the public key is a matrix, in the FL system, the public key is only a vector, resulting in a much smaller key size. At the time when the FL cryptosystem was designed, it was only conjectured that Gabidulin codes cannot be list decoded efficiently. As this was proven in the last years for many families of Gabidulin codes [38, 46, 48], the FL system could be a very promising post-quantum secure public-key cryptosystem. However, the recent structural attack by Gaborit, Otmani and Talé Kalachi [21] can recover an alternative public key in cubic time complexity.

In this paper, a new system is presented which is based on the original FL system, and therefore relies on the proven hardness of list decoding Gabidulin codes, but makes the attack from [21] impossible. Our contributions are as follows. First, a new coding-theoretic interpretation of the original FL system is given and an alternative decryption algorithm is proposed. Second, we show that the public key can be seen as a corrupted codeword of an interleaved Gabidulin code. We prove that the failure condition of the GOT attack [21] on the public key is equivalent to the failure condition of decoding the public key as a corrupted interleaved Gabidulin codeword. This observation enables us to design a new code-based public-key encryption scheme, as well as a corresponding key encapsulation mechanism - data encapsulation mechanism (KEM-DEM), based on the hardness of list and interleaved decoding Gabidulin codes: LIGA. In LIGA, we choose the public key in a way that the corresponding interleaved decoder is guaranteed to fail, and thus, the system is secured against the attack from [21]. We also prove that the public-key encryption version of LIGA is IND-CPA secure in the standard model and the KEM version is IND-CCA2 secure in the random oracle model, both under hardness assumptions on problems related to list and interleaved decoding of Gabidulin codes. We analyze possible (exponential-time) attacks on these hard problems, provide sets of parameters for LIGA, and compare them amongst others to NIST proposals (RQC, ROLLO, BIKE, McEliece).

The structure of this paper is as follows. In Sect. 2, the notation is introduced and definitions are given. In Sect. 3, the key generation of the original FL system is shown and a new coding-theoretic interpretation of the ciphertext and the public key is derived. After summarizing the attack from [21], we prove its equivalence to decoding the public key as an interleaved Gabidulin code. Based on this equivalence, the new system LIGA is proposed in Sect. 4 and its IND-CPA and IND-CCA2 security are proven in Sect. 5. A security analysis of our system is given in Sect. 6. In Sect. 7, example parameters for security levels 128, 192, and 256 bit are proposed and compared to the NIST proposals RQC [2], ROLLO [1], BIKE [3], ClassicMcEliece [8] and Loidreau’s McEliece-like system from [29]. Conclusions are given in Sect. 8.

Parts of these results have been presented at the IEEE International Symposium on Information Theory 2018 [50]. The content of this journal paper contains various new results that were not shown in [50]. For instance, in this paper,

  • we generalize LIGA’s Key Generation algorithm, i.e., the choice of the \(\mathbf {z}_i\)’s (the interleaved errors in the public key) is more flexible now (in [50], \(\mathbf {z}_1 = \mathbf {z}_2 = \dots = \mathbf {z}_u\)),

  • we present a KEM/DEM version of LIGA,

  • we identify formal problems in the rank metric on which the security of LIGA relies and prove the IND-CPA/CCA2 security of the KEM/DEM version under the assumption that some of these problems are hard,

  • we analyze new exponential-time attacks on these problems.

2 Preliminaries

2.1 Notations

Let q be a power of a prime and let \(\mathbb {F}_q\) denote the finite field of order q. Then, \(\mathbb {F}_{q^m}\) and \(\mathbb {F}_{q^{mu}}\) denote extension fields of \(\mathbb {F}_q\) of order \(q^m\) and \(q^{mu}\), respectively. We use \(\mathbb {F}_q^{m \times n}\) to denote the set of all \(m\times n\) matrices over \(\mathbb {F}_q\) and \(\mathbb {F}_{q^m}^n =\mathbb {F}_{q^m}^{1 \times n}\) for the set of all row vectors of length n over \(\mathbb {F}_{q^m}\). Further, we use another field extension \(\mathbb {F}_{q^{mu}}\) with \(u>1\). Thus, \(\mathbb {F}_q\subseteq \mathbb {F}_{q^m}\subseteq \mathbb {F}_{q^{mu}}\).

For a field \(\mathbb {F}\), the vector space that is spanned by \(\mathbf {v}_1,\hdots ,\mathbf {v}_l \in \mathbb {F}^n\) is denoted by

$$\begin{aligned} \langle \mathbf {v}_1,\hdots ,\mathbf {v}_l\rangle _{\mathbb {F}} := \Bigg \{\sum _{i=1}^{l}a_i\mathbf {v}_i \, : \, \ a_i \in \mathbb {F} \Bigg \}. \end{aligned}$$

Denote the set of integers \([a,b] = \{i: a \le i \le b\}\). Rows and columns of \(m\times n\)-matrices are indexed by \(1,\dots , m\) and \(1,\dots , n\), where \(A_{i,j}\) is the element in the i-th row and j-th column of the matrix \(\mathbf {A}\). Further,

$$\begin{aligned} \mathbf {A}_{[a,b]} := \begin{pmatrix} A_{1,a} &{} \hdots &{} A_{1,b} \\ \vdots &{} \ddots &{} \vdots \\ A_{m,a} &{} \hdots &{} A_{m,b} \\ \end{pmatrix}. \end{aligned}$$

By \({{\,\mathrm{rk}\,}}_q(\mathbf {A})\) and \({{\,\mathrm{rk}\,}}_{q^m}(\mathbf {A})\), we denote the rank of a matrix \(\mathbf {A}\) over \(\mathbb {F}_q\), respectively \(\mathbb {F}_{q^m}\). Let \((\gamma _1,\gamma _2,\dots ,\gamma _{u})\) be an ordered basis of \(\mathbb {F}_{q^{mu}}\) over \(\mathbb {F}_{q^m}\). By utilizing the vector space isomorphism \(\mathbb {F}_{q^{mu}}\cong \mathbb {F}_{q^m}^u\), we can relate each vector \(\mathbf{a} \in \mathbb {F}_{q^{mu}}^n\) to a matrix \(\mathbf{A} \in \mathbb {F}_{q^m}^{u \times n}\) according to

$$\begin{aligned} {{\,\mathrm{ext}\,}}_{\varvec{\gamma }}:\mathbb {F}_{q^m}^{n}&\rightarrow \mathbb {F}_q^{m \times n}\\ \mathbf {a} = (a_1,\hdots ,a_n)&\mapsto \mathbf {A} = \begin{pmatrix} A_{1,1} &{} \hdots &{} A_{1,n} \\ \vdots &{} \ddots &{} \vdots \\ A_{m,1} &{} \hdots &{} A_{m,n} \\ \end{pmatrix}, \end{aligned}$$

where \(\varvec{\gamma } = (\gamma _1,\gamma _2,\dots ,\gamma _{u})\) and

$$\begin{aligned} \quad a_j = \sum _{i=1}^{m} A_{i,j} \gamma _i, \quad \forall j \in [1,n]. \end{aligned}$$

The trace operator of a vector \(\mathbf {a}\in \mathbb {F}_{q^{mu}}\) to \(\mathbb {F}_{q^m}\) is defined by

$$\begin{aligned} {{\,\mathrm{Tr}\,}}:\mathbb {F}_{q^{mu}}^{n}&\rightarrow \mathbb {F}_{q^m}^{n}\\ \mathbf {a} = (a_1,a_2,\hdots ,a_n)&\mapsto {\Bigg ( \sum _{i=0}^{u-1} a_1^{q^{mi}}, \sum _{i=0}^{u-1} a_2^{q^{mi}},\hdots , \sum _{i=0}^{u-1} a_n^{q^{mi}} \Bigg )}. \end{aligned}$$

A dual basis \((\gamma _1^*,\gamma _2^*,\dots ,\gamma _{u}^*)\) to \((\gamma _1,\gamma _2,\dots ,\gamma _{u})\) is a basis that fulfills

$$\begin{aligned} {{\,\mathrm{Tr}\,}}(\gamma _i \gamma _j^{*}) = {\left\{ \begin{array}{ll} 1 ~\text {if}\quad i=j\\ 0 ~\text {else} \end{array}\right. }, \end{aligned}$$

where \(i,j \in [1,u]\). Note that a dual basis always exists.

Denote by \(\mathcal {M}_{s,q}\left( \mathbf {a} \right) \in \mathbb {F}_{q^m}^{s \times n}\) the \(s \times n\) Moore matrix for a vector \(\mathbf {a} = (a_1,a_2,\dots ,a_n) \in \mathbb {F}_{q^m}^n\), i.e.,

$$\begin{aligned} \mathcal {M}_{s,q}\left( \mathbf {a} \right) := \begin{pmatrix} a_{1} &{} a_{2} &{} \dots &{} a_{n}\\ a_{1}^{q} &{} a_{2}^{q} &{} \dots &{} a_{n}^{q}\\ \vdots &{}\vdots &{}\ddots &{} \vdots \\ a_{1}^{q^{s-1}} &{} a_{2}^{q^{s-1}} &{} \dots &{} a_{n}^{q^{s-1}}\\ \end{pmatrix}. \end{aligned}$$

If \(a_1, a_2,\dots \), \(a_{n}\in \mathbb {F}_{q^m}\) are linearly independent over \(\mathbb {F}_q\), then \({{\,\mathrm{rk}\,}}_{q^m}(\mathcal {M}_{s,q}\left( \mathbf {a} \right) )=\min \{s,n\}\), cf. [26, Lemma 3.15]. This definition can also be extended to matrices by

$$\begin{aligned} \mathcal {M}_{s,q}\left( \mathbf {A} \right) := \begin{pmatrix} A_{1,1} &{} A_{1,2} &{} \dots &{} A_{1,n}\\ A_{2,1} &{} A_{2,2} &{} \dots &{} A_{2,n}\\ \vdots &{}\vdots &{}\ddots &{} \vdots \\ A_{l,1} &{} A_{l,2} &{} \dots &{} A_{l,n}\\ A_{1,1}^{q} &{} A_{1,2}^{q} &{} \dots &{} A_{1,n}^{q}\\ A_{2,1}^{q} &{} A_{2,2}^{q} &{} \dots &{} A_{2,n}^{q}\\ \vdots &{}\vdots &{}\ddots &{} \vdots \\ A_{l,1}^{q^{s-1}} &{} A_{l,2}^{q^{s-1}} &{} \dots &{} A_{l,n}^{q^{s-1}}\\ \end{pmatrix}, \end{aligned}$$

where \(\mathbf {A} \in \mathbb {F}_{q^m}^{l \times n}\).

The Gaussian binomial coefficient is denoted by

$$\begin{aligned} {s \brack r}_q := {\left\{ \begin{array}{ll} \dfrac{ (1-q^s)(1-q^{s-1})\cdots (1-q^{s-r+1})}{(1-q)(1-q^2)\cdots (1-q^{r})} &{}\text { for }r \le s\\ 0 &{}\text { for }r> s, \end{array}\right. } \end{aligned}$$

where s and r are non-negative integers.

Let \(\mathcal {X}\) be a set. When x is drawn uniformly at random from the set \(\mathcal {X}\), we denote it by \(x \xleftarrow {\$}\mathcal {X}\). Further, by \(x \leftarrow y\) we mean that we assign y to x.

2.2 Rank-metric codes and Gabidulin codes

The rank norm \({{\,\mathrm{rk}\,}}_q(\mathbf {a})\) is the rank of the matrix representation \(\mathbf {A}\in \mathbb {F}_q^{m \times n}\) over \(\mathbb {F}_{q}\). The rank distance between \(\mathbf {a}\) and \(\mathbf {b}\) is the rank of the difference of the two matrix representations, i.e.,

$$\begin{aligned} d_{R }(\mathbf {a},\mathbf {b}) := {{\,\mathrm{rk}\,}}_q(\mathbf {a}-\mathbf {b}) = {{\,\mathrm{rk}\,}}_q(\mathbf {A}-\mathbf {B}). \end{aligned}$$

An \([n,k,d]_q^\mathsf {R}\) code \(\mathcal {C}\) over \(\mathbb {F}_{q^m}\) is a linear rank-metric code, i.e., it is a linear subspace of \(\mathbb {F}_{q^m}^n\) of dimension k and minimum rank distance

$$\begin{aligned} d := \min _{\begin{array}{c} {\mathbf {a},\mathbf {b}} \in \mathcal {C}\\ \mathbf {a} \ne \mathbf {b} \end{array}} \big \lbrace d_{\mathsf {R}}(\mathbf {a},\mathbf {b}) = {{\,\mathrm{rk}\,}}_q(\mathbf {a}-\mathbf {b}) \big \rbrace . \end{aligned}$$

For linear codes with \(n \le m\), the Singleton-like upper bound [14, 18] implies that \(d \le n-k+1\). If \(d=n-k+1\), the code is called a maximum rank distance (MRD) code.

Gabidulin codes [18] are a special class of rank-metric codes and can be defined by their generator matrices.

Definition 1

(Gabidulin Code [18]) A linear \(\mathcal {G}(n,k)\) code over \(\mathbb {F}_{q^m}\) of length \(n \le m\) and dimension k is defined by its \(k \times n\) generator matrix

$$\begin{aligned} \mathbf{G}_{\mathcal {G}} = \mathcal {M}_{k,q}\left( g_1,g_2,\dots ,g_n \right) , \end{aligned}$$

where \(\mathbf {g}=(g_1,g_2, \dots , g_{n}) \in \mathbb {F}_{q^m}^n\) and \({{\,\mathrm{rk}\,}}_q(\mathbf {g}) = n\).

In [18], it is shown that Gabidulin codes are MRD codes, i.e., \(d=n-k+1\).

For a short description on decoding of Gabidulin codes, denote by \(\mathbf {C}_\mathcal {G}\in \mathbb {F}_q^{m \times n}\) the transmitted codeword (i.e., the matrix representation of \(\mathbf {c}_\mathcal {G}\in \mathbb {F}_{q^m}^n\)) of a \(\mathcal {G}(n,k)\) code that is corrupted by an additive error \(\mathbf{E} \in \mathbb {F}_q^{m \times n}\). At the receiver side, only the received matrix \(\mathbf{R} \in \mathbb {F}_q^{m \times n}\), where \(\mathbf{R} = \mathbf {C}_\mathcal {G}+ \mathbf{E}\), is known. The channel might provide additional side information in the form of erasures:

  • \(\varrho \) row erasures (in [45] called “deviations”) and

  • \(\gamma \) column erasures (in [45] called “erasures”),

such that the received matrix can be decomposed into

$$\begin{aligned} \mathbf {R} =\mathbf {C}_\mathcal {G}+ \underbrace{\mathbf {A}^{(R)} \mathbf {B}^{(R)} + \mathbf {A}^{(C)} \mathbf {B}^{(C)} + {\mathbf {E}^{(E)}}}_{= \,\mathbf {E}_\mathrm {total}}, \end{aligned}$$

where \(\mathbf {A}^{(R)} \in \mathbb {F}_q^{m \times \varrho }\), \(\mathbf {B}^{(R)} \in \mathbb {F}_q^{\varrho \times n}\), \(\mathbf {A}^{(C)} \in \mathbb {F}_q^{m \times \gamma }\), \(\mathbf {B}^{(C)} \in \mathbb {F}_q^{\gamma \times n}\) are full-rank matrices, respectively, and \(\mathbf {E}^{(E)} \in \mathbb {F}_q^{m \times n}\) is a matrix of rank t. The decoder knows \(\mathbf {R}\) and additionally \(\mathbf {A}^{(R)}\) and \(\mathbf {B}^{(C)}\). Further, t denotes the number of errors without side information. The rank-metric error-erasure decoding algorithms from [20, 45, 51] can then reconstruct \(\mathbf {c}_\mathcal {G} \in \mathcal {G}(n,k)\) with asymptotic complexity \(\mathcal O(n^2)\) operations over \(\mathbb {F}_{q^m}\), or in sub-quadratic complexity using the fast operations described in [36, 37], if

$$\begin{aligned} 2t + \varrho + \gamma \le d-1 = n-k \end{aligned}$$

is fulfilled.

2.3 Interleaved rank-metric codes

Interleaved Gabidulin Codes are a code class for which efficient decoders are known that are able to correct w.h.p. random errors of rank larger than \(\lfloor \frac{d-1}{2}\rfloor \).

Definition 2

(Interleaved Gabidulin Codes [27]) A linear (vertically, homogeneous) interleaved Gabidulin code \(\mathcal {IG}(u;n,k)\) over \(\mathbb {F}_{q^m}\) of length \(n \le m\), dimension \(k \le n\), and interleaving order u is defined by

$$\begin{aligned} \mathcal {IG}(u;n,k) := \left\{ \begin{pmatrix} \mathbf {c}_{\mathcal {G}}^{(1)}\\ \mathbf {c}_{\mathcal {G}}^{(2)}\\ \vdots \\ \mathbf {c}_{\mathcal {G}}^{(u)} \end{pmatrix} : \mathbf {c}_{\mathcal {G}}^{(i)} \in \mathcal {G}(n,k) , \forall i \in [1,u] \right\} . \end{aligned}$$

As a short-term notation, we also speak about a u-interleaved Gabidulin code. When considering random errors of rank weight t, the code \(\mathcal {IG}(u;n,k)\) can be decoded uniquely with high probability up to \(w \le \lfloor \frac{u}{u+1}(n-k)\rfloor \) errors,Footnote 1 cf. [27, 43, 51]. However, it is well-known that there are many error patterns for which the known efficient decoders fail. In fact, we can explicitly construct a large class of such errors as shown in the following lemma.

Lemma 1

(Interleaved Decoding [27, 43, 49, p. 64]) Let \(\mathbf {c}_i=\mathbf {x}_i \cdot \mathbf {G}_{\mathcal {G}}\). All knownFootnote 2 efficient decoders for \(\mathcal {IG}(u;n,k)\) codes fail to correct an error \(\mathbf {z}\in \mathbb {F}_{q^{mu}}^n\) with \(\mathbf {z} = \sum _{i=1}^{u} \mathbf {z}_i\gamma _i^*\) and \({{\,\mathrm{rk}\,}}_q(\mathbf {z})=w\) if

$$\begin{aligned} {{\,\mathrm{rk}\,}}_{q^m} \begin{pmatrix} \mathcal {M}_{n-w-1,q}\left( \mathbf {g} \right) \\ \mathcal {M}_{n-k-w,q}\left( \mathbf {c}_1+\mathbf {z}_1 \right) \\ \mathcal {M}_{n-k-w,q}\left( \mathbf {c}_2+\mathbf {z}_2 \right) \\ \vdots \\ \mathcal {M}_{n-k-w,q}\left( \mathbf {c}_u+\mathbf {z}_u \right) \\ \end{pmatrix} < n-1. \end{aligned}$$

It is widely conjectured that there cannot be a decoder that decodes the error patterns of Lemma 1uniquely. Decoding these failing error patterns has been subject to intensive research since the Loidreau–Overbeck decoder [27] was found in 2006. In the Hamming metric, the equivalent problem for Reed–Solomon codes has been studied since 1997 [25] and more than a dozen papers have dealt with decoding algorithms for these codes. None of these papers was able to give a polynomial-time decoding algorithm for the cases of Lemma 1. It seems that all unique decoders have to fail for the error patterns of Lemma 1 since for these cases, there is no unique decision, i.e., more then one interleaved codeword lies in the ball of radius w around the received word.

3 Key generation in the original Faure–Loidreau system

In this section, we recall the key generation algorithm of the original FL cryptosystem, we give a coding-theoretic interpretation of the original public key, and analyze the structural attack from [21].

3.1 The original algorithm

Let \(q,m,n,k,u,w,t_{\mathsf {pub}}\) be positive integers that fulfill the restrictions given in Table 1 and are publicly known. In the following, we consider the three finite fields \(\mathbb {F}_q\), \(\mathbb {F}_{q^m}\), and \(\mathbb {F}_{q^{mu}}\), which are extension fields of each other, i.e.:

$$\begin{aligned} \mathbb {F}_q\subseteq \mathbb {F}_{q^m}\subseteq \mathbb {F}_{q^{mu}}. \end{aligned}$$
Table 1 Summary of the parameters

The original FL key generation is shown in Algorithm 1.

figure a

3.2 Coding-theoretic interpretation of the original public key

The public key \(\mathbf {k}_{\mathsf {pub}}\) of the FL system is a corrupted codeword of a u-interleaved Gabidulin code. To our knowledge, this connection between the public key and interleaved Gabidulin codes has not been known before. This interpretation is central to this paper and will be used in Sect. 4.1 to define the public key of LIGA such that is not vulnerable against the attacks from [21] and described in Sect. 3.3.

Theorem 1

Fix a basis \(\mathbf {\gamma }\) of \(\mathbb {F}_{q^{mu}}\) over \(\mathbb {F}_{q^m}\). Let \(\mathbf {\gamma }^*\) be a dual basis to \(\mathbf {\gamma }\) and write \(\mathbf {k}_{\mathsf {pub}}= \sum _{i=1}^{u} \mathbf {k}_{\mathsf {pub}}^{(i)}\gamma _i^*\). Then,

$$\begin{aligned} \begin{pmatrix} \mathbf {k}_{\mathsf {pub}}^{(1)}\\ \mathbf {k}_{\mathsf {pub}}^{(2)}\\ \vdots \\ \mathbf {k}_{\mathsf {pub}}^{(u)} \end{pmatrix} = \begin{pmatrix} \mathbf {c}_{\mathcal {G}}^{(1)}\\ \mathbf {c}_{\mathcal {G}}^{(2)}\\ \vdots \\ \mathbf {c}_{\mathcal {G}}^{(u)} \end{pmatrix} +\! \begin{pmatrix} \mathbf {z}_1\\ \mathbf {z}_2\\ \vdots \\ \mathbf {z}_u \end{pmatrix}, \end{aligned}$$

where the \(\mathbf {c}_{\mathcal {G}}^{(i)} \in \mathbb {F}_{q^m}^n\) are codewords of the Gabidulin code \(\mathcal {G}(n,k)\) with generator matrix \(\mathbf {G}_\mathcal {G}\) and the \(\mathbf {z}_i \in \mathbb {F}_{q^m}^n\) are obtained from the vector \(\mathbf {z} \in \mathbb {F}_{q^{mu}}^n\) by \(\mathbf {z} = \sum _{i=1}^{u} \mathbf {z}_i\gamma _i^*\).


Recall the definition of the public key

$$\begin{aligned} \mathbf {k}_{\mathsf {pub}}= \mathbf {x} \cdot \mathbf{G}_{\mathcal {G}} + \mathbf {z}, \end{aligned}$$

where \(\mathbf {x} \in \mathbb {F}_{q^{mu}}^k\), \(\mathbf {G}_{\mathcal {G}} \in \mathbb {F}_{q^m}^{k\times n}\) is the generator matrix of a \(\mathcal {G}(n,k)\) code, and \(\mathbf {z} \in \mathbb {F}_{q^{mu}}^n\) with \({{\,\mathrm{rk}\,}}_q(\mathbf {z})=w\). Let \(\mathbf {x} = \sum _{i=1}^{u} \mathbf {x}_i\gamma _i^*\), where the \(\mathbf {x}_i\) have coefficients in \(\mathbb {F}_{q^m}\).

Then, we obtain the following representation of the public key \(\mathbf {k}_{\mathsf {pub}}\) as a \(u \times n\) matrix in \(\mathbb {F}_{q^m}\)

$$\begin{aligned} \begin{pmatrix} \mathbf {k}_{\mathsf {pub}}^{(1)}\\ \mathbf {k}_{\mathsf {pub}}^{(2)}\\ \vdots \\ \mathbf {k}_{\mathsf {pub}}^{(u)} \end{pmatrix} = \begin{pmatrix} \mathbf {x}_1\\ \mathbf {x}_2\\ \vdots \\ \mathbf {x}_u \end{pmatrix}\!\cdot \mathbf {G}_{\mathcal {G}} +\! \begin{pmatrix} \mathbf {z}_1\\ \mathbf {z}_2\\ \vdots \\ \mathbf {z}_u \end{pmatrix} = \begin{pmatrix} \mathbf {x}_1\cdot \mathbf {G}_{\mathcal {G}}\\ \mathbf {x}_2\cdot \mathbf {G}_{\mathcal {G}}\\ \vdots \\ \mathbf {x}_u\cdot \mathbf {G}_{\mathcal {G}} \end{pmatrix} +\! \begin{pmatrix} \mathbf {z}_1\\ \mathbf {z}_2\\ \vdots \\ \mathbf {z}_u \end{pmatrix}. \end{aligned}$$

Since \(\mathbf {x}_i \cdot \mathbf {G}_{\mathcal {G}}\) is a codeword of a \(\mathcal {G}(n,k)\) code, \(\forall i \in [1,u]\), the matrix representation of \(\mathbf {k}_{\mathsf {pub}}\) can be seen as a codeword from an \(\mathcal {IG}(u;n,k)\) code, corrupted by an error. \(\square \)

Note that the error \((\mathbf {z}_1^\top ,\dots ,\mathbf {z}_u^\top )^\top \) in (3) has \(\mathbb {F}_q\)-rank at most w due to the structure of \(\mathbf {z}= (\mathbf {s}|\mathbf {0}) \mathbf {P}^{-1}\).

3.3 Efficient key recovery of the original FL key

The attack by Gaborit, Otmani and Talé Kalachi (GOT) on the original FL system in [21] (see Algorithm 2 below) is an efficient structural attack which computes a valid private key of the FL system in cubic time when the public key fulfills certain conditions. We recall this attack in the following and derive an alternative, equally powerful, attack based on interleaved decoding the public key, utilizing the observation of the previous subsection. We prove that the failure conditions of both attacks are equivalent. The interleaved decoding attack does not have any advantage in terms of cryptanalysis compared to [21], but enables us to exactly predict for which public keys both attacks work and for which the attacks fail.

3.3.1 GOT attack

The key recovery in the GOT attack (Algorithm 2) succeeds under the conditions of the following theorem.

figure b

Theorem 2

(GOT Attack [21, Thm. 1]) Let \(\gamma _1,\dots ,\gamma _u \in \mathbb {F}_{q^{mu}}\) be a basis of \(\mathbb {F}_{q^{mu}}\) over \(\mathbb {F}_{q^m}\) and let \(\mathbf {z}_i = {{\,\mathrm{Tr}\,}}(\gamma _i \mathbf {z})\), for \(i=1,\dots u\).

If the matrix \(\mathbf {Z} \in \mathbb {F}_{q^m}^{u \times n}\) with \(\mathbf {z}_1,\dots ,\mathbf {z}_u\) as rows, satisfies

$$\begin{aligned} {{\,\mathrm{rk}\,}}_{q^m} (\mathcal {M}_{n-k-w,q}\left( \mathbf {Z} \right) ) = w, \end{aligned}$$

then \((\mathbf {x}, \mathbf {z})\) can be recovered from \((\mathbf {G}_{\mathcal {G}},\mathbf {k}_{\mathsf {pub}})\) with \(\mathcal {O}(n^3)\) operations in \(\mathbb {F}_{q^{mu}}\) by using Algorithm 2.

If the key is generated by Algorithm 1, the GOT attack breaks the original FL system with high probability.

3.3.2 Interleaved decoding attack

Recall from Theorem 1 that the public key \(\mathbf {k}_{\mathsf {pub}}\) is a corrupted interleaved codeword. Based on this observation we will derive a structural attack on the original FL system to which we refer as Interleaved Decoding Attack in the following. We prove that interleaved decoding and the GOT attack fail (i.e, do not provide any information) for the same public keys. The idea is to decode \(\mathbf {k}_{\mathsf {pub}}\) in an interleaved Gabidulin code. Since \(w \le \frac{u}{u+1}(n-k)\), such a decoder will return \(\mathbf {x}\) with high probability, but fail in certain cases, see Sect. 2.3.

Since \({{\,\mathrm{rk}\,}}_{q^m}(\mathcal {M}_{n-w-1,q}\left( \mathbf {g} \right) )=n-w-1\), the interleaved decoder fails if (compare Lemma 1):

$$\begin{aligned} {{\,\mathrm{rk}\,}}_{q^m} \big (\tilde{\mathbf {Z}}\big ) := \varphi < w, \end{aligned}$$


$$\begin{aligned} \tilde{\mathbf {Z}}= \begin{pmatrix} \mathcal {M}_{n-k-w,q}\left( \mathbf {z}_1 \right) \\ \mathcal {M}_{n-k-w,q}\left( \mathbf {z}_2 \right) \\ \vdots \\ \mathcal {M}_{n-k-w,q}\left( \mathbf {z}_u \right) \end{pmatrix}. \end{aligned}$$

3.3.3 Equivalence of GOT attack and interleaved decoding attack

In the following, we prove that the failure condition of the GOT Attack is equivalent to the condition that decoding \(\mathbf {k}_{\mathsf {pub}}\) in an interleaved Gabidulin code fails.

Theorem 3

The GOT Attack from [21] fails if and only if the Interleaved Decoding Attack fails. In particular, both fail if (4) holds.


Rewrite the matrix from Theorem 2 as

$$\begin{aligned} \mathcal {M}_{n-w-k,q}\left( \mathbf {Z} \right) = \begin{pmatrix} \mathbf {z}_1\\ \vdots \\ \mathbf {z}_u\\ \mathbf {z}_1^{q}\\ \vdots \\ \mathbf {z}_u^{q}\\ \vdots \\ \mathbf {z}_1^{q^{n-w-k-1}}\\ \vdots \\ \mathbf {z}_u^{q^{n-w-k-1}}\\ \end{pmatrix} \end{aligned}$$

and the matrix from equation (5) as

$$\begin{aligned} \tilde{\mathbf {Z}}= \begin{pmatrix} \mathbf {z}_1\\ \vdots \\ \mathbf {z}_1^{q^{n-w-k-1}}\\ \mathbf {z}_2\\ \vdots \\ \mathbf {z}_2^{q^{n-w-k-1}}\\ \vdots \\ \mathbf {z}_u\\ \vdots \\ \mathbf {z}_u^{q^{n-w-k-1}}\\ \end{pmatrix}. \end{aligned}$$

Since the matrix in (6) and in (7) only differ in row permutations, they are row-space equivalent, implying that they have the same rank. Further, the rank of the matrix in (7) cannot become larger than w (since any vector in the right kernel of this matrix has rank weight at least \(n-w\) [34, Algorithm 3.2.1]). Thus, the failures of Theorem 2 and Lemma 1 are equivalent. \(\square \)

In the next section, we will exploit the observation of Theorem 3, i.e., we propose a new key generation algorithm that avoids public keys that can be efficiently decoded by an interleaved decoder, thereby rendering the GOT attack useless.

4 The new system LIGA

In this section, we propose a public-key code-based encryption scheme \({{\Pi }}^{\mathrm{PKE}}= (\mathsf {KeyGen},\mathsf {Encrypt},\mathsf {Decrypt})\) called \(\textsf {LIGA}\). The system is based on the original FL system [16], where we keep both the original encryption and decryption algorithm, but replace the insecure key-generation algorithm. Further, we present a KEM-DEM version of \(\textsf {LIGA}\) denoted by \({\Pi }^{\mathrm{KEM}}=(\mathsf {KeyGen},\mathsf {Encaps},\mathsf {Decaps})\).

Later, in Sect. 5, we will analyze the security of the system. We single out problems from coding theory and we prove that the encryption version is IND-CPA secure and the KEM-DEM version is IND-CCA2 secure under the assumption that the stated problems are hard. Furthermore, we study new and known attacks on these problems and show that they all run in exponential time (see Sect. 6).

4.1 The new key generation algorithm

We introduce a new key generation algorithm that is based on choosing \(\mathbf {z} = \sum _{i=1}^{u} \mathbf {z}_i\gamma ^*_i\) in a way that \(\varphi < w\), where \(\varphi \) is the rank of the interleaved Moore matrix of the errors \(\mathbf {z}_i\) in the public key, see (5). Based on the dimension of the span of the \(\mathbf {z}_i\), we will upper bound \(\varphi \) in the following Theorem 4. Recall that when \(\varphi < w\), the GOT attack [21] and interleaved decoding of the public key fail, see Theorem 3. In this case, retrieving any knowledge about the private key from the public key requires to solve Problem 1 (defined later), which basically corresponds to decoding the interleaved codeword when error patterns occur for which all known decoders fail.

Theorem 4

Let \(\dim (\langle \mathbf {z}_1,\hdots ,\mathbf {z}_u \rangle _{q^m}) = \zeta \). Then

$$\begin{aligned} \varphi = {{\,\mathrm{rk}\,}}_{q^m} \big (\tilde{\mathbf {Z}}\big ) \le \min \{\zeta (n-k-w),w\}. \end{aligned}$$


The dimension of \(\langle \mathbf {z}_1,\hdots ,\mathbf {z}_u \rangle _{q^m}\) implies that at most \(\zeta (n-k-w)\) rows of \(\tilde{\mathbf {Z}}\) are linearly independent over \(\mathbb {F}_{q^m}\), meaning that \(\varphi \le \zeta (n-k-w)\).

The definition of \(\mathbf {z}=(\mathbf {s} \ | \ \mathbf {0}) \cdot \mathbf {P}^{-1}\) leads to

$$\begin{aligned} \varphi&= {{\,\mathrm{rk}\,}}_{q^m}(\tilde{\mathbf {Z}}) \\&= {{\,\mathrm{rk}\,}}_{q^m} \begin{pmatrix} \begin{pmatrix} \begin{array}{c | c} \mathcal {M}_{n-k-w,q}\left( \mathbf {s}_1 \right) &{} \mathbf {0}\\ \vdots &{} \vdots \\ \mathcal {M}_{n-k-w,q}\left( \mathbf {s}_u \right) &{} \mathbf {0}\\ \end{array} \end{pmatrix}\mathbf {P}^{-1} \end{pmatrix} \\&= {{\,\mathrm{rk}\,}}_{q^m} \begin{pmatrix} \begin{array}{c} \mathcal {M}_{n-k-w,q}\left( \mathbf {s}_1 \right) \\ \vdots \\ \mathcal {M}_{n-k-w,q}\left( \mathbf {s}_u \right) \\ \end{array} \end{pmatrix} \\&\le w, \end{aligned}$$

where the last inequality holds since \(\mathbf {s}_1,\hdots ,\mathbf {s}_u\) are vectors of length w. \(\square \)

We propose the following modification to Line 3 of the Key Generation, depending on the parameter \(\zeta \):

Clearly, \(\dim (\langle \mathbf {z}_1,\hdots ,\mathbf {z}_u \rangle _{q^m}) = \zeta \) in this case. To avoid that the GOT attack [21] runs in polynomial time, Theorem 4 implies that the parameter \(\zeta \) must always be chosen such that \(\zeta < \frac{w}{n-k-w}\). In Sect. 6, we will discuss several further exponential-time attacks on LIGA. Some of these attacks have a work factor depending on \(\zeta \), which must be considered in the parameter design.

Furthermore, the condition \({{\,\mathrm{rk}\,}}_{q}(\mathbf {s}_i') = w\) ensures that \({{\,\mathrm{rk}\,}}_{q}(\mathbf {z}_i)=w\), i.e., as large as possible for a given subspace \(\mathcal {A}\). This choice maximizes the work factor of generic decoding attacks on the rows of the public key (seen as a received word of an interleaved Gabidulin code), see Sect. 6.

The restriction of the choice of \(\mathcal {A}\) to subspaces that contain a basis of full-\(\mathbb {F}_q\)-rank codewords is to ensure that the set from which we sample in Line 3’ is non-empty. Hence, the key generation always works.

Compared to the choice of \(\mathbf {z}\) in Line 3 of the original Key Generation algorithm, we restrict the choice of \(\mathbf {z}\), but we will see in Sect. 6 that there are still enough possibilities for \(\mathbf {z}\) to prevent an efficient naive brute-force attack.

Appendix 1 contains a more detailed discussion on how to realize Lines 3 and \(3'\) in practice.

4.2 The public key encryption version

The new key generation algorithm \(\mathsf {KeyGen}\), the encryption algorithm \(\mathsf {Encrypt}\) and the decryption algorithm \(\mathsf {Decrypt}\) are shown in Algorithm 3, Algorithm 4 and Algorithm 5, respectively. Compared to original key generation algorithm, the algorithm \(\mathsf {KeyGen}\) has one more input parameter \(\zeta \) (cf. Sect. 4.1).

The proposed system has no decryption failures as proven in the following theorem.

Theorem 5

(Correctness [16]) Algorithm 5 returns the correct plaintext \(\mathbf {m}\).


Line 1 computes

$$\begin{aligned} \mathbf {c}\mathbf {P} = (\mathbf {m} + {{\,\mathrm{Tr}\,}}(\alpha \mathbf {x}))\mathbf {G}_{\mathcal {G}}\mathbf {P} + ({{\,\mathrm{Tr}\,}}(\alpha \mathbf {s}) | \mathbf {0}) + \mathbf {e}\mathbf {P}, \end{aligned}$$

whose last \(n-w\) columns are given by

$$\begin{aligned} \mathbf {c}^{\prime } = (\mathbf {m} + {{\,\mathrm{Tr}\,}}(\alpha \mathbf {x}))\mathbf {G}^{\prime } + \mathbf {e}^{\prime }, \end{aligned}$$

where \(\mathbf {G}^{\prime }:= \mathbf {G}_{\mathcal {G}}\mathbf {P}_{[w+1,n]} \in \mathbb {F}_{q^m}^{k \times (n-w)}\) and \(\mathbf {e}^{\prime }:= \mathbf {e}\mathbf {P}_{[w+1,n]}\). By decoding in \(\mathcal {\mathcal {G}}'\), we thus obtain the vector

$$\begin{aligned} \mathbf {m}' = \mathbf {m} + {{\,\mathrm{Tr}\,}}(\alpha \mathbf {x}). \end{aligned}$$

Since the last u positions of the plaintext \(\mathbf {m}\) are zero (i.e., \(m_i=0\) for \(i=k-u+1,\dots ,k\)), we get \(\alpha = \sum _{i=k-u+1}^{k}m_i^{\prime }x_i^*\), where \(\{x_{k-u+1}^*,\dots ,x_k^*\}\) is a dual basis to \(\{x_{k-u+1},\dots ,x_k\}\). As we know \(\alpha \) and \(\mathbf {x}\), we can compute the plaintext \(\mathbf {m}\). \(\square \)

figure c
figure d
figure e

Remark 1

Steps 1 to 3 of Algorithm 5 can be interpreted as an error-erasure decoder of a Gabidulin code. As this observation may have advantages, especially for implementations, we present this connection formally in Appendix 2.

A SageMath v8.8 [47] implementation of the public key encryption version of LIGA can be downloaded from https://bitbucket.org/julianrenner/liga_pke. The purpose of the source code is to clarify the shown algorithms but not to provide a secure and efficient instance. Developing an implementation that offers the latter two properties and can serve for a performance comparison with other schemes is outside of the scope of this paper and is left for future research.

4.3 KEM/DEM version \({{\Pi }}^{\mathrm{PKE}}\) and \({\Pi }^{\mathrm{KEM}}\)

In [23], generic transformations of IND-CPA secure public key encryptions into IND-CCA2 secure KEMs are proposed. In the following, we apply one of the transformations directly to \({{\Pi }}^{\mathrm{PKE}}\) to obtain \({\Pi }^{\mathrm{KEM}}\). Later, in Sect. 5.2, we will prove that \({{\Pi }}^{\mathrm{PKE}}\) fulfills the requirements such that the applied transformation is secure.

Let \(\mathcal {G}\), \(\mathcal {H}\) and \(\mathcal {K}\) be hash functions, where \(\mathcal {G}\ne \mathcal {H}\). In Algorithm 6 and Algorithm 7, we show the encapsulation and decapsulation algorithms of the KEM \({\Pi }^{\mathrm{KEM}}= (\mathsf {KeyGen},\mathsf {Encaps},\mathsf {Decaps})\). The algorithm \(\mathsf {KeyGen}\) remains Algorithm 3.

figure f
figure g

4.4 Complexity

4.4.1 Timing attacks

Resistance against timing attacks is essential in many applications and systems that do not enable a constant-time implementation are thus considered insecure. Due to the fact that Step 4 of Algorithm 4 can be easily implemented in constant time, the proposed encryption algorithm does not reveal any information about secret knowledge through timing attacks. The same holds for the presented decryption algorithm since there exists an efficient constant-time decoding algorithm for Gabidulin codes [10] and all other steps of Algorithm 5 can be realized in constant time as well.

4.4.2 Asymptotically fastest methods

In some scenarios, a constant-time implementation of the system may not be required but we want that the key generation, encryption, and decryption are as fast as possible. The following results were not known when the original FL system was proposed, but have a major impact on its efficiency.

The complexity of key generation and encryption is dominated by the cost of encoding a Gabidulin code (Line 8 of Algorithm 3 and Line 4 of Algorithm 4).Footnote 3 The asymptotically fastest-known algorithms [13, 36, 37] for this require

  • \(O^\sim (n^{\min \{\frac{\omega +1}{2},1.635\}})\) operations in \(\mathbb {F}_{q^m}\) or \(O^\sim (n^{\omega -2}m^2)\) operations in \(\mathbb {F}_q\) in generalFootnote 4 and

  • \(O^\sim (n)\) operations in \(\mathbb {F}_{q^m}\) if the entries of \(\mathbf {g}\) are a normal basis of \(\mathbb {F}_{q^m}/\mathbb {F}_q\),

where \(\omega \) is the matrix multiplication exponent and \(O^\sim \) means that \(\log \) factors are neglected.

The bottleneck of decryption is (error-erasure) decoding of a Gabidulin code (Line 3 of Algorithm 5, see also Appendix 2 below), where the asymptotically fastest algorithm costs

$$\begin{aligned} O^\sim \!\left( n^{\min \{\frac{\omega +1}{2},1.635\}}\right) \end{aligned}$$

operations in \(\mathbb {F}_{q^m}\) [36, 37] or

$$\begin{aligned} O^\sim (n^{\omega -2}m^2) \end{aligned}$$

operations in \(\mathbb {F}_q\) (decoder in [36] with linearized-polynomial operations in [13]).

For small lengths n, the algorithms from [22, 40, 44, 49], which have quadratic complexity over \(\mathbb {F}_{q^m}\) (or cubic complexity over \(\mathbb {F}_q\)), might be faster than the mentioned algorithms due to smaller hidden constants in the O-notation.

5 Difficult problems & semantic security of LIGA

In this section, we introduce problems in the rank metric that are considered to be difficult. Furthermore, we prove that the public-key encryption version of LIGA is IND-CPA secure and the KEM version is IND-CCA2 secure under the assumption that there does not exist probabilistic polynomial-time algorithms that can solve them. A detailed complexity analysis of existing and new algorithms solving the stated problems is given in Sect. 6.

5.1 Difficult problems in the rank metric

LIGA is based on several difficult problems which are stated in this section. Note that the search variants of the problems correspond exactly to retrieving information about the private key from the public key (not necessarily a valid private key as explained in the following) or the plaintext from the ciphertext. The decisional problems are equivalent to distinguishing the public key or the ciphertext from random vectors.

Definition 3

(ResIG-Distribution: Restricted Interleaved Gabidulin Code Distribution)

Input: \(q,m,n,k,w >\lfloor \frac{n-k}{2}\rfloor ,\zeta< \frac{w}{n-k-w}, u<w\).

Choose uniformly at random

  • \(\mathbf {G}\xleftarrow {\$}\mathcal {G}\), where \(\mathcal {G}\) is the set of all generator matrices of [nk] Gabidulin codes over \(\mathbb {F}_{q^m}\)

  • \(\mathbf {M}\xleftarrow {\$}\{\mathbf {X}\in \mathbb {F}_{q^m}^{u\times k} : {{\,\mathrm{rk}\,}}_{q^m}(\mathbf {X}_{[k-u+1,k]}) = u\} \).

  • \(\mathcal {A} \xleftarrow {\$}\{ \text {subspace } \mathcal {U} \subseteq \mathbb {F}_{q^m}^w \, : \, \dim \mathcal {U} = \zeta , \, \mathcal {U} \text { has a basis of full}-\) \(\mathbb {F}_q\) \(-\text { rank elements} \}\)

  • \(\mathbf {E}' \xleftarrow {\$}\left\{ \begin{pmatrix} \mathbf {s}_1' \\ \vdots \\ \mathbf {s}_u' \end{pmatrix} \in \mathbb {F}_{q^m}^{u \times w}\, : \, \langle \mathbf {s}_1',\dots ,\mathbf {s}_u'\rangle _{\mathbb {F}_{q^m}} = \mathcal {A}, \, {{\,\mathrm{rk}\,}}_{q}(\mathbf {s}_i') = w \, \forall \, i \right\} \)

  • \(\mathbf {Q} \xleftarrow {\$}\{ \mathbf {A}\in \mathbb {F}_q^{w \times n} : {{\,\mathrm{rk}\,}}_{q}(\mathbf {A}) = w \}\)

  • \(\mathbf {E}\leftarrow \mathbf {E}' \mathbf {Q}\)

Output: \((\mathbf {G},\mathbf {M}\mathbf {G}+\mathbf {E})\).

Problem 1

(ResIG-Search: Restricted interleaved Gabidulin Code Search Problem)

Input: \((\mathbf {G},\mathbf {Y})\) from ResIG-Distribution with input \(q,m,n,k,w,\zeta ,u\) (Definition 3).

Goal: Find \(\mathbf {M}\in \mathbb {F}_{q^m}^{u\times k}\) and \(\mathbf {E}\in \{\mathbf {X}\in \mathbb {F}_{q^m}^{u \times n} : {{\,\mathrm{rk}\,}}_{\mathbb {F}_q}(\mathbf {X}) \le w\}\) s.t. \(\mathbf {M}\mathbf {G}+\mathbf {E}= \mathbf {Y}\).

Problem 1 (ResIG-Search) is equivalent to decoding a codeword of a u-interleaved Gabidulin code that is corrupted by an error \(\mathbf {E}\), see also Sect. 6.1.2 and is therefore the underlying problem of the structural attacks from Sect. 3.3.

Note however that not necessarily every solution of this problem can be used directly as a valid private key since some additional structure on \(\mathbf {E}\) is introduced in LIGA (i.e., Problem 1 is easier to solve than retrieving a valid private key of LIGA).

Problem 2

(ResIG-Dec: Restricted Interleaved Gabidulin Code Decisional Problem)

Input: \((\mathbf {G},\mathbf {Y}) \in \mathbb {F}_{q^m}^{k\times n} \times \mathbb {F}_{q^m}^{u\times n}\).

Goal: Decide with non-negligible advantage whether \(\mathbf {Y}\) came from ResIG-Distribution with input \(q,m,n,k,w,\zeta ,u\) (Definition 3) or the uniform distribution over \(\mathbb {F}_{q^m}^{u\times n}\).

To solve ResIG-Dec (Problem 2), we do not know a better approach than trying to solve the associated search problem (i.e., ResIG-Search), which is usually done for all decoding-based problems.

Definition 4

(ResErr-Distribution: Restricted Error Distribution)

Input: \(q,m,n,k,w,t_{\mathsf {pub}},u,\varvec{\gamma }, (\mathbf {G},\mathbf {K}) \) from ResIG-Distribution (Definition 3).

Choose uniformly at random

  • \(\mathbf {e}\xleftarrow {\$}\{ \mathbf {x}\in \mathbb {F}_{q^m}^{n}\, : \, {{\,\mathrm{rk}\,}}_q(\mathbf {x}) = t_{\mathsf {pub}}\} \)

  • \(\alpha \xleftarrow {\$}\mathbb {F}_{q^{mu}}\)

  • \(\mathbf {k}\leftarrow {{\,\mathrm{ext}\,}}_{\mathbf {\gamma }}^{-1}(\mathbf {K})\)

  • \(\mathbf {y}\leftarrow {{\,\mathrm{Tr}\,}}(\alpha \mathbf {k}) +\mathbf {e} = {{\,\mathrm{Tr}\,}}(\alpha \mathbf {m}) \mathbf {G}+ {{\,\mathrm{Tr}\,}}(\alpha \mathbf {z}) + \mathbf {e}\)

Output: \(\mathbf {y}\).

Problem 3

(ResG-Search: Restricted Gabidulin Code Search Problem)

Input: \(q,m,n,k,w,t_{\mathsf {pub}},u,\varvec{\gamma }, (\mathbf {G},\mathbf {K}) \) from ResIG-Distribution (Definition 3), \(\mathbf {y}\) from ResErr-Distribution (Definition 4) with input \((\mathbf {G},\mathbf {K})\).

Goal: Find \(\mathbf {m}\in \mathbb {F}_{q^m}^{k}\) and \(\mathbf {e}\in \{\mathbf {x}\in \mathbb {F}_{q^m}^{n} : {{\,\mathrm{rk}\,}}_{\mathbb {F}_q}(\mathbf {x}) \le t_{\mathsf {pub}}\}\) such that \(\mathbf {m}\mathbf {G}+\mathbf {e}= \mathbf {y}\).

Problem 3 is equivalent to decoding a codeword of a Gabidulin code that is corrupted by an error that has with high probability a rank weight of \(> (n-k)/2\), see Appendix 3.

Problem 4

(ResG-Dec: Restricted Gabidulin Code Decisional Problem)

Input: \(q,m,n,k,w,t_{\mathsf {pub}},u,\varvec{\gamma }, (\mathbf {G},\mathbf {K}) \) from ResIG-Distribution (Definition 3), \(\mathbf {y}\in \mathbb {F}_{q^m}^{n}\).

Goal: Decide with non-negligible advantage whether \(\mathbf {y}\) came from ResErr-Distribution with input \(q,m,n,k,w,t_{\mathsf {pub}},u,\varvec{\gamma }, (\mathbf {G},\mathbf {K})\) or the uniform distribution over \( \mathbb {F}_{q^m}^{n}\).

As before, we are not aware of a faster approach to solve ResG-Dec than through the solution of the associated search problem.

We will see in the next subsection that LIGA is IND-CCA2 secure under the assumption that \(\textsf {ResG-Dec}\) is a hard problem. As mentioned above, there is an obvious reduction of \(\textsf {ResG-Dec}\) to \(\textsf {ResG-Search}\), which can again be efficiently reduced to \(\textsf {ResIG-Search}\). In fact, all relevant attacks studied in Sect. 6 make use of this chain of reduction and aim at solving one of the two search problems.

We are not aware of a reduction of \(\textsf {ResIG-Dec}\) to \(\textsf {ResIG-Search}\) or one of the other problems. Hence, it might very well be that \(\textsf {ResIG-Dec}\) is significantly easier than the other problems. In Sect. 6.3, we show that there is a distinguisher for \(\textsf {ResIG-Dec}\) that is efficiently computable if the system parameter \(\zeta \) is chosen too small. Due to the missing reduction, it is not clear whether or not this distinguisher influences the security of the system.

5.2 Semantic security

In this section, we prove that the public key encryption system \({{\Pi }}^{\mathrm{PKE}}\) is semantically secure against chosen plaintext attacks in the standard model under the assumption that ResG-Dec (Problem 4) is difficult. In addition, we show that the IND-CCA2 security of \({\Pi }^{\mathrm{KEM}}\) reduces tightly to the IND-CPA security of \({{\Pi }}^{\mathrm{PKE}}\) in the random oracle model.

5.2.1 IND-CPA security of \({{\Pi }}^{\mathrm{PKE}}\)

To show that \({{\Pi }}^{\mathrm{PKE}}\) is secure against chosen plaintext attacks, we use the definition of admissibility as in [33].

Definition 5

(Admissibility [33]) The public key encryption scheme \({{\Pi }}^{\mathrm{PKE}}= (\mathsf {KeyGen},\mathsf {Encrypt},\mathsf {Decrypt})\) with a message space \(\mathcal {M}\) and a random space \(\mathcal {R}\) is called admissible if there is a pair of deterministic polynomial-time algorithms \(\mathsf {Encrypt}_1\) and \(\mathsf {Encrypt}_2\) satisfying the following property:

  • Partible: \(\mathsf {Encrypt}_1\) takes as input a public key \(\mathsf {pk}\) and \(r \in \mathcal {R}\), and outputs a \(p(\lambda )\) bit-string, where \(\lambda \) is the security parameter. \(\mathsf {Encrypt}2\) takes as input a key \(\mathsf {pk}\), and \(\mathbf {m}\in \mathcal {M}\) and outputs a \(p(\lambda )\) bit-string. Here p is some polynomial in the security parameter \(\lambda \). Then for any \(\mathsf {pk}\) given by \(\mathsf {KeyGen}\), \(r \in \mathcal {R}\), and \(\mathbf {m}\in \mathcal {M}\), \(\mathsf {Encrypt}_1( \mathsf {pk}, r ) \oplus \mathsf {Encrypt}_2 ( \mathsf {pk}, \mathbf {m}) = \mathsf {Encrypt}( \mathsf {pk}, \mathbf {m}; r ) \).

  • Pseudorandomness: Let D be a probabilistic algorithm and let

    $$\begin{aligned} \mathsf {Adv}_{D,\mathsf {Encrypt}_1}^{ind}(\lambda ) =&\Pr \Big [ D(\mathsf {pk},\mathsf {Encrypt}_1(\mathsf {pk},r)) = 1 |r \xleftarrow {\$}\mathcal {R}, (\mathsf {sk}, \mathsf {pk}) \leftarrow \mathsf {KeyGen}\big (1^{\lambda }\big ) \Big ] \\&- \Pr \big [ D(\mathsf {pk},s) = 1 |s \xleftarrow {\$}\mathcal {U}_{p(\lambda )}, (\mathsf {sk}, \mathsf {pk}) \leftarrow \mathsf {KeyGen}\big (1^{\lambda } \big ) \big ]. \end{aligned}$$

    We define the advantage function of the problem as follows. For any t,

    $$\begin{aligned} \mathsf {Adv}_{\mathsf {Encrypt}_1}^{ind}(\lambda ,t) = \max _{D} \big \{\mathsf {Adv}_{D,\mathsf {Encrypt}_1}^{ind}(\lambda ) \big \}, \end{aligned}$$

    where the maximum is taken over all D with time-complexity t. Then, the function \(\mathsf {Adv}_{\mathsf {Encrypt}_1}^{ind}(\lambda ,t)\) is negligible for every polynomial bounded t and every sufficiently large \(\lambda \).

In the following we will prove that \({{\Pi }}^{\mathrm{PKE}}\) is IND-CPA secure by showing that is fulfills the definition of admissibility.

Theorem 6

The system \({{\Pi }}^{\mathrm{PKE}}= (\mathsf {KeyGen},\mathsf {Encrypt}, \mathsf {Decrypt})\) is an IND-CPA secure encryption scheme in the standard model under the assumption that the ResG-Dec problem is difficult.


Let \(\mathsf {Encrypt}_1 := {{\,\mathrm{Tr}\,}}(\alpha \mathbf {k}_{\mathsf {pub}}) +\mathbf {e}\) and \(\mathsf {Encrypt}_2 := \mathbf {m}\mathbf {G}_\mathcal {G}\). Then, one observes that \(\mathsf {Encrypt}= \mathsf {Encrypt}_1 \oplus \mathsf {Encrypt}_2\) and thus \({{\Pi }}^{\mathrm{PKE}}\) is partible. Since ResG-Dec (Problem 4) is assumed to be difficult, the encryption scheme fulfills pseudorandomness and thus, the system is admissibile. As proven in [33, Lemma 1], if \({{\Pi }}^{\mathrm{PKE}}\) fulfills Definition 5, then it is an IND-CPA secure encryption scheme. \(\square \)

5.2.2 IND-CCA2 security of \({\Pi }^{\mathrm{KEM}}\)

We used a transformation proposed in [23] to transform the public key encryption scheme \({{\Pi }}^{\mathrm{PKE}}\) into the KEM \({\Pi }^{\mathrm{KEM}}\). In the following, we prove that \({\Pi }^{\mathrm{KEM}}\) is IND-CCA2 secure.

The applied transformation requires that the encryption scheme is \(\gamma \)-spread which is proven to be the case for \({{\Pi }}^{\mathrm{PKE}}\) in the following.

Definition 6

(\(\gamma \)-spread, [17, 23]) For valid \((\mathsf {pk},\mathsf {sk})\), the min-entropy of \(\mathsf {Encrypt}(\mathbf {m},\mathsf {pk})\) is defined by

$$\begin{aligned} \gamma (\mathbf {m},\mathsf {pk}) := - \log _2 \max _{\mathbf {c}\in \hat{\mathcal {C}}} \Pr _{r\leftarrow \mathcal {R}} [\mathbf {c}= \mathsf {Encrypt}(\mathbf {m},\mathsf {pk};r)], \end{aligned}$$

where \(\hat{\mathcal {C}}\) is the set of possible ciphertexts. A public key encryption scheme is called \(\gamma \)-spread if for every valid key pair \((\mathsf {pk},\mathsf {sk})\) and every message \(\mathbf {m}\in \mathcal {M}\), \(\gamma (\mathbf {m},\mathsf {pk}) \ge \gamma \). It follows that for all \(\mathbf {c}\in \hat{\mathcal {C}}\),

$$\begin{aligned} \Pr _{r\leftarrow \mathcal {R}} [\mathbf {c}= \mathsf {Encrypt}(\mathbf {m},\mathsf {pk};r)] \le 2^{- \gamma }. \end{aligned}$$

Lemma 2

The public key encryption system \({{\Pi }}^{\mathrm{PKE}}\) is \(\gamma \)-spread, where \(\gamma = m(t_{\mathsf {pub}}-u)+t_{\mathsf {pub}}(n-t_{\mathsf {pub}}-1)\).


We observe that

$$\begin{aligned} \max _{\mathbf {c}\in \hat{\mathcal {C}}}\Pr _{r\leftarrow \mathcal {R}} [\mathbf {c}= \mathsf {Encrypt}(\mathbf {m},\mathsf {pk};r)] {=}&\max _{\mathbf {c}\in \hat{\mathcal {C}}} \Pr _{r\leftarrow \mathcal {R}} [\mathbf {c}= (\mathbf {m},\mathbf {0}_u)\mathbf{G}_{\mathcal {G}} + {{\,\mathrm{Tr}\,}}(\alpha \mathbf {k}_{\mathsf {pub}}) +\mathbf {e}] \\ \overset{\text {(i)}}{\le }&\max _{\mathbf {c}' \in \hat{\mathcal {C}}'}q^{mu}\Pr _{r\leftarrow \mathcal {R}} [\mathbf {c}' = (\mathbf {m},\mathbf {0}_u)\mathbf{G}_{\mathcal {G}} +\mathbf {e}] \\ {=}&~ q^{mu} \frac{1}{|\{\mathbf {e}\in \mathbb {F}_{q^m}^{n} : {{\,\mathrm{rk}\,}}_{q}(\mathbf {e}) = t_{\mathsf {pub}}\}|}, \end{aligned}$$

where \(\hat{\mathcal {C}}'\) is the set of all vectors in rank distance \(t_{\mathsf {pub}}\) from \((\mathbf {m},\mathbf {0}_u)\mathbf{G}_{\mathcal {G}}\) and \(\text {(i)}\) follows from the fact that there at most \(q^{mu}\) choices for \(\alpha \). In [19, Section IV.B], a constructive way of obtaining rank-\(t_{\mathsf {pub}}\) matrices is given. More precisely, an injective mapping \( \varphi \, : \, \mathbb {F}_q^{t(n+m-t-1)} \rightarrow \{\mathbf {A}\in \mathbb {F}_q^{n \times m} \, : \, {{\,\mathrm{rk}\,}}\mathbf {A}= t\}\) is given. Hence, we have \(|\{\mathbf {e}\in \mathbb {F}_{q^m}^{n} : {{\,\mathrm{rk}\,}}_{q}(\mathbf {e}) = t_{\mathsf {pub}}\}| \ge q^{t(n+m-t-1)}\). It follows that

$$\begin{aligned} \frac{q^{mu}}{ |\{\mathbf {e}\in \mathbb {F}_{q^m}^{n} : {{\,\mathrm{rk}\,}}_{q}(\mathbf {e}) = t_{\mathsf {pub}}\}|}&\le \frac{q^{mu}}{ q^{t_{\mathsf {pub}}(n+m-t_{\mathsf {pub}}-1)} } \\&= q^{-m(t_{\mathsf {pub}}-u)-t_{\mathsf {pub}}(n-t_{\mathsf {pub}}-1)} \\&\le q^{-\gamma }. \end{aligned}$$

\(\square \)

Theorem 7

The KEM-DEM \({\Pi }^{\mathrm{KEM}}= (\mathsf {KeyGen},\mathsf {Encaps}, \mathsf {Decaps})\) is IND-CCA2 secure in the random oracle model under the assumption that the ResG-Dec problem is difficult.


Assuming the ResG-Dec is difficult, the encryption \({{\Pi }}^{\mathrm{PKE}}\) is IND-CPA secure, see Theorem 6. Further, it is proven in Lemma 2 that \({{\Pi }}^{\mathrm{PKE}}\) has \(\gamma \)-spread encryptions. Thus, the system \({\Pi }^{\mathrm{KEM}}\) can be tightly reduced to \({\Pi }^{\mathrm{KEM}}\) in the random oracle model as shown in [23]. \(\square \)

6 Security analysis of LIGA

In this section, we analyze the security of LIGA. As proven in Theorem 6 and 7, the encryption version is IND-CPA secure and the KEM version is IND-CCA2 secure under the assumption that ResG-Dec is difficult. Since there are obvious reductions from ResG-Dec to ResG-Search and from ResG-Dec to ResIG-Search, we will study the hardness of these two search problems in this section (Sect. 6.1 for ResIG-Search and Sect. 6.2 for ResG-Search). In fact, we are not aware of a more efficient method to solve ResG-Dec than through these two search problems.

Although no formal reduction from any of the other three studied problems to ResIG-Dec is known, we study also the hardness of ResIG-Dec (Sect. 6.3). We derive a distinguisher for the public key with exponential complexity in the system parameters, which can be avoided by proper parameter choice.

Due to the nature of the (random) encryption, there are public keys for which the probability that the work factor of some of the ciphertext attacks on ResG-Search (ciphertext attack) is below the designed minimal work factor is not negligible (i.e., \(> 2^{-\lambda }\)). We show in Sect. 6.4 that these weak keys occur with negligible probability (i.e., \(\le 2^{-\lambda }\)) during the random key generation if the parameters are chosen in a suitable way.

6.1 Exponential-time attacks on ResIG-Search

We propose new and summarize known methods that solve ResIG-Search (Problem 1). All studied algorithms have exponential complexity in the code parameters.

Recall that in the decryption algorithm of LIGA, the last u positions of the private key \(\mathbf {x}\) have to be a basis of \(\mathbb {F}_{q^{mu}}\) over \(\mathbb {F}_{q^m}\). Therefore, not every solution of Problem 1 (ResIG-Search) can be used as valid private key and Problem 1 is a strictly easier problem than retrieving a valid private key corresponding to a given public key.

6.1.1 Brute-force the vector \(\mathbf {z}\) attack

The number of vectors \(\mathbf {z}\in \mathbb {F}_{q^{mu}}^n\) that fulfill the conditions stated in Sect. 4.1 is equal to number of possible vectors \(\mathbf {s}\in \mathbb {F}_{q^{mu}}^w\) times the number of full rank matrices in \(\mathbb {F}_{q^m}^{w\times n}\) in reduced row Echelon form. Formally, the number of vectors \(\mathbf {z}\) is

$$\begin{aligned} \underbrace{\left| \left\{ \mathbf {z}\, : \, \mathbf {z}\text { can occur in Alg.~3}\right\} \right| }_{\ge 1} \cdot \underbrace{\left| \left\{ \mathbf {P}\, : \, \mathbf {P}\text { can occur in Alg.~3}\right\} \right| }_{\ge \genfrac[]{0.0pt}{}{n}{w}_{q}} \ge \genfrac[]{0.0pt}{}{n}{w}_{q}. \end{aligned}$$

Thus, brute-forcing a vector \(\mathbf {z}\) that is a solution to ResIG-Search has work factor

$$\begin{aligned} \text {WF}_{{\mathbf {z}}}\ge \frac{\genfrac[]{0.0pt}{}{n}{w}_{q}}{\mathcal {N}} \ge \frac{q^{w(n-w)}}{\mathcal {N}}, \end{aligned}$$

where the latter inequality follows from a lower bound on q-binomials (see [24, Lemma 4]), and

$$\begin{aligned} \mathcal {N} = \max \left\{ |\mathcal {C}| \cdot \frac{\sum _{i=0}^{w-1}{\prod _{j=0}^{i-1}{(q^{mu} - q^j)} {\displaystyle {n \brack i}_q }} }{q^{mun}} , \ 1\right\} \end{aligned}$$

is the average number of interleaved codewords in a ball of radius w around a uniformly at random chosen interleaved received word.

6.1.2 Interleaved decoding attack

As described in Sect. 3.3, an attacker can apply an interleaved decoder on \(\mathbf {k}_{\mathsf {pub}}\) to retrieve an alternative private key. A major ingredient of LIGA is that the public key is chosen in a way that this decoding will always fail (i.e., the corresponding linear system of equations does not have a unique solution). However, it is still possible to brute-force search in the solution space of the involved system of equations. This is analyzed in the following. Notice thereby that any interleaved codeword in radius at most w is a solution to ResIG-Search.

Problem 1 (ResIG-Search) is equivalent to decoding a codeword of a u-interleaved Gabidulin code that is corrupted by an error \(\mathbf {E}\). The error \(\mathbf {E}\) fulfills

$$\begin{aligned} \bigg \lfloor \frac{n-k}{2} \bigg \rfloor< {{\,\mathrm{rk}\,}}_{q}(\mathbf {E}) \le \frac{u}{u+1} (n-k) \quad \text {and} \quad {{\,\mathrm{rk}\,}}_{q^m}(\mathbf {E})< \frac{w}{n-k-w} < w \end{aligned}$$

and thus, no known algorithm is able to correct it efficiently.

The crucial point of the interleaved decoding algorithms from [27, 43] is solving a linear system of equations based on the syndromes with \(w+1\) unknowns and \(\varphi \) linearly independent equations which is equivalent to finding the kernel of the matrix in (5), cf. [49, Section 4.1]. For \(\zeta \ge \frac{w}{n-k-w}\), the dimension of the solution space is one and all solutions are valid for the remaining decoding steps. For \(\zeta < \frac{w}{n-k-w}\), the dimension of the solution space is \(w+1-\varphi \) but each valid solution forms only a one-dimensional subspace. An attacker can therefore search in the solution space for a valid solution which requires on average

$$\begin{aligned} \frac{(q^m)^{w+1-\varphi }}{q^m \cdot \mathcal {N}} = \frac{q^{m(w-\varphi )}}{\mathcal {N}} \end{aligned}$$

trials, where \(\mathcal {N}\) is the average number of interleaved codewords, see (9).

The size of the solution space is \(w+1-\varphi \) and clearly maximized for the smallest-possible value of \(\varphi \), i.e., \(\varphi = n-k-w\). In this case, the search through the solution space has work factor

$$\begin{aligned} \text {WF}_{\text {ILD}}=\frac{q^{m(w-\zeta (n-k-w))}}{\mathcal {N}}. \end{aligned}$$

Since the size of the solution space is maximal for \(\varphi = n-k-w\), the repair from Sect. 4.1 with the explicit parameter value \(\zeta =1\) (i.e., \(\dim \big ( \langle \mathbf {z}_1 , \dots , \mathbf {z}_u\rangle _{\mathbb {F}_{q^m}}\big ) = 1\)) is the most secure choice in this sense. However, we keep the choice of \(\zeta \) flexible as the pair-wise linear dependence of the \(\mathbf {z}_i\) might decrease the security (we are however not aware of how this fact could be used).

Besides the syndrome-based interleaved decoding algorithms in [27, 43], and [49, p. 64], there is an interpolation-based decoding algorithm [49, Section 4.3 (page 72)]. This interpolation-based algorithm can be interpreted both as a list decoder of interleaved Gabidulin codes with exponential worst-case and average list size or as a probabilistic unique decoder. The probabilistic unique interpolation-based decoder fails if and only if the decoding algorithms in [27, 43, 49, p. 64] fail and therefore the previous analysis applies here as well. For the list decoder, cf. [49, Lemma 4.5], the work factor of the resulting attack is

$$\begin{aligned} \text {WF}_\text {list, public key} \le \frac{q^{m(u-1)k}}{\mathcal {N}}. \end{aligned}$$

Notice that the list of size \(q^{m(u-1)k}\) contains many words which are no valid codewords, but we have to go through the whole list to find all valid codewords in radius up to w.

6.1.3 List decoding of the public key attack

Recall that \(\mathbf {k}_{\mathsf {pub}}= \mathbf {x} \cdot \mathbf{G}_{\mathcal {G}} + \mathbf {z}\). Previously, we have explained why this vector is a corrupted version of a codeword of a u-interleaved Gabidulin code. At the same time, \(\mathbf {x} \cdot \mathbf{G}_{\mathcal {G}}\) can be seen as a short Gabidulin code over a large field \(\mathbb {F}_{q^{mu}}\) and therefore, if existing, one could apply list decoding algorithms to decode \(\mathbf {k}_{\mathsf {pub}}\) and obtain \(\mathbf {x}\). The weight of the error \(\mathbf {z}\) is larger than the unique decoding radius and therefore a unique decoder cannot be applied to reconstruct \(\mathbf {x}\) and a list decoder for radius w is required.

However, such an algorithm has not been found yet. It was even shown in [38, 46, 48] that for most classes of Gabidulin codes such a polynomial-time list decoding algorithm cannot exist. Note that these results were not known when the original FL cryptosystem was proposed. These results also imply that there is no polynomial-time list decoding algorithm for arbitrary Gabidulin codes beyond the unique decoding radius (such as the Guruswami–Sudan algorithm for Reed–Solomon codes).

6.1.4 Randomized Gabidulin decoding attack on the public key

The public key can be seen as the sum of a Gabidulin codeword over the field \(\mathbb {F}_{q^{mu}}\) and an error of weight \(w > \frac{n-k}{2}\). Alternatively, as shown in Sect. 3.2, the public key can be seen as an interleaved Gabidulin codeword that is corrupted by an error of weight w (note that this is the reason why all the \(\mathbf {s}_i\)’s must have full \(\mathbb {F}_q\)-rank in Algorithm 3). Each row of (3) is a codeword of a Gabidulin code over \(\mathbb {F}_{q^m}\) that is corrupted by an error of rank weight w. Both the corrupted Gabidulin codeword over \(\mathbb {F}_{q^{mu}}\) as well as over \(\mathbb {F}_{q^m}\) can be decoded using the randomized decoding approach proposed in [39]. Since applying the attack on each row of the unfolded public key is more efficient, we conclude that the randomized Gabidulin decoding attack on the public key has an average complexity of

$$\begin{aligned} \text {WF}_{\text {RGD}} = \frac{n}{64}\cdot q^{m(n-k)-w(n+m)+w^2+\min \{2\xi (\frac{n+k}{2}-\xi ),wk\} } \end{aligned}$$

over \(\mathbb {F}_{q^m}\).

6.1.5 Moving to another close error attack

The following attack was suggested by Rosenkilde [41]. It tries to move the vector \(\mathbf {z}\) (which we have chosen such that the interleaved decoder fails) to a close vector of the same or smaller rank weight w for which the interleaved decoder for \(\mathbf {k}_{\mathsf {pub}}\) does not fail.

The idea is to find a vector \(\mathbf {y} \in \mathbb {F}_{q^m}^{u \times n}\) such that \(\mathbf {z}^\prime := \mathbf {z} + \mathbf {y}\) still has rank weight \({{\,\mathrm{rk}\,}}_q(\mathbf {z}^\prime ) \le w\) and that the rank of the matrix from (5) over \(\mathbb {F}_{q^m}\) is at least w. To guarantee the first condition, we want to construct \(\mathbf {y}\) such that its extended \(um \times n\) matrix over \(\mathbb {F}_q\) has a row space \(\mathcal {R} := \mathrm {RowSpace}_{\mathbb {F}_q}\big ({{\,\mathrm{ext}\,}}_{\varvec{\gamma }}(\mathbf {y})\big )\) that is contained in the one of \(\mathbf {z}\). Since for the original error \(\mathbf {z}\), the matrix (5) has rank \(\varphi \le \zeta (n-k-w)\), \(\mathcal {R}\) must have at least \(\mathbb {F}_q\)-dimension \(w-\varphi \ge w(\zeta +1)-\zeta (n-k)\). By choosing a random \(\mathcal {R}\) with this property and taking a random matrix \(\mathbf {y}\) whose extended matrix has \(\mathbb {F}_q\)-row space \(\mathrm {RowSpace}_{\mathbb {F}_q}\big ({{\,\mathrm{ext}\,}}_{\varvec{\gamma }}(\mathbf {y})\big ) = \mathcal {R}\), the second condition is fulfilled with high probability.

The complexity of the attack is hence dominated by the complexity of finding a subspace \(\mathcal {R} \subseteq \mathbb {F}_q^{n}\) of dimension \(w-\varphi \) that is contained in the w-dimensional \(\mathbb {F}_q\)-row space of \(\mathbf {z}\). Since this is unknown, we can find it in a Las-Vegas fashion by repeatedly drawing a subspace uniformly at random. The expected number of iterations until we find a suitable row space is thus one over the probability that a random \((w-\varphi )\)-dimensional subspace of \(\mathbb {F}_q^n\) is contained in a given w-dimensional subspace, which is (cf. [15, Proof of Lemma 7]):

$$\begin{aligned} \frac{\genfrac[]{0.0pt}{}{w}{w-\varphi }_q}{\genfrac[]{0.0pt}{}{n}{w-\varphi }_q} \approx \frac{q^{\varphi (w-\varphi )}}{q^{(n-w+\varphi )(w-\varphi )}} = q^{-(n-w)(w-\varphi )} \le q^{-(n-w)(w(\zeta +1)-\zeta (n-k))}. \end{aligned}$$

Hence, the attack has work factor

$$\begin{aligned} \text {WF}_{\text {MCE}}&= q^{(n-w)(w-\varphi )} \ge q^{(n-w)(w(\zeta +1)-\zeta (n-k))}. \end{aligned}$$

6.2 Exponential-time attacks on ResG-Search

Retrieving information about the plaintext from the ciphertext and the public key is equal to solving ResG-Search (Problem 4). In this section, methods that solve this problem are summarized.

6.2.1 Randomized Gabidulin decoding attack on the ciphertext

Each ciphertext of LIGA can be seen as a Gabidulin codeword over \(\mathbb {F}_{q^m}\) plus an error:

$$\begin{aligned} \mathbf {c}&= (\mathbf {m},\mathbf {0}_u) \cdot \mathbf{G}_{\mathcal {G}} + {{\,\mathrm{Tr}\,}}(\alpha \mathbf {k}_{\mathsf {pub}}) + \mathbf {e} \\&= \underbrace{\left[ (\mathbf {m},\mathbf {0}_u) + {{\,\mathrm{Tr}\,}}(\alpha \mathbf {x})\right] \cdot \mathbf{G}_{\mathcal {G}}}_{\text {codeword}} + \underbrace{{{\,\mathrm{Tr}\,}}(\alpha \mathbf {z}) + \mathbf {e}}_{\text {error}} \end{aligned}$$

Denote \(\tilde{w} := {{\,\mathrm{rk}\,}}_{\mathbb {F}_q}({{\,\mathrm{Tr}\,}}(\alpha \mathbf {z}) + \mathbf {e})\). Then we can use the decoding algorithm proposed in [39], which requires on average at least

$$\begin{aligned} \frac{n}{64}\cdot q^{m(n-k)-\tilde{w}(n+m)+\tilde{w}^2+\min \{2\xi (\frac{n+k}{2}-\xi ),\tilde{w}k\} } \end{aligned}$$

operations in \(\mathbb {F}_{q^m}\).

Clearly, the complexity of the algorithm strongly depends on the value \(\tilde{w}\), which in turn depends on the generated keys. In general, \(\tilde{w} = w +t_{\mathsf {pub}}\), but for some choices of \(\mathbf {z}\), \(\alpha \), and \(\mathbf {e}\), the rank \(\tilde{w}\) is smaller. For this issue, see Sect. 6.4 and Appendix 3, where we study the probability that \(\tilde{w}\) is small, both for randomness in the encryption (random choice of \(\alpha \) and \(\mathbf {e}\)) and the key generation (random choice of \(\mathbf {z}\)). Some extremely rarely occurring keys (weak keys) thereby result in relatively high probabilities that \(\tilde{w}\) is small.

However, we can choose the system parameters such that both, the probability of a weak key as well as the conditional probability that \(\tilde{w}<w\) given a non-weak key is below \(2^{-\lambda }\). Hence, with overwhelming probability, a random key and ciphertext result in a ciphertext error of rank weight \(\tilde{w}\ge w\) and the work factor of this attack is always at least as large as the “Randomized Gabidulin Decoding Attack on the Public Key” in Sect. 6.1.

6.2.2 List decoding of the ciphertext attack

As described in the Randomized Gabidulin Decoding Attack on the Ciphertext above, the ciphertext of LIGA is a codeword of a Gabidulin code, corrupted by an error of rank weight \(\tilde{w}\), Hence, an attacker can try to decode the ciphertext directly. Since \(\tau \) is always greater than the unique decoding radius \(\left\lfloor \frac{n-k}{2}\right\rfloor \) of the Gabidulin code, this would require the existence of an efficient (list) decoding algorithm up to radius \(\tau \). As explained previously, there is no such algorithm and bounds on the list size prove that there cannot exist a generic list decoding algorithm for all Gabidulin codes which indicates that list decoding is a hard problem.

However, to be secure, we have considered list decoding as follows for the security level of our system. The list size \(\mathcal {L}_\mathbf {c},\text {worst}\) denotes a lower bound on the worst-case work factor of list decoding. For example, for a Gabidulin code with parameters \(n|m\) and \(\hbox {gcd}(n,n-\tau )\ge 2\), there is a received word such that there are at least

$$\begin{aligned} \mathcal {L}_\mathbf {c},\text {worst} \ge \max \left\{ \frac{\genfrac[]{0.0pt}{}{n/g}{(n-\tau )/g}_{q^g}}{q^{n (\tau /g-1)}} \, : \, g \ge 2, \, g |\hbox {gcd}(n,n-\tau ) \right\} \end{aligned}$$

codewords in rank distance at most \(\tau \) to it.

Although \(\mathcal {L}_\mathbf {c},\text {worst}\) does not imply any statement about the average list size/average work factor, it provides an estimate of the order of magnitude of the work factor of a hypothetical list decoding attack. For our suggested parameters, we have ensured that the value of \(\mathcal {L}_\mathbf {c},\text {worst}\) is sufficiently large in the proposed sets of parameters in Sect. 7.

6.2.3 Combinatorial rank syndrome decoding (RSD) attack

The ciphertext can be interpreted as a codeword from a code of dimension k (see [16]), generated by the generator matrix

$$\begin{aligned} \begin{pmatrix} \mathcal {M}_{k-u,q}\left( \mathbf {g} \right) \\ {{\,\mathrm{Tr}\,}}(\gamma _1 \mathbf {k}_{\mathsf {pub}})\\ \vdots \\ {{\,\mathrm{Tr}\,}}(\gamma _u \mathbf {k}_{\mathsf {pub}})\\ \end{pmatrix}. \end{aligned}$$

Since the structure of this code only permits decoding like a random rank-metric code, it can be decoded with the combinatorial syndrome decoding attack from [4] whose complexity is in the order of

$$\begin{aligned} \text {WF}_{\text {CRSD}}=(n-k)^3m^3q^{t_{\mathsf {pub}}\lceil \frac{(k+1)m}{n} \rceil -m}. \end{aligned}$$

6.2.4 Algebraic RSD attack

As described in the previous section, the ResG-Search problem can be solved by decoding an error of rank weight \(t_{\mathsf {pub}}\) in a random [nk] code. Beside the combinatorial approach, there exist algebraic algorithms to solve the Problem.

In [6], the RSD problem is expressed as a multivariate polynomial system and is solved by computing a Gröbner basis. The complexity of that attack is generally smaller than the combinatorial approach. In case there is a unique solution to the system, then the work factor of the algorithm is

$$\begin{aligned} \text {WF}_{\text {Gr}} = {\left\{ \begin{array}{ll} \left[ \frac{((m+n)t_{\mathsf {pub}})^{t_{\mathsf {pub}}}}{t_{\mathsf {pub}}!} \right] ^\mu &{} \text {if }\quad m{n-k-1 \atopwithdelims ()t_{\mathsf {pub}}} \le {n \atopwithdelims ()t_{\mathsf {pub}}} \\ \left[ \frac{((m+n)t_{\mathsf {pub}})^{t_{\mathsf {pub}}+1}}{(t_{\mathsf {pub}}+1)!} \right] ^\mu &{} \text {otherwise, } \end{array}\right. } \end{aligned}$$

where \(\mu \) is the exponent in the complexity expression of the used matrix multiplication algorithm. The asymptotically fastest known matrix multiplication algorithm has an exponent of \(\mu < 2.373\). As the authors of [6], we use \(\mu = 2.807\) to compute the work factors in this paper since it corresponds to Strassen’s algorithm, which is (due to the hidden constant) in practice the fastest algorithm for large (but still reasonably small) matrix sizes.

Very recently, a new algebraic algorithm was proposed to solve the RSD problem [7]. It divides the RSD problem instances into two categories. If

$$\begin{aligned} m\left( {\begin{array}{c}n-k-1\\ t_{\mathsf {pub}}\end{array}}\right) \ge \left( {\begin{array}{c}n\\ t_{\mathsf {pub}}\end{array}}\right) -1, \end{aligned}$$

we are in the overdetermined case and the proposed algorithm has work factor

$$\begin{aligned} \text {WF}_{\text {Wogr}} = m \left( {\begin{array}{c}n-p-k-1\\ t_{\mathsf {pub}}\end{array}}\right) \left( {\begin{array}{c}n-p\\ t_{\mathsf {pub}}\end{array}}\right) ^{\mu -1} \end{aligned}$$

in \(\mathbb {F}_q\), where \(p= \min \{i: i \in \{1,\hdots ,n\}, m \left( {\begin{array}{c}n-i-k-1\\ t_{\mathsf {pub}}\end{array}}\right) \ge \left( {\begin{array}{c}n-i\\ t_{\mathsf {pub}}\end{array}}\right) -1\}\). Otherwise, we are in the underdetermined case in which the algorithm has work factor

$$\begin{aligned} \text {WF}_{\text {Wogr}} = \min \{ \text {WF}_{\text {Under}}, \text {WF}_{\text {Hybrid}} \}. \end{aligned}$$

We have

$$\begin{aligned} \text {WF}_{\text {Hybrid}} = q^{a t_{\mathsf {pub}}} m \left( {\begin{array}{c}n-k-1\\ t_{\mathsf {pub}}\end{array}}\right) \left( {\begin{array}{c}n-a\\ t_{\mathsf {pub}}\end{array}}\right) ^{\mu -1} \end{aligned}$$

with \(a = \min \{i: i \in \{1,\hdots ,n\}, m \left( {\begin{array}{c}n-k-1\\ t_{\mathsf {pub}}\end{array}}\right) \ge \left( {\begin{array}{c}n-i\\ t_{\mathsf {pub}}\end{array}}\right) \ - 1\}\). Further, for \(0<b<t_{\mathsf {pub}}+2\) and \(A_b -1 \le B_b + C_b\),

$$\begin{aligned} \text {WF}_{\text {Under}} = \frac{B_b \left( {\begin{array}{c}k+t_{\mathsf {pub}}+1\\ t_{\mathsf {pub}}\end{array}}\right) + C_b(mk+1)(t_{\mathsf {pub}}+1)}{B_b+C_b}\Bigg ( \sum _{j=1}^{b} \left( {\begin{array}{c}n\\ t_{\mathsf {pub}}\end{array}}\right) \left( {\begin{array}{c}mk+1\\ j\end{array}}\right) \Bigg )^2, \end{aligned}$$

where \(A_b:=\sum _{j=1}^{b}\left( {\begin{array}{c}n\\ t_{\mathsf {pub}}\end{array}}\right) \left( {\begin{array}{c}mk+1\\ j\end{array}}\right) \), \(B_b:=\sum _{j=1}^{b} m\left( {\begin{array}{c}n-k-1\\ t_{\mathsf {pub}}\end{array}}\right) \left( {\begin{array}{c}mk+1\\ j\end{array}}\right) \) and

$$\begin{aligned} C_b := \sum _{j=1}^{b} \sum _{i=1}^{j} \bigg ( (-1)^{i+1} \left( {\begin{array}{c}n\\ t_{\mathsf {pub}}+i\end{array}}\right) \left( {\begin{array}{c}m+i-1\\ i\end{array}}\right) \left( {\begin{array}{c}mk+1\\ j-i\end{array}}\right) \bigg ). \end{aligned}$$

We denote the minimum of the work factors of the two algorithms as the work factor of the algebraic RSD attack, i.e.,

$$\begin{aligned} \text {WF}_{\text {ARSD}} = \min \{ \text {WF}_{\text {Gr}}, \text {WF}_{\text {Wogr}} \}. \end{aligned}$$

Note that for algebraic decoding, it is neither known how to improve the complexity by using the fact that there are multiple solutions, nor it is known how to speed up the algorithm in the quantum world.

6.2.5 Linearization attack

In [16], a message attack was proposed which succeeds for some parameters with high probability in polynomial time.

Lemma 3

(Linearization Attack [16]) Let \(\mathbf {k}_{\mathsf {pub}}^{(i)} = {{\,\mathrm{Tr}\,}}(\gamma _i \mathbf {k}_{\mathsf {pub}})\) for \(i=1,\dots ,u\) and

$$\begin{aligned} \mathbf {M}= \begin{pmatrix} \mathcal {M}_{t_{\mathsf {pub}}+1,q}\left( \mathbf {c} \right) \\ -\mathcal {M}_{t_{\mathsf {pub}}+1,q}\left( \mathbf {k}_{\mathsf {pub}}^{(1)} \right) \\ \vdots \\ -\mathcal {M}_{t_{\mathsf {pub}}+1,q}\left( \mathbf {k}_{\mathsf {pub}}^{(u)} \right) \\ -\mathcal {M}_{k+t_{\mathsf {pub}}-u,q}\left( \mathbf {g} \right) \end{pmatrix}. \end{aligned}$$

Then, the encrypted message \(\mathbf {m}\) can be efficiently recovered if the left kernel of \(\mathbf {M}\) has dimension \(\dim (\ker (\mathbf {M})) = 1\).

If \((u+2)t_{\mathsf {pub}}+ k > n\), then \(\mathbf {M}\) has at least two more rows than columns and we have \(\dim (\ker (\mathbf {M}))>1\). If \(\mathbf {k}_{\mathsf {pub}}\) is random and \((u+2)t_{\mathsf {pub}}+ k \le n\), the attack is efficient with high probability [16].

Lemma 4

Let \(\mathbf {M}\) be as in (13). Then,

$$\begin{aligned} {{\,\mathrm{rk}\,}}_{q^m}(\mathbf {M}) \le \min \{\varphi + k + 2t_{\mathsf {pub}}-u,n \}. \end{aligned}$$


We can write

$$\begin{aligned} \mathbf {k}_{\mathsf {pub}}^{(i)}&= {{\,\mathrm{Tr}\,}}(\gamma _i \mathbf {k}_{\mathsf {pub}}) \\&= {{\,\mathrm{Tr}\,}}(\gamma _i \mathbf {x}) \cdot \mathcal {M}_{k,q}\left( \mathbf {g} \right) + \mathbf {z}_i, \end{aligned}$$

so by elementary row operations, we can transform \(\mathbf {M}\) into

$$\begin{aligned} \mathbf {M}' = \begin{pmatrix} \mathcal {M}_{t_{\mathsf {pub}}+1,q}\left( \mathbf {c} \right) \\ -\mathcal {M}_{t_{\mathsf {pub}}+1,q}\left( \mathbf {z}_1 \right) \\ \vdots \\ -\mathcal {M}_{t_{\mathsf {pub}}+1,q}\left( \mathbf {z}_u \right) \\ -\mathcal {M}_{k+t_{\mathsf {pub}}-u,q}\left( \mathbf {g} \right) \end{pmatrix}. \end{aligned}$$

Due to \(w+2t_{\mathsf {pub}}<n-k\), the matrix \(\mathcal {M}_{t_{\mathsf {pub}}+1,q}\left( \mathbf {z}_i \right) \) is a sub-matrix of \(\mathcal {M}_{n-k-w,q}\left( \mathbf {z}_i \right) \), so

$$\begin{aligned}&{{\,\mathrm{rk}\,}}_{q^m} (\mathbf {M}) = {{\,\mathrm{rk}\,}}_{q^m} (\mathbf {M}') \\&\quad \le \varphi + {{\,\mathrm{rk}\,}}_{q^m} (\mathcal {M}_{t_{\mathsf {pub}}+1,q}\left( \mathbf {c} \right) ) + {{\,\mathrm{rk}\,}}_{q^m} (\mathcal {M}_{k+t_{\mathsf {pub}}-u,q}\left( \mathbf {g} \right) ) \\&\quad = \varphi + k + 2t_{\mathsf {pub}}-u. \end{aligned}$$

Further, since the number of columns of \(\mathbf {M}\) is equal to n,

$$\begin{aligned} {{\,\mathrm{rk}\,}}_{q^m} (\mathbf {M}) \le n. \end{aligned}$$

\(\square \)

The linearization attack is inefficient if the rank of \(\mathbf {M}\) is smaller than its number of rows, which implies the following, stronger version of the original statement in [16].

Theorem 8

If \(t_{\mathsf {pub}}> \tfrac{n-k}{u+2}\) or \(\varphi < u (t_{\mathsf {pub}}+1)\), the linearization attack in [16] is inefficient and its work factor is

$$\begin{aligned} \text {WF}_{\mathrm {Lin}} = q^{m \cdot \max \{u t_{\mathsf {pub}}+ u + 1 - \varphi ,(u+2)t_{\mathsf {pub}}+ k +1 -n\}}. \end{aligned}$$

The first condition in Theorem 8 is again fulfilled by the choice of w in Table 1. The second one reads \(t_{\mathsf {pub}}> \tfrac{\varphi }{u}+1\), and for any valid \(\varphi \), there are choices of w such that \(t_{\mathsf {pub}}\) fulfills this inequality for any \(u>1\).

6.2.6 Algebraic attacks

Faure and Loidreau [16] also described two message attacks of exponential worst-case complexity. The first one is based on computing gcds of polynomials of degrees

$$\begin{aligned} \text {WF}_{\text {GCD}}= q^{m(u-1)}\frac{q^{t_{\mathsf {pub}}+1}-1}{q-1}. \end{aligned}$$

Since computing the gcd of two polynomials can be implemented in quasi-linear time in the polynomials’ degree, (14) gives an estimate on the work factor of this attack. The second algebraic attack is based on finding Gröbner bases of a system of \(n_\mathrm {p} = \genfrac(){0.0pt}1{n}{k+2 t_{\mathsf {pub}}-u+1}\) many polynomials of degree approximately \(d_\mathrm {p} = \tfrac{q^{t_{\mathsf {pub}}+1}-1}{q-1}\). The attack is only efficient for small code parameters, cf. [16, Sec. 5.3]. Since the average-case complexity of Gröbner bases algorithms is hard to estimate, we cannot directly relate \(n_\mathrm {p}\) and \(d_\mathrm {p}\) to the attack’s work factor. Faure and Loidreau choose the code parameters such that \(n_\mathrm {p} \approx 2^{32}\) and \(d_\mathrm {p} = 127\) and claim that the attack is inefficient for these values. Our example parameters in Sect. 7 result in at least these values.

6.2.7 Overbeck-like attack

The key attack described in [28, Ch. 7, Sec. 2.1] is based on a similar principle as Overbeck uses to attack the McEliece cryptosystem based on Gabidulin codes [35]. The attack from [28, Ch. 7, Sec. 2.1] cannot be applied if

$$\begin{aligned} w \ge n-k -\frac{k-u}{u-1}. \end{aligned}$$

6.2.8 Brute-force attack on the element \(\alpha \)

An attacker can brute-force \(\alpha \in \mathbb {F}_{q^{mu}}\), which has a complexity of

$$\begin{aligned} \text {WF}_{\alpha }= q^{mu}. \end{aligned}$$

By knowing \(\alpha \), he just needs to apply an efficient decoding algorithm on \(\tilde{\mathbf {c}} = \mathbf {c} - {{\,\mathrm{Tr}\,}}(\alpha \mathbf {k}_{\mathsf {pub}})\) to retrieve the secret message.

6.3 Exponential-time attacks on ResIG-Dec

We have seen in Sect. 5 that LIGA is IND-CCA2 secure under the assumption that ResG-Dec is a hard problem. The two previous subsections analyzed all known attacks on the ResG-Search and ResIG-Search problems, which are relevant since there is an obvious reduction of ResG-Dec to these search problems.

In the following, we study Problem ResIG-Dec (which translates to distinguishing the public key from a random vector in \(\mathbb {F}_{q^{mu}}^n\)), which is different in the sense that we do not know an efficient reduction from ResG-Dec (or one of the search problems) to ResIG-Dec. In other words, even if distinguishing the public key is easy, it might still be hard to distinguish the ciphertext. Nevertheless, we study the hardness of ResIG-Dec in the following and present a distinguisher, which is efficient to compute if \(\zeta \) is chosen small. The distinguisher is as follows.

Recall the choice of \(\mathbf {k}_{\mathsf {pub}}\) in Algorithm 3. We have

$$\begin{aligned} \mathbf {k}_{\mathsf {pub}}= \mathbf {x} \cdot \mathbf{G}_{\mathcal {G}} + \mathbf {z} \in \mathbb {F}_{q^{mu}}^n. \end{aligned}$$

Expand \(\mathbf {k}_{\mathsf {pub}}\) into a \(u \times n\) matrix over \(\mathbb {F}_{q^m}\) and choose any \(\zeta +1\) rows. As the \(\mathbb {F}_{q^m}\)-expansion of the error \(\mathbf {z}\) has \(\mathbb {F}_{q^m}\)-rank \(\zeta \), there are at least \(q^m-1\) many non-trivial \(\mathbb {F}_{q^m}\)-linear combinations of these \(\zeta +1\) rows that are codewords of \(G_{\mathcal {G}}\). This is not true with high probability for a random \(u \times n\) matrix over \(\mathbb {F}_{q^m}\).

Thus, by repeatedly randomly linearly combining these \(\zeta +1\) rows and checking whether the result is a codeword of \(G_{\mathcal {G}}\), we obtain a Monte-Carlo algorithm with an expected work factor of

$$\begin{aligned} \text {WF}_{\mathbf {k}_{\mathsf {pub}},\mathrm {distinguisher}} = q^{m\zeta }, \nonumber \end{aligned}$$

neglecting the cost of checking whether a vector in \(\mathbb {F}_{q^m}^n\) is a codeword. Hence, if \(m\zeta \) is smaller than the security parameter of the system, this distinguisher is feasible to compute.

6.4 Avoiding Weak Keys

As already discussed in Sect. 6.2, the work factors of the “Randomized Gabidulin Decoding Attack on the Ciphertext” and the “List Decoding of the Ciphertext Attack” depend on the rank of the error part \({{\,\mathrm{Tr}\,}}(\alpha \mathbf {z}) + \mathbf {e}\) of the ciphertext (seen as codeword plus error). Generically, this error has weight \(t_{\mathsf {pub}}+\mathbf {e}\), but due to the trace operation and the addition, the rank might be smaller.

In Appendix 3, we will analyze the probability that for a given key (i.e., \(\mathbf {z}\) in this case) and a random encryption (random choices of \(\alpha \) and \(\mathbf {e}\)) the rank is significantly smaller than expected (we use \(<w\) as a threshold, see Sect. 6.2). Briefly summarized, we get the following results.

It turns out that this probability heavily depends on the minimum distance of the code \(\mathcal {A}\) used to generate \(\mathbf {z}\) in Algorithm 3. The smaller this minimum distance, the larger the probability that the rank is low. More precisely, for a given \(\mathcal {A}\) of minimum distance \(2 \le t \le w-\zeta +2\)

$$\begin{aligned} \Pr ({{\,\mathrm{rk}\,}}_{\mathbb {F}_q}({{\,\mathrm{Tr}\,}}(\alpha \mathbf {z}) + \mathbf {e})<w) \le q^{-m\zeta }+ {256} \min \{t,t_{\mathsf {pub}}\}^2 q^{-(t+t_{\mathsf {pub}}-w+1)\left( n+\frac{t-3w-t_{\mathsf {pub}}}{2}\right) }, \end{aligned}$$

cf. Theorem 11 in Appendix 3.

Due to the above discussion, we call a key with \(\Pr ({{\,\mathrm{rk}\,}}_{\mathbb {F}_q}({{\,\mathrm{Tr}\,}}(\alpha \mathbf {z}) + \mathbf {e})<w)>2^{-\lambda }\) a weak key. In Appendix 3, we derive an upper bound on the probability of choosing weak key (i.e., an \(\mathcal {A}\) of too small minimum distance) in Algorithm 3. For \(\zeta q^{\zeta w-m} \le \tfrac{1}{2}\), this bound is roughly

$$\begin{aligned} \Pr (\text {weak key}) \le \Theta \!\left( q^{m[t-(w-\zeta +2)]} \right) , \end{aligned}$$

cf. Remark 2 (see Theorem 11 for a non-asymptotic bound) in Appendix 3, where t is the smallest minimum distance for which the key is not weak.

It can be seen that the parameters of LIGA can be chosen such that there is a t with \(2 \le t \le w-\zeta +2\) such that both \(\Pr ({{\,\mathrm{rk}\,}}_{\mathbb {F}_q}({{\,\mathrm{Tr}\,}}(\alpha \mathbf {z}) + \mathbf {e})<w)\) (for any non-weak key) and \(\Pr (\text {weak key})\) are smaller than \(2^{-\lambda }\). This is the case for all parameters proposed in Table 1.

6.5 Summary of the work factors

In this section, we recall the conditions on the choice of the parameters such that all known attacks are inefficient and summarize their work factors. Furthermore, we give specific parameters and compare LIGA to other code-based cryptosystems.

In the following, we choose the parameters q, m, n, k, u, w, and \(t_{\mathsf {pub}}\) as in Table 1. Recall that this choice of w prevents the Overbeck-like attack (Sect. 6.2.7) and results in an exponential work factor of the linearization attack (Sect. 6.2.5).

Furthermore, we choose \(\zeta \) to be small such that the work factor of searching the exponentially-large output of the interleaved decoding attack (Sect. 6.1.2) is large. Note that the latter attack returns an exponentially-large output if and only if the GOT [21] attack fails, cf. Theorem 3.

The resulting considered work factors are summarized in Table 2. In addition to these work factors, we have considered the following requirements:

  • The work factor of the second algebraic attack in [16] (cf. Sect. 6.2.6) is unknown. Hence, we choose the code parameters such that the resulting non-linear system of equations occurring in the attack consists of more than \(n_\mathrm {p} \approx 2^{32}\) many polynomials of degree at least \(d_\mathrm {p} = 127\). This is the same choice as in [16].

  • Since there is no efficient list decoder for Gabidulin codes, the work factor of the list decoding the public key or the ciphertext in Sect. 6.2.2 is not known. However, we do have a lower bound on the worst-case work factor for some codes, given by the maximal list size \(\mathcal {L}_\mathbf {c},\text {worst}\) in (12). In all examples for which the bound holds, we chose the parameters such that \(\log _2(\mathcal {L}_\mathbf {c},\text {worst})\) is much larger than the claimed security level.

  • The probability of generating a weak key should be negligible. Thus, we choose the parameters such that \(\zeta q^{\zeta w-m} \le \tfrac{1}{2}\) and

    $$\begin{aligned} \Pr (\text {weak key})&\le \frac{q^{m\zeta }-1}{(q^m-1)(q^{mw}-1)}\left( \sum _{i=0}^{t-1} \genfrac[]{0.0pt}{}{w}{i}_{q} \prod _{j=0}^{i-1} \left( q^m-q^j\right) -1 \right) \\&\le 2^{-\lambda }, \end{aligned}$$

    where \(\lambda \) is the security parameter and

    $$\begin{aligned} t := \min \left\{ t \, : \, q^{-m\zeta }+ {256} \min \{t,t_{\mathsf {pub}}\}^2 q^{-(t+t_{\mathsf {pub}}-w+1)\left( n+\frac{t-3w-t_{\mathsf {pub}}}{2}\right) } \le 2^{-\lambda } \right\} . \end{aligned}$$
Table 2 Summary of the Discussed Attacks’ Work Factor
Table 3 Parameter sets for 128, 192 and 256 bit security

7 Parameters and key sizes

We propose parameters for security levels of 128 bit, 192 bit and 256 bit in Table 3, where \(R=\frac{k-u}{n}\) denotes the rate. The parameters are chosen in a way that we can send at least 256 bit of information and thus the system can be used as a KEM. Further, we use a security margin of at least 20 bit. For all parameters, the algebraic attack based on computing gcds of polynomials is the most efficient attack.

To evaluate the performance of LIGA, we compare it to the IND-CCA-secure version [42] of Loidreau’s system [29] and the NIST proposals RQC [2], ROLLO [1], BIKE [3] and Classic McEliece [8]. We show the sizes of the private key \(\mathsf {sk}\), the public key \(\mathsf {pk}\) and the ciphertext \(\mathsf {ct}\) in Byte in Table 4 as proposed by the authors.Footnote 5 Note that in LIGA we can use a similar representation of the secret key and public key as in RQC, see [2, Section 2.3.3]. More precisely, we just store a seed of size 40 bytes to generate the secret key \(\mathsf {sk}= (\mathbf {x},\mathbf {P}_{[w+1,n]})\) which leads to secret key size of 40 bytes. The vector \(\mathbf {g}\) in the public key \(\mathsf {pk}= (\mathbf {g},\mathbf {k}_{\mathsf {pub}})\) can be also stored as a seed of size 40 bytes. Thus, the size of the public key \(\mathsf {pk}\) is equal to \(\big \lceil \frac{m n u\log _2(q)}{8} \big \rceil + 40\) bytes. The size of the ciphertext \(\mathsf {ct}\) is given by \(\lceil nm\log _2(q) \rceil \) bytes.

Table 4 Comparison of memory costs of \(\mathsf {sk}\), \(\mathsf {pk}\) and the ciphertext \(\mathsf {ct}\) in Byte with IND-CCA-secure Loidreau [42] and the NIST proposals RQC [2], ROLLO [1], BIKE [3] and Classic McEliece [8]. The entry ’yes’ in the column DFR indicates that a scheme has a decryption failure rate larger than 0

In [11], a generalization of Grover’s algorithm is proposed that finds the roots of a function f in \(\sqrt{2^b/r}\) function evaluations on average, where r is the number of roots and \(2^b\) is the number of possible inputs of f. Thus, in a post-quantum world, all shown attacks on \(\textsf {LIGA}\) may be accelerated using Grover’s algorithm except the GCD based attack and the Algebraic RSD attack. Similar to the quantum information-set decoding algorithm described in [9], the mentioned attacks have in common that they guess an element from a large set and then evaluate in polynomial time whether the guess leads to the desired outcome. If the desired outcome is obtained, the system can be broken in polynomial time using exactly this guess. Thus, the work factor of these algorithms is the product of the complexity of checking whether the guess leads to the desired outcome times the inverse of the probability that the guess leads to the desired outcome. Thus, we can easily construct a function f that takes as input a guess and checks in polynomial time whether the guess is as desired. If this is the case, f outputs 0 and otherwise anything except 0. Then, we can apply Grover’s algorithm to find a root of that function f. Such a root is then an element of the set that leads to the desired outcome. In this way, Grover’s algorithm reduces the work factor to the product of the polynomial time required for checking the guess times the square-root of the inverse of the probability that the guess is as desired. For the GCD based attack, we do not know how to improve the work factor by a quantum computer since the stated complexity already assumes a running time linear in the polynomials’ degree. Further, at the current state of research, there is no quantum speed up known for the Algebraic RSD attack [2, Section 6.3]. Using the described work factors, we obtain for LIGA-128, LIGA-192 and LIGA-256 a post-quantum security level of 97.5 bit, 127.5 bit and 157.5 bit, respectively, where Moving to Close Error is the most efficient attack for all three parameter sets.

8 Conclusion

In this paper, we presented a new rank-metric code-based cryptosystem: LIGA. LIGA uses a new coding-theoretic interpretation of the Faure–Loidreau system. We showed that the ciphertext is a corrupted codeword of a Gabidulin code, where to an unauthorized receiver, the error weight is too large to be correctable. The authorized user knows the row space of a part of the error and is thus able to correct the error. Further, we derived that a part of the public key can be seen as a corrupted codeword of an interleaved Gabidulin code and that in the original FL system, an interleaved Gabidulin decoder can efficiently recover the private key from this part of the public key with high probability. We proved that the condition that interleaved Gabidulin decoders fail is equal to the condition that the severe attack by Gaborit, Otmani and Talé Kalachi fails.

Based on the latter observation, we chose LIGA’s key generation algorithm such that interleaved Gabidulin decoders fail which in turn implies that the attack by Gaborit et al. fails.

We proposed two versions of LIGA and proved that the public key encryption is IND-CPA secure and the KEM is IND-CCA2 secure under the assumption that the ResG-Dec problem is hard. We extensively analyzed the security of this decisional problem by studying attacks on the ResG-Search, ResIG-Search, and ResIG-Dec (recall that there is a reduction of ResG-Dec to each of the two search problems). All studied attacks have an exponential work factor in the proposed parameter ranges and can be avoided by parameter choice.

Finally, we presented parameters for security levels of 128, 192 and 256 bit and compared them to the NIST proposals RQC, ROLLO, BIKE, Classic McEliece and a rank-metric McEliece-like system proposed by Loidreau. It was observed that LIGA has small ciphertext sizes as well as relatively small key sizes. Encryption and decryption correspond to encoding and decoding of Gabidulin codes, for which efficient and constant-time algorithms exist. Further, the proposed system guarantees decryption and is not based on hiding the structure of a code. Hence, the LIGA system should be considered as an alternative of small ciphertext and key size.