Finding orientations of supersingular elliptic curves and quaternion orders

Abstract

An oriented supersingular elliptic curve is a curve which is enhanced with the information of an endomorphism. Computing the full endomorphism ring of a supersingular elliptic curve is a known hard problem, so one might consider how hard it is to find one such orientation. We prove that access to an oracle which tells if an elliptic curve is $\mathfrak {O}$-orientable for a fixed imaginary quadratic order $\mathfrak {O}$ provides non-trivial information towards computing an endomorphism corresponding to the $\mathfrak {O}$-orientation. We provide explicit algorithms and in-depth complexity analysis. We also consider the question in terms of quaternion algebras. We provide algorithms which compute an embedding of a fixed imaginary quadratic order into a maximal order of the quaternion algebra ramified at p and $\infty$. We provide code implementations in Sagemath (in Stein et al. Sage Mathematics Software (Version 10.0), The Sage Development Team, http://www.sagemath.org, 2023) which is efficient for finding embeddings of imaginary quadratic orders of discriminants up to O(p), even for cryptographically sized p.

1 Introduction

Isogeny-based cryptography is a relatively new branch of post-quantum cryptography which is based on hard problems supposedly intractable even for quantum computers. The underlying hard problems were first introduced publicly in 2006 by the hash-function proposal of Charles-Goren-Lauter [12], and the works of Couveignes [15] and Rostovtsev-Stolbunov [56]. Since then, this field has blossomed with the introductions of new schemes such as SIDH [31] (now broken by [10, 41, 55]), CSIDH [11], and SQISign [27]. The hardness of all isogeny-based schemes is based on some variant of the path finding problem, which asks to find an isogeny between two given supersingular elliptic curves. The quaternion analogue of this hard problem has been efficiently solved [34], but the problem remains hard for supersingular elliptic curves. Path finding in the supersingular isogeny graph is equivalent to endomorphism ring computation, which was first heuristically proven in [21] and then rigorously (assuming GRH) proven in [65]. The key recovery of CSIDH was reduced to endomorphism ring computations in [66].

To study the hardness of the path finding problem it is natural to add some data to the elliptic curves and study how this data interacts with the graph structure. One way to do this is to add the information of an orientation to the elliptic curve vertices. Informally, an orientation on an elliptic curve E is an embedding of an imaginary quadratic order $\mathfrak {O}$ into the endomorphism ring of E which cannot be extended to a superorder of $\mathfrak {O}$. The resulting isogeny graph admits an abelian group action, which is used in cryptographic protocols such as CSIDH [11], Scallop [26], OSIDH [13], and SETA [18]. The group action is crucial for defining the Uber Isogeny Problem [66, Problem 4], whose hardness underlies all isogeny-based schemes. One might suspect that being given the information of an orientation could weaken the difficulty of the path finding problem, but this depends heavily on the given orientation and does not typically weaken the hardness of the path finding problem [1, 66]. A natural question to consider would be how to find an orientation on a curve, given that one exists. This is the $\mathfrak {O}$-Orienting Problem. It is also natural to consider the decisional version of this problem: given a supersingular elliptic curve E and a quadratic order ${\mathfrak {O}}$, can one decide whether E is orientable by ${\mathfrak {O}}$?

The search version of the $\mathfrak {O}$-Orienting Problem underlies the security of:

• OSIDH [13, 46], and

• FESTA [3] and its variants [44, 45] due to the following observation.

In all of these schemes there is a secret isogeny of known degree d from a curve which has small endomorphisms (usually the curve $y^2=x^3+x$). In all these cases the codomain of this isogeny is primitively oriented by $\mathbb {Z}[d\omega ]$ where $\omega$ is the mentioned special small degree endomorphism. Finding this orientation will reveal the secret isogeny. Furthermore, the problem of finding a fixed degree isogeny to a special curve (if it exists) is believed to be a hard problem. Since the best algorithms for attacking the aforementioned schemes are exponential time, the best algorithms for solving the $\mathfrak {O}$-Orienting Problem are also exponential time. The problem of deciding whether a curve is oriented is semantically a weaker problem and has not been previously studied. It is natural to question how the decision variant relates to the search variant of this problem, e.g. can it be solved in subexponential time? The search variant of a problem is always at least as hard as the decision variant. If there exists a reduction from search to decision, it shows that the decision variant of the problem reveals some nontrivial information. In cryptography, such “search-to-decision” reductions are helpful in understanding hardness assumptions.

Interestingly, these problems are not even efficiently solved on the quaternion side, except in very special cases. In [66] the case where ${{\,\textrm{disc}\,}}(\mathfrak {O})$ is small (i.e., ${{\,\textrm{disc}\,}}(\mathfrak {O})<\sqrt{p}$) is studied. This can be accomplished in polynomial time as in general this is just the smallest non-scalar endomorphism (this argument should actually work with a small modification for cases where ${{\,\textrm{disc}\,}}(\mathfrak {O})<p^{2/3}$). In [4, Appendix A] this is conjecturally improved to $p^{0.8}$ using Coppersmith techniques.

In this work, we give reductions between the search and decisional variants of these problems, and provide algorithms for the quaternion variant of these problems.

1.1 Our contributions

1.1.1 Reduction from Search to Decisional $\mathfrak {O}$-Orienting Problem

When the discriminant of $\mathfrak {O}$ is smaller than the characteristic p of the base field, we prove a subexponential reduction from the computational to the decisional version of the $\mathfrak {O}$-Orienting Problem. In particular, we provide an explicit algorithm (Algorithm 4.3) to find an $\mathfrak {O}$-orientation of an orientable elliptic curve in subexponential time and space when given access to an oracle deciding whether any elliptic curve is $\mathfrak {O}$-orientable. Access to this oracle is unlikely to be possible, but theoretical access to such a perfect oracle shows that finding an endomorphism is not significantly more difficult than simply deciding if one exists. In Sect. 4.2 we provide an in-depth analysis and proof of the complexity of the algorithm (Theorem 4.13). This proves that such an oracle gives non-trivial information since finding an orientation automatically yields a non-scalar endomorphism and the best known algorithms to find a non-scalar endomorphism on a supersingular elliptic curve are exponential [19, Sect. 4], [22, 28]. Note that in this paper we always invoke a perfect oracle which always returns the correct answer. The case of imperfect oracles (i.e., oracles which return the correct answer with probability $1-\epsilon$ for some fixed $\epsilon >0$) is left for future work.

Before treating the general case, we prove a polynomial reduction when $\mathfrak {O}$ is the maximal order of $\mathbb {Q}(\sqrt{-d})$ and d is the product of small distinct primes in Sect. 3. This allows us to illustrate the spirit of the more general algorithm in a less complicated setting. We provide an explicit algorithm for this case (Algorithm 3.1) and prove in Theorem 3.10 that this algorithm runs in polynomial time.

1.1.2 Quaternion Order Embedding Problem

In Sect. 5, we consider the Quaternion Order Embedding Problem (Problem 2.6) which is the quaternion analogue of the $\mathfrak {O}$-Orienting Problem. That is, given a maximal quaternion order $\mathcal {O} \subset B_{p, \infty }$ and a quadratic order $\mathfrak {O}$ which embeds into $\mathcal {O}$, find an embedding $\iota : \mathfrak {O} \hookrightarrow \mathcal {O}$ that cannot be extended to a superorder of $\mathfrak {O}$. In Sect. 5.1 we present a general algorithm to solve the problem of finding embeddings using a factorization oracle. We provide a complexity analysis based on several heuristics in Sect. 5.2. In Sect. 5.5 we show that finding embeddings which cannot be extended (i.e., orientations), only adds a small factor to the running time. We prove efficiency for the curve with j-invariant 1728, and describe a practical method for removing the dependence on the factorization oracle. Our algorithm improves the state of the art, since it is efficient up to even ${{\,\textrm{disc}\,}}(\mathfrak {O}) = O(p)$. We provide an implementation in Sagemath [61] which, for small discriminant orders, is fast for cryptographically sized p.

2 Preliminaries

We provide a concise summary of the necessary background and the state of the art algorithms which we use in this paper.

2.1 Supersingular elliptic curves and quaternion algebras

Let p be a prime. An elliptic curve over $\overline{\mathbb {F}_p}$ is called supersingular if any one of the following equivalent conditions holds:

• (1) ${{\,\mathrm{{\text {End}}}\,}}(E)$ is isomorphic to a maximal order in a quaternion algebra

• (2) $E[p^r] = 0_E$ for all $r\ge 1$

• (3) $j(E)\in \mathbb {F}_{p^2}$ and the multiplication-by-p map [p] is purely inseparable

• (4) The dual to the $p^r$-power Frobenius is purely inseparable for all $r\ge 1$.

See [60] for additional properties and proofs of equivalence.

We use the endomorphism ring heavily in what follows, so we describe here the necessary definitions and properties of quaternion objects. For more generality and more detail, we encourage the reader to see [62].

A (definite) quaternion algebra $\mathcal {A}$ is a noncommutative algebra which has rank 4 over $\mathbb {Q}$, and can be specified by generators i, j such that:

$$\begin{aligned} \mathcal {A} = \mathbb {Q}+ \mathbb {Q}i + \mathbb {Q}j + \mathbb {Q}k: \quad i^2,j^2\in \mathbb {Q},\quad i^2,j^2<0,\quad k:= ij = -ji. \end{aligned}$$

An order $\mathcal {O}$ in $\mathcal {A}$ is a $\mathbb {Z}$-submodule of $\mathcal {A}$ of rank 4 which is also a subring. An order is said to be maximal if it is not properly contained in any other order. For any full rank lattice (sub-$\mathbb {Z}$-module of rank 4) I in $\mathcal {A}$, we define its left order

$$\begin{aligned} O_L(I):=\{\alpha \in \mathcal {A}: \alpha I\subseteq I\}. \end{aligned}$$

The right order $O_R(I)$ is defined analogously. A full rank lattice I of $\mathcal {A}$ is said to be invertible if there exists a lattice $I'$ such that $II' = O_L(I) = O_R(I')$ and $I'I = O_R(I) = O_L(I')$. A full rank lattice in $\mathcal {A}$ is said to be a left (resp. right) $\mathcal {O}$-ideal if $\mathcal {O}\subseteq O_L(I)$ (resp. $\mathcal {O}\subseteq O_R(I)$). For every order $\mathcal {O}$ of $\mathcal {A}$ we define a left class set of equivalence classes of invertible left ideals: invertible left $\mathcal {O}$ ideals I, J are equivalent in the left class set of $\mathcal {O}$ if and only if there exists $\gamma \in \mathcal {A}^\times$ such that $I = \gamma J$. The left class set of invertible ideals is finite. The right class set of invertible ideals is analogously defined and is also finite.

For a fixed prime p, we define the (unique up to isomorphism) quaternion algebra $B_{p,\infty }$ to be the definite quaternion algebra ramified precisely at p and $\infty$. The endomorphism rings of supersingular elliptic curves over $\overline{\mathbb {F}_p}$ are isomorphic to maximal orders in $B_{p,\infty }$:

Theorem 2.1

(Deuring [20]) Fix a maximal order M of the quaternion algebra $B_{p,\infty }$ ramified precisely at p and $\infty$. There is a bijection between isomorphism classes of supersingular elliptic curves over $\overline{\mathbb {F}_p}$ and the left class set of the order M.

Given a supersingular elliptic curve $E/\overline{\mathbb {F}_p}$, one might ask to compute ${{\,\mathrm{{\text {End}}}\,}}(E)$ in different forms: to compute endomorphisms of E which generate ${{\,\mathrm{{\text {End}}}\,}}(E)$, or to compute the isomorphism class of ${{\,\mathrm{{\text {End}}}\,}}(E)$ abstractly in the quaternion algebra $B_{p,\infty }$. This problem is computationally difficult in all formulations. The supersingular endomorphism problem was recently shown to be equivalent to the problem of finding a single non-scalar endomorphism [47]. A priori, the information of one endomorphism $\omega$ of E reveals an imaginary quadratic order $\mathbb {Z}[\omega ]$ embedded within ${{\,\mathrm{{\text {End}}}\,}}(E)$. In Sect. 2.2, we provide more background information on such embeddings.

2.2 Orientations

Definition 2.2

(Orientation) Let $\mathfrak {O}$ be an imaginary quadratic order. An $\mathfrak {O}$-orientation of a supersingular elliptic curve $E/\overline{\mathbb {F}_p}$ is an embedding $\iota :\mathfrak {O}\hookrightarrow {{\,\mathrm{{\text {End}}}\,}}(E)$ which cannot be extended to a larger quadratic order containing $\mathfrak {O}$. The pair $(E,\iota )$ is called an $\mathfrak {O}$-oriented supersingular elliptic curve.

Definition 2.2 corresponds to the definition of primitive $\mathfrak {O}$-orientation found elsewhere in the literature [1, 13, 46]. We omit the word “primitive" in our definition, as almost all of our $\mathfrak {O}$-orientations are primitive. When we want to discuss an embedding $\mathfrak {O}\hookrightarrow {{\,\mathrm{{\text {End}}}\,}}(E)$ which can be extended to a superorder of $\mathfrak {O}$, we highlight this by using the term “imprimitive".

The notion of an orientation as in Definition 2.2 was recently introduced to isogeny-based cryptography by Colò and Kohel [13] and was subsequently studied [1, 2, 26, 46, 66]. The quaternion counterpart of this notion has a longer history, dating back to Chevalley, Hasse, and Noether and often referred to as the theory of optimal embeddings.

Supersingular elliptic curves which admit an $\mathfrak {O}$-orientation are called $\mathfrak {O}$-orientable. There is an action of the class group ${{\,\mathrm{{\text {Cl}}}\,}}(\mathfrak {O})$ on the set of $\mathfrak {O}$-oriented supersingular elliptic curves induced by the following action of an invertible $\mathfrak {O}$-ideal $\mathfrak {a}$:

$$\begin{aligned} \mathfrak {a}*(E,\iota ) :=(E_\mathfrak {a},(\varphi _{\mathfrak {a}})_*\iota ), \end{aligned}$$

where $E_\mathfrak {a}$ is the codomain of the degree-$N(\mathfrak {a})$ isogeny $\varphi _\mathfrak {a}:E\longrightarrow E_\mathfrak {a}$ with kernel $\cap _{\alpha \in \mathfrak {a}}\ker \alpha$. The orientation $(\varphi _{\mathfrak {a}})_*\iota :\mathfrak {O}\hookrightarrow {{\,\mathrm{{\text {End}}}\,}}(E_\mathfrak {a})$ is given via $(\varphi _{\mathfrak {a}})_*\iota (-) :=\frac{1}{N(\mathfrak {a})}\varphi _\mathfrak {a}\circ \iota (-)\circ \widehat{\varphi _{\mathfrak {a}}}$.

For an imaginary quadratic field K, K embeds into the quaternion algebra $B_{p,\infty }$ if and only if p does not split in K. However, for a particular imaginary quadratic order $\mathfrak {O}$ and a particular supersingular elliptic curve E, it is generally difficult to decide if E is $\mathfrak {O}$-orientable. Naturally we are inclined to study the following problems and the relationship between them:

Problem 2.3

(Decisional $\mathfrak {O}$-Orienting Problem) Given an elliptic curve E and an imaginary quadratic order $\mathfrak {O}$, determine if E is orientable by $\mathfrak {O}$.

Problem 2.4

($\mathfrak {O}$-Orienting Problem) Given an elliptic curve E which is orientable by an imaginary quadratic order $\mathfrak {O}$, find the orientation.

Remark 2.5

In a cryptographic context decisional problems are often defined in a slightly different fashion. Namely an adversary is presented with two supersingular elliptic curves $E_1$ and $E_2$ and an imaginary quadratic order $\mathfrak {O}$ one of which is oriented by $\mathfrak {O}$. Then one has to decide whether $E_1$ or $E_2$ is oriented by $\mathfrak {O}$. The reason is that whenever the discriminant is sufficiently small (i.e., much smaller than $p^2$), then a random supersingular elliptic curve is likely not oriented by $\mathfrak {O}$. Hence an adversary that always says "not oriented" succeeds in this game with overwhelming probability.

The search-to-decision reduction shows that a decision oracle provides non-trivial information towards finding the endomorphism ring of the curve, and this will be the focus of Sects. 3 and 4.

We explore the following quaternion variant of Problem 2.4 in Sect. 5.

Problem 2.6

(Quaternion Order Embedding Problem) Given a maximal quaternion order $\mathcal {O}$ and an imaginary quadratic order $\mathfrak {O}$ which embeds into $\mathcal {O}$, find the embedding.

One may also consider the group action variant of the Uber-isogeny problem, originally introduced in [18], although we do not pursue this perspective in this work:

Problem 2.7

($\mathfrak {O}$-Uber Isogeny Problem) Given a supersingular elliptic curve E with an $\mathfrak {O}$-orientation $\iota :\mathfrak {O}\hookrightarrow {{\,\mathrm{{\text {End}}}\,}}(E)$ and an $\mathfrak {O}$-orientable supersingular elliptic curve F, find an ideal $\mathfrak {a}\in {{\,\mathrm{{\text {Cl}}}\,}}(\mathfrak {O})$ such that $\mathfrak {a}*E = F$.

2.3 Complexity

Throughout this paper, we shall give time and space complexity results. By default, we say an algorithm has (time) complexity O(f(n)) or terminates in time O(f(n)) when it runs in O(f(n)) bit operations. In particular, an algorithm is polynomial in the parameter n if it runs in number of bit operations which is bounded by a polynomial function of n. Very often, we shall also mention time complexity in terms of arithmetic operations over $\mathbb {Z}$ or finite fields $\mathbb {F}_{p}, \mathbb {F}_{p^2}$ and their extensions. By arithmetic operations or simply operations we mean additions, substractions, multiplications, inversions and random sampling (up to a bound over $\mathbb {Z}$). Unless explicitely stated otherwise, space complexity will always be counted in bits.

2.4 Computing modular polynomials and j-invariants

Given a prime number $\ell \ll p$ and the j-invariant $j(E)\in \mathbb {F}_{p^2}$ of a supersingular elliptic curve, we explain how to find all $\ell$-isogenous j-invariants $j(E')\in \mathbb {F}_{p^2}$ using modular polynomials $\Phi _\ell (X,Y)$. By [54, Theorem 6.3], $\Phi _\ell (j(E),Y)\in \mathbb {F}_{p^2}[Y]$ can be computed with ${\tilde{O}}(\ell ^2\log (p))$ operations over $\mathbb {F}_{p^2}$, where the ${\tilde{O}}$ means that polynomial factors in $\log (\ell )$ are omitted.¹ In [39, Sect. 5] the authors provide an algorithm with similar complexity. We then find all the roots over $\mathbb {F}_{p^2}$ of the degree-$(\ell +1)$ polynomial $\Phi _\ell (j(E),Y)$ in ${\tilde{O}}(\ell ^2\log (p))$ operations over $\mathbb {F}_{p^2}$ [63, Theorem 14.14] to find the j-invariants $j(E')\in \mathbb {F}_{p^2}$ that are $\ell$-isogenous to j(E). On the whole, the computation costs ${\tilde{O}}(\ell ^2\log (p))$ operations over $\mathbb {F}_{p^2}$.

2.5 Computing an $\ell$-isogeny between two j-invariants

Given two supersingular j-invariants $j(E)\in \mathbb {F}_{p^2}$ and $j(E')\in \mathbb {F}_{p^2}$ we explain how to find an $\ell$-isogeny $\phi : E\longrightarrow E'$ in ${\tilde{O}}(\ell ^2\log (p))$ operations over $\mathbb {F}_{p^2}$ using a method due to Elkies.

By [7, Theorem 2], given Weierstrass equations of E and $E'$, we can find (if it exists) a normalized $\ell$-isogeny $\phi : E\longrightarrow E'$ with only ${\tilde{O}}(\ell )$ arithmetic operations over $\mathbb {F}_{p^2}$. By normalized, we mean that $\phi$ pulls back the invariant differential $\omega ':=dx'/2y'$ of $E'$ to the invariant differential $\omega :=dx/2y$ of E ($\phi ^*\omega '=\omega$).

The existence of such a normalized isogeny $\phi$ only depends on the choice of Weierstrass equations for E and $E'$ which determine the constant $\lambda :=\phi ^*\omega '/\omega$. Knowing only j(E) and $j(E')$, we have multiple choices of Weierstrass equations and we have to pick one so that $\lambda =1$. We fix an equation for $E: y^2=x^3+Ax+B$, then find an equation for $E'$ so that $\lambda = 1$. Following the method given by [58, Sect. 7] (referring to ideas introduced in [23, Sect. 3]), we take $E':y^2=x^3+A'x+B'$, with

$$\begin{aligned} A':=-\frac{j'(E')^2\ell ^4}{48j(E')(j(E')-1728)} \quad B':=-\frac{j'(E')^3\ell ^6}{864j(E')^2(j(E')-1728)}, \end{aligned}$$

(1)

and

$$\begin{aligned} j'(E'):=-\frac{j'(E)}{\ell }\frac{\partial \Phi _\ell }{\partial X}(j(E), j(E'))\left( \frac{\partial \Phi _\ell }{\partial Y}(j(E), j(E'))\right) ^{-1}, \end{aligned}$$

(2)

where

$$\begin{aligned} j'(E):=\left\{ \begin{array}{cr} \frac{18B j(E)}{A}&{} \hbox { if } A\ne 0 \\ 0 &{} \hbox { if } A=j(E)=0 \end{array} \right. . \end{aligned}$$

(3)

The derivatives $\partial \Phi _\ell /\partial X$ and $\partial \Phi _\ell /\partial Y$ can be precomputed with ${\tilde{O}}(\ell ^2\log (p))$ operations over $\mathbb {F}_{p^2}$ using the techniques in Sect. 2.4 (see [51, Remark 5.3.10]). Hence, in total, computing an isogeny $\phi : E\longrightarrow E'$ costs ${\tilde{O}}(\ell ^2\log (p))$ operations over $\mathbb {F}_{p^2}$.

The above formulas fail when j(E) or $j(E')\in \{0, 1728\}$, and when $\partial \Phi _\ell /\partial Y(j(E), j(E'))$ vanishes. However, when $\partial \Phi _\ell /\partial Y(j(E), j(E'))=0$ and $\partial \Phi _\ell /\partial X(j(E), j(E'))\ne 0$, this implies there are more than one $\ell$-isogeny from the isomorphism class determined by j(E) to the isomorphism class determined by $j(E')$, but only one $\ell$-isogeny in the reverse direction. This is only possible if $j(E)\in \{0,1728\}$, in which case the formulas fail anyways.

Hence, the only cases when this method does not apply are j(E) or $j(E')\in \{0, 1728\}$, which are very unlikely (probability O(1/p)) and $\partial \Phi _\ell /\partial X(j(E), j(E'))=\partial \Phi _\ell /\partial Y(j(E), j(E'))=0$, i.e. when $(j(E),j(E'))$ is a singular point of the affine curve given by the modular equation $\Phi _\ell (X,Y)=0$ over $\mathbb {F}_{p^2}$. Following [58, Sect. 7], we prove in Appendix A that this is very unlikely when $\log (\ell )\ll \log (p)$ (which will be the case in our paper).

We can still handle singular cases at a higher cost of ${\tilde{O}}(\ell ^{7/2})$ operations over $\mathbb {F}_{p^2}$ with a naive algorithm. We enumerate all the cyclic subgroups of order $\ell$ of $E[\ell ]$ (there are $\ell +1$ of them) and use [6] to compute each $\ell$-isogeny with ${\tilde{O}}(\sqrt{\ell })$ operations over the field extension $K/\mathbb {F}_{p^2}$ where $E[\ell ]$ is defined. As will be proved in Lemma 2.12, K has degree $O(\ell )$ over $\mathbb {F}_{p^2}$ so one arithmetic operation over K is equivalent to at most $O(\ell ^2)$ operations over $\mathbb {F}_{p^2}$. Since singular cases are very unlikely when $\log (\ell )\ll \log (p)$, we may assume throughout this paper that computing $\ell$-isogenies between j-invariants costs ${\tilde{O}}(\ell ^2\log (p))$ operations over $\mathbb {F}_{p^2}$ on average by Lemma A.2.

2.6 Efficiently representing an isogeny of any degree with Kani’s lemma

Let $\varphi : E\longrightarrow E'$ be an isogeny of degree d between supersingular elliptic curves $E,E'/\mathbb {F}_{p^2}$. In general, we can represent $\varphi$ with data of size O(d). We either have direct formulas to evaluate $\varphi$ (given by rational fractions) or equivalently, generators of the kernel (defined over an $\mathbb {F}_{p^2}$-extension of degree O(d)) from which we can derive these formulas by [64]. In this case, evaluating $\varphi$ on a point takes linear time in d. We can do much better when d is smooth by representing $\varphi$ as a product of small degree isogenies. This is an efficient representation, in the sense of the following definition.

Definition 2.8

Let $\mathscr {A}$ be an algorithm (to compute isogenies). An efficient representation of an isogeny $\varphi : E\longrightarrow E'$ defined over a finite field $\mathbb {F}_q$ (with respect to $\mathscr {A}$) is given by some data $D\in \{0,1\}^*$ such that:

• (i) D has polynomial size in $\log (\deg (\varphi ))$ and $\log (q)$ (in bits).

• (ii) On input D and $P\in E(\mathbb {F}_{q^k})$, $\mathscr {A}$ returns $\varphi (P)$ in polynomial time in $k\log (q)$ and $\log (\deg (\varphi ))$.

We can also efficiently represent $\varphi$ when d is not smooth, using an idea first introduced in the attacks against SIDH [10, 41, 52] and then reused for several other applications [16, 53, 54]: provided we can evaluate $\varphi$ on some torsion points, we can “embed" $\varphi$ in a smooth degree higher dimensional isogeny F. Knowing F, we can evaluate $\varphi$ everywhere in polynomial time. This provides an efficient representation of $\varphi$.

In this section, we explain how to obtain such an efficient representation F of $\varphi$ when we are given access to the image of $\varphi$ on some torsion points (Algorithm 2.1). The evaluation of these points may be costly but other operations to compute F take polynomial time (Proposition 2.12).

Definition 2.9

(d-isogeny in higher dimension) Let $\alpha : (A,\lambda _A)\longrightarrow (B,\lambda _B)$ be an isogeny between principally polarized abelian varieties (PPAV). We denote by ${\widetilde{\alpha }}$ the isogeny

$$\begin{aligned} B\overset{\lambda _B}{\longrightarrow }{\widehat{B}}\overset{{\widehat{\alpha }}}{\longrightarrow }{\widehat{A}} \overset{\lambda _A^{-1}}{\longrightarrow }A, \end{aligned}$$

where ${\widehat{\alpha }}$ is the dual isogeny of $\alpha$.

We say that $\alpha$ is a d-isogeny if ${\widetilde{\alpha }}\circ \alpha =[d]_A$, or equivalently if $\alpha \circ {\widetilde{\alpha }}=[d]_B$.

We use the following result due to Kani [33, Theorem 2.3]. A concise expression of this result may be found in [52, Lemma 3.6].

Lemma 2.10

(Kani) Consider a commutative diagram of isogenies between PPAV:

where $\varphi$ and $\varphi '$ are a-isogenies and $\psi$ and $\psi '$ are b-isogenies.

Then, the isogeny chain $F: A\times B'\longrightarrow B\times A'$ given in matrix notation by

$$\begin{aligned} F:=\left( \begin{matrix} \varphi &{} \widetilde{\psi '}\\ -\psi &{} \widetilde{\varphi '} \end{matrix}\right) \end{aligned}$$

is a d-isogeny with $d:=a+b$, for the product polarizations.

If a and b are coprime, the kernel of F is

$$\begin{aligned} \ker (F)=\{({\widetilde{\varphi }}(x),\psi '(x))\mid x\in B[d]\}. \end{aligned}$$

Let $N>d$ be a powersmooth integer coprime with d. We can always write $N=d+a_1^2+a_2^2+a_3^2+a_4^2$ for some $a_1, a_2, a_3, a_4\in \mathbb {Z}$, by Lagrange’s four square theorem. Let $\alpha \in {{\,\mathrm{{\text {End}}}\,}}(E^4)$ be the isogeny written in matrix form as follows:

$$\begin{aligned} \alpha :=\left( \begin{array}{cccc} a_1 &{} -a_2 &{} -a_3 &{} -a_4 \\ a_2 &{} a_1 &{} a_4 &{} -a_4 \\ a_3 &{} -a_4 &{} a_1 &{} a_2 \\ a_4 &{} a_3 &{} -a_2 &{} a_1 \end{array}\right) , \end{aligned}$$

(4)

and $\alpha '$ be its analogue in ${{\,\mathrm{{\text {End}}}\,}}(E')$. Let $\Phi :={{\,\textrm{Diag}\,}}(\varphi ,\varphi ,\varphi ,\varphi ): E^4\longrightarrow E'^4$. Then, $\Phi$ is a d-isogeny, $\alpha$ and $\alpha '$ are $(N-d)$-isogenies and we have a commutative diagram:

that yields an 8-dimensional N-isogeny:

$$\begin{aligned} F:=\left( \begin{array}{cc} \alpha &{} {\widetilde{\Phi }} \\ -\Phi &{} {\widetilde{\alpha }} \end{array}\right) \in {{\,\mathrm{{\text {End}}}\,}}(E^4\times E'^4), \end{aligned}$$

(5)

with kernel:

$$\begin{aligned} \ker (F):=\{({\widetilde{\alpha }}(P),\Phi (P))\mid P\in E^4[N]\}, \end{aligned}$$

(6)

since N and d are coprime. By the above formula, we can compute $\ker (F)$ if we can evaluate $\varphi$ on generators of E[N]. We can then compute F as a product of small degree isogenies (as in dimension 1) and evaluate $\varphi$ efficiently everywhere as a component of F. Indeed, if we want to evaluate $\varphi (P)$ for some point $P\in E$, then we can compute:

$$\begin{aligned} F(P,0,\cdots ,0)=([a_1]P,[a_2]P,[a_3]P,[a_4]P,-\varphi (P),-\varphi (P),-\varphi (P),-\varphi (P)). \end{aligned}$$

Lemma 2.11

Let $F: E^4\times E'^4\longrightarrow E^4\times E'^4$ be the N-isogeny defined by (5) and suppose $N=\prod _{i=1}^s q_i^{e_i}$, where $q_1, \cdots , q_s$ are distinct primes. Then, we can decompose F as:

$$\begin{aligned} \mathcal {A}_0\overset{F_1}{\longrightarrow }\ \mathcal {A}_1 \overset{F_2}{\longrightarrow } \cdots \mathcal {A}_{s-1}\overset{F_s}{\longrightarrow }\mathcal {A}_s, \end{aligned}$$

with $\mathcal {A}_0=\mathcal {A}_s:=E^4\times E'^4$ and where $F_i$ is a $q_i^{e_i}$-isogeny for all $i\in \{1,\cdots , s\}$.

Let $K:=\ker (F)$. Moreover,

$$\begin{aligned} \ker (F_1)=K[q_1^{e_1}] = \{P\in K: [q_1^{e_1}] P = 0\} \quad \hbox {and}\\ \quad \ker (F_i)=F_{i-1}\circ \cdots \circ F_1(K[q_i^{e_i}]),\, \text {for } 2\le i\le s. \end{aligned}$$

Proof

The decomposition $F=F_s\circ \cdots \circ F_1$ was proven in [16, Proposition 5.4.1].

Now, let $K_1:=K[q_1^{e_1}]$ and $K_i:=F_{i-1}\circ \cdots \circ F_1(K[q_i^{e_i}])$ for all $2\le i\le s$. Then, $F_s\circ \cdots \circ F_i(K_i)=F(K[q_i^{e_i}])=\{0\}$, so that

$$\begin{aligned} F_i(K_i)\subseteq \ker (F_s\circ \cdots \circ F_{i+1})\subseteq \mathcal {A}_i\left[ \prod _{j\ge i+1}q_j^{e_j}\right] . \end{aligned}$$

But $F_i(K_i)\subset \mathcal {A}_i[q_i^{e_i}]$ and the primes $q_j\ne q_i$ for $j> i$, so we must have $F_i(K_i)=\{0\}$ and $K_i\subseteq \ker (F_i)$. Now, $\# K_i=\# K[q_i^{e_i}]=q_i^{8e_i}=\deg (F_i)$ since $F_{i-1}\circ \cdots \circ F_1$ has degree coprime with $q_i$. It follows that $K_i=\ker (F_i)$. $\square$

Each $q_i^{e_i}$-isogeny $F_i$ in Lemma 2.11 can be computed with $O(q_i^{8e_i})$ operations over the field of definition of their kernel with the theta model [40]. We summarize this computation in Algorithm 2.1.

Lemma 2.12

Let $E/\mathbb {F}_{p^2}$ be a supersingular elliptic curve and $n\in \mathbb {Z}_{>0}$. Then E[n] is defined over an extension of degree at most $6\phi (n)$ of $\mathbb {F}_{p^2}$, where $\phi$ is Euler’s totient function.

Proof

We first compute the characteristic polynomial of iterates of the Frobenius. Let $\chi _{p^2}:=(X-\alpha )(X-\beta )$ be the characteristic polynomial of the $p^2$ Frobenius $\pi _{p^2}$. Then $\chi _{p^{2\delta }}:=(X-\alpha ^\delta )(X-\beta ^\delta )$ is the characteristic polynomial of the $p^{2\delta }$ Frobenius $\pi _{p^{2\delta }}$, for any $\delta \in \mathbb {Z}_{>0}$.

Since E is supersingular $p\mid {{\,\textrm{Tr}\,}}(\pi _{p^2})=\alpha +\beta$, and $|{{\,\textrm{Tr}\,}}(\pi _{p^2})|\le 2p$ so ${{\,\textrm{Tr}\,}}(\pi _{p^2})\in \{0,\pm p,\pm 2p\}$ [58, Proposition 3.6]. We consider these possibilities in three cases:

If ${{\,\textrm{Tr}\,}}(\pi _{p^2})=0$, then $\chi _{p^2}=X^2+p^2=(X-ip)(X+ip)$ so

$$\begin{aligned}{{\,\textrm{Tr}\,}}(\pi _{p^{2\delta }})=(ip)^\delta +(-ip)^\delta =\left\{ \begin{array}{cl} 2p^\delta &{} \hbox {if } \delta \equiv 0 \mod 4 \\ -2p^\delta &{} \hbox {if } \delta \equiv 2 \mod 4 \\ 0 &{} \hbox {otherwise} \end{array}\right. . \end{aligned}$$

If ${{\,\textrm{Tr}\,}}(\pi _{p^2})=\pm p$, then $\chi _{p^2}=X^2\mp pX+p^2=(X\mp pe^{i\pi /3})(X\mp pe^{-i\pi /3})$ so

$$\begin{aligned} {{\,\textrm{Tr}\,}}(\pi _{p^{2\delta }})=(\pm 1)^\delta p^\delta (e^{i\delta \pi /3}+e^{-i\delta \pi /3})=\left\{ \begin{array}{cl} 2p^\delta &{} \hbox {if } \delta \equiv 0 \mod 6 \\ \mp 2p^\delta &{} \hbox {if } \delta \equiv 3 \mod 6 \\ \pm p^\delta &{} \hbox {if } \delta \equiv \pm 1 \mod 6 \\ -p^\delta &{} \hbox {if } \delta \equiv \pm 2 \mod 6 \end{array}\right. . \end{aligned}$$

Finally, if ${{\,\textrm{Tr}\,}}(\pi _{p^2})=\pm 2p$, then $\chi _{p^2}=X^2\mp 2pX+p^2=(X\mp p)^2$ so

$$\begin{aligned} {{\,\textrm{Tr}\,}}(\pi _{p^{2\delta }})=2(\pm p)^\delta . \end{aligned}$$

In all cases, if $\delta \equiv 0 \mod 12$, we have ${{\,\textrm{Tr}\,}}(\pi _{p^{2\delta }})=2p^\delta$, so $\chi _{p^{2\delta }}=(X-p^\delta )^2$. It follows that $\pi _{p^{2\delta }}=[p^\delta ]$.

Now, if we assume $\phi (n) \mid \delta$, then $p^\delta \equiv 1 \mod n$, so for all $P\in E[n]$, $\pi _{p^{\delta }}(P)=[p^\delta ]P=P$, so that $P\in E(\mathbb {F}_{p^{2\delta }})$.

Hence, E[n] is defined over $\mathbb {F}_{p^{2\delta }}$ provided that 12 and $\phi (n)$ divide $\delta$. If n has an odd prime factor or $n=2^k$ with $k\ge 2$, then $\phi (n)$ is even so $12 \mid 6\phi (n)$ so $\delta =6\phi (n)$ satisfy the desired conditions. If $n=2$, then E[n] is formed by 0 and the points (x, 0) where x is the root of a cubic polynomial equation over $\mathbb {F}_{p^2}$ (Weierstrass equation of E). Hence, E[2] is defined over an extension of degree at most 3 of $\mathbb {F}_{p^2}$. $\square$

Proposition 2.13

Algorithm 2.1 terminates and is correct. It requires:

• $O(\log (d))$ evaluations of the input d-isogeny $\varphi$ on points defined over an extension of degree O(D) of $\mathbb {F}_{p}$;

• $O(D^3\log (p)\log (d)+D^{10}\log ^2(d))$ arithmetic operations over $\mathbb {F}_{p}$;

• and $O(\log ^2(d)+\log (d)D\log \log (D))$ arithmetic operations over integers of size at most $O(D\log (p))$ bits.

Recall that by arithmetic operations, we mean addition, substraction, multiplication and inversion. Over $\mathbb {F}_{p}$, we count sampling as an arithmetic operation.

Proof

Correctness has been justified by Eq. 6 and Lemma 2.11. Termination is clear.

Now we compute the complexity. If N is only slightly bigger than d, then $s=O(\log (d))$ and finding a suitable N on line 1 of the algorithm takes $O(\log (d)D\log \log (D))$ arithmetic operations over $\mathbb {Z}$ (assuming we use the sieve of Eratosthenes).

Finding the $a_i$ on line 2 costs $O(\log ^2(N)/\log \log (N))=O(\log ^2(d))$ operations over $\mathbb {Z}$ with Pollack and Treviño’s second algorithm [49, §4].

For $i\in \{1,\cdots , s\}$, a basis of $E[q_i^{e_i}]$ is defined over a field extension of degree $\delta =O(q_i^{e_i})=O(D)$ of $\mathbb {F}_{p^2}$ by Lemma 2.12. To generate such a basis, we first sample a random point $P'\in E(\mathbb {F}_{p^{2\delta }})$ and compute $P_{i,1}:=[M/q_i^{e_i}]P'$, where $M:=\# E(\mathbb {F}_{p^{2\delta }})$ until P has order $q_i^{e_i}$. By the same method, we sample $P_{i,2}\in E[q_i^{e_i}]$ until $(P_{i,1},P_{i,2})$ is a basis of $E[q_i^{e_i}]$.

To sample $P'\in E(\mathbb {F}_{p^{2\delta }})$, we first sample $x\in \mathbb {F}_{p^{2\delta }}$ repeatedly (O(1) times at most) until we find $y\in \mathbb {F}_{p^{2\delta }}$ such $P'=(x,y)\in E$. Each sampling over $\mathbb {F}_{p^{2\delta }}$ costs $2\delta =O(D)$ samplings over $\mathbb {F}_{p}$. Computing y requires a square root computation in $\mathbb {F}_{p^{2\delta }}$ which costs $O(\log (p^{2\delta }))=O(D\log (p))$ multiplications over $\mathbb {F}_{p^{2\delta }}$ by Cipolla-Lehmer’s algorithm [37]. Without loss of generality (see the proof of Lemma 2.12), we may assume that $\pi _{p^{2\delta }}=[p^\delta ]$, so that:

$$\begin{aligned} M=\# E(\mathbb {F}_{p^{2\delta }})=p^{2\delta }+1-{{\,\textrm{Tr}\,}}(\pi _{p^{2\delta }})=(p^\delta -1)^2 \end{aligned}$$

and computing M costs O(D) operations over $\mathbb {Z}$. The scalar multiplication by $M/q_i^{e_i}$ costs $O(\log (M))=O(D\log (p))$ arithmetic operations over $\mathbb {F}_{p^{2\delta }}$. Testing that $(P_{i,1},P_{i,2})$ is a basis costs O(D) elliptic curve additions so O(D) arithmetic operations over $\mathbb {F}_{p^{2\delta }}$ (we compute the $[k]P_{i,1}$ and $[l]P_{i,2}$ for $1\le k, l\le q_i^{e_i}-1$ and conclude that we have a basis if these two sets are disjoint). Only O(1) samplings of $P_{i,1}$ and $P_{i,2}$ are necessary before we find a basis. Each arithmetic operation over $\mathbb {F}_{p^{2\delta }}$ costs at most $O(\delta )=O(D)$ arithmetic operations over $\mathbb {F}_{p}$, so the overall complexity to find a basis is $O(D^3\log (p))$ operations over $\mathbb {F}_p$ (and O(D) operations over $\mathbb {Z}$).

Line 6 costs two evaluations of $\varphi$ and line 9 costs eight scalar multiplications by the $a_i$, costing $O(D^2\log (d))$ operations over $\mathbb {F}_{p}$ each. Hence, the total cost of the loop of lines 4–9 is

$$\begin{aligned} O(s(D^2\log (d)+D^3\log (p)))=O(\log (d)(D^2\log (d)+D^3\log (p))) \end{aligned}$$

arithmetic operations over $\mathbb {F}_{p}$, $O(D\log (d))$ operations over $\mathbb {Z}$ and $2s=O(\log (d))$ evaluations of $\varphi$.

Finally, computing each $F_i$ costs $O(q_i^{8e_i})=O(D^8)$ arithmetic operations over $\mathbb {F}_{p^2}$ and computing the basis $\mathcal {C}_j$ ($i+1\le j\le s$) on line 14 costs $8(s-i)$ point evaluations, each costing $O(q_i^{8e_i})=O(D^8)$ arithmetic operations over an extension of degree O(D) of $\mathbb {F}_{p^2}$. The total cost of the loop of lines 11 –16 is

$$\begin{aligned} O(sD^{10}+s^2D^{10})=O(D^{10}\log ^2(d)) \end{aligned}$$

arithmetic operations over $\mathbb {F}_{p}$. $\square$

2.7 Smoothness test and factorization with the ECM method

In this section, we explain how to test if an integer N is B-smooth and find its factorization if it is the case. A naive method would be to use trial division, but it is not optimal when B is subexponential (which will be the case in the paper). An alternate method would be to factor N with the General Number Field Sieve (GNFS) [9] and test if its prime factors are $\le B$. However, GNFS underperforms with smooth integers. Hence, as Lenstra himself suggested [38, §2.12], we use the elliptic-curve factorization method (ECM) for that purpose. Finding a prime factor of N with this method is conjectured to take $L_\ell (1/2,\sqrt{2}){{\,\mathrm{{\text {polylog}}}\,}}(N)$ bit operations [38, Conjecture 2.10], where $\ell$ is the smallest prime divisor of N and with the usual notation

$$\begin{aligned} L_x(\alpha ,\beta ):=\exp \left( (\beta +o(1))(\log (x))^\alpha (\log \log (x))^{1-\alpha }\right) , \end{aligned}$$

where o(1) is for $x \rightarrow \infty$. Hence, to test the B-smoothness of N, we simply apply ECM to find a factor $k \mid N$ after expected time $L_B(1/2,\sqrt{2}){{\,\mathrm{{\text {polylog}}}\,}}(N)$. If the running time exceeds what it should be, it means that N is not B-smooth and we stop. Otherwise, we continue and try to factor k and N/k recursively until we have either completely factored N or concluded it is not B-smooth. Algorithm 2.2 follows.

If r is the number of prime divisors of N (with multiplicity), then $r=O(\log (N))$ and Algorithm 2.2 can terminate with at most r calls to ECM, so it terminates in time $L_B(1/2,\sqrt{2}){{\,\mathrm{{\text {polylog}}}\,}}(N)$. This time complexity is still a conjecture and Pomerance provides a proved complexity result in [50, Theorem 2.1] but we use the conjectured complexity to obtain better results.

3 Reduction of $\mathfrak {O}$-orienting problem for special discriminants

We begin with a special case of the problem that depends on the discriminant of the imaginary quadratic order $\mathfrak {O}$. The key ideas from this special case provide a foundation for the general cases we consider in Sect. 4.

We remind the reader that all orientations discussed are primitive orientations. The oracle which we use to solve the Decisional $\mathfrak {O}$-Orienting Problem 2.3 is assumed to be perfect.

Let $d = \prod _{i=1}^r\ell _i$ be a product of small distinct primes. Let $\mathfrak {O}$ denote the maximal order of $K:=\mathbb {Q}(\sqrt{-d})$, so $\Delta _\mathfrak {O}=-d$ if $d\equiv -1 \mod 4$ and $-4d$ otherwise. In particular, $(\Delta _\mathfrak {O}/\ell _i) = 0$ for all $i = 1,\dots ,r$ and $\mathfrak {O}$ is generated by $\omega :=(1+\sqrt{-d})/2$ if $d\equiv -1 \mod 4$ and by $\omega :=\sqrt{-d}$ otherwise. Hence, $\alpha :=\sqrt{-d}$ generates $\mathfrak {O}$ if $d\not \equiv -1 \mod 4$ or $(\mathbb {Z} + 2\mathfrak {O})$ if $d\equiv -1 \mod 4$. We use an oracle which solves Problem 2.3 to find an endomorphism $\varphi$ of E to which we map $\alpha$, thus determining an embedding $\mathfrak {O}\hookrightarrow {{\,\mathrm{{\text {End}}}\,}}(E)$ either by mapping $\alpha =\omega$ to $\varphi$ or $(1+\alpha )/2=\omega$ to $(1+\varphi )/2$. We use the fact that the primes $\ell _i$ are ramified in K.

We walk the $\mathfrak {O}$-oriented $\ell _i$-isogeny volcanoes in order to obtain the endomorphism $\varphi$ on E which is the image of the generator $\omega$ under an embedding $\iota :\mathfrak {O}\hookrightarrow {{\,\mathrm{{\text {End}}}\,}}(E)$. The ideals $\mathfrak {l}_i$ above $\ell _i$ in $\mathfrak {O}$ determine horizontal degree-$\ell _i$ isogenies between $\mathfrak {O}$-oriented curves, beginning and ending with E. To see this, we need the following fact about horizontal isogenies of $\mathfrak {O}$-oriented elliptic curves:

Proposition 3.1

[46, Proposition 4.1] Let $(E,\iota )$ be an $\mathfrak {O}$-oriented supersingular elliptic curve and $\ell$ be a prime number. Then:

• (i) If $\ell$ does not divide the conductor of $\mathfrak {O}$, there is no ascending, $(\Delta _{\mathfrak {O}}/\ell )+1$ horizontal and $\ell -(\Delta _{\mathfrak {O}}/\ell )$ descending $\ell$-isogenies.

• (ii) If $\ell$ divides the conductor of $\mathfrak {O}$, there is one ascending, no horizontal and $\ell$ descending $\ell$-isogenies.

Let $\iota :\mathfrak {O}\hookrightarrow {{\,\mathrm{{\text {End}}}\,}}(E)$ be an orientation and $\varphi :=\iota (\alpha )$. Then $\deg (\varphi )=N(\alpha )=\prod _{i=1}^r\ell _i$ so we may write $\varphi :=\varphi _r\circ \cdots \circ \varphi _1$, where $\varphi _i$ is an isogeny of degree $\ell _i$ for all $i\in \{1,\cdots ,r\}$. For each i, let $\mathfrak {O}\ell _i = (\mathfrak {l}_{i})^2$. The ideals $\mathfrak {l}_i$ determine the horizontal isogenies of $\mathfrak {O}$-oriented curves:

Lemma 3.2

In the setting described above, all of the isogenies $\varphi _i$ in the decomposition of $\varphi$ are horizontal.

Proof

Since $N(\alpha )=\prod _{i=1}^r\ell _i$, we have $\mathfrak {O}\alpha =\prod _{i=1}^r\mathfrak {l}_i$, $\mathfrak {l}_i$ being the unique prime ideal of $\mathfrak {O}$ lying above $\ell _i$ for all $i\in \{1,\cdots , r\}$. Hence, the $\varphi _i$ intervening in the decomposition of $\varphi =\iota (\alpha )$ are horizontal isogenies given by the action of $\mathfrak {l}_i$. $\square$

Now, we describe the steps to obtain an endomorphism $\varphi =\varphi _r\circ \cdots \circ \varphi _1\in {{\,\mathrm{{\text {End}}}\,}}(E)$ which will be the image of $\alpha$. Let $E_0:=E$.

For $i=0$, we find the unique isogeny $\varphi _1:E_0\longrightarrow E_1$ which corresponds to the action of $[\mathfrak {l}_1]$ on $(E_0,\iota )$ by computing each of the $\ell _1 + 1$ outgoing isogenies and querying our oracle to find the one whose codomain $E_1$ is in fact orientable by $\mathfrak {O}$. We continue this process to compute each $\varphi _i$ by using the oracle to find the correct degree-$\ell _i$ isogeny to another $\mathfrak {O}$-orientable curve. At the last step, we compute the degree-$\ell _r$ isogeny from $E_{r-1}\longrightarrow E_r$ and then post-compose with an isomorphism $E_r\cong E_0$: We let $\varphi _r$ denote this composition. See Algorithm 3.1 for the algorithmic description of this process.

A question arises: If $\psi _i: E_{i-1}\longrightarrow E_i'$ is an $\ell _i$-isogeny with $E_i'$ $\mathfrak {O}$-orientable, how do we know that $\psi _i$ is the unique horizontal isogeny $\varphi _i$ given by the action of $\mathfrak {l}_i$ on $(E_{i-1},\iota )$? In fact, $\varphi _i$ and $\psi _i$ could be distinct horizontal isogenies for distinct primitive orientations $(E_{i-1},\iota )\ne (E_{i-1},\iota ')$ (as in Example 3.3). Or $\psi _i$ could even be descending and $E_i'$ $\mathfrak {O}$-oriented by a different orientation than the one induced by $\psi _i$, $(E_i,(\psi _i)_*(\iota ))$.

Example 3.3

Let $p=41$ and $E_0:y^2=x^3+1$ defined over $\mathbb {F}_{41^2}=\mathbb {F}_{41}[\zeta ]$ with $\zeta ^2+\zeta +1=0$. Consider the Frobenius endomorphism $\pi :(x,y)\mapsto (x^p,y^p)$ and the automorphism $\tau : (x,y)\mapsto (\zeta x,y)$. Then $\varphi :=\pi +\tau$ satisfies the polynomial equation $\varphi ^2+\varphi +42=0$ so it defines an orientation of the maximal order $\mathcal {O}_K:=\mathbb {Z}\left[ (1+\sqrt{-167})/2\right]$ of the imaginary quadratic field $K:=\mathbb {Q}(\sqrt{-167})$, mapping $(1+\sqrt{-167})/2$ to $\varphi$.

The prime ideal 2 splits in K so there are two horizontal 2-isogenies and one descending 2-isogeny with domain $E_0$. However, all three of these isogenies have the same codomain $E_1$ (up to isomorphism) with j-invariant $j(E_1)=3$. So $E_1$ is both $\mathcal {O}_K$-oriented and $(\mathbb {Z}+2\mathcal {O}_K)$-oriented.

In order to guarantee a unique horizontal isogeny given by the action of $\mathfrak {l}_i$ on $(E_{i-1},\iota )$, we assume $p>|\Delta _{\mathfrak {O}}|\max _{1\le i\le r}\ell _i$ and prove that there is precisely one (primitive) $\mathfrak {O}$-orientation $\iota$ on $E_{i-1}$, which ensures that there is only one isogeny $\varphi _i$ corresponding to the action of $[\mathfrak {l}_i]$ on $(E_{i-1},\iota )$. We also prove that codomains of descending isogenies are not $\mathfrak {O}$-orientable. These are consequences of [32, Theorem 2’], that we recall below.

Theorem 3.4

[32, Theorem 2’] Let $\mathcal {O}\subset B_{p,\infty }$ be a maximal order in the quaternion algebra ramifying at p and $\infty$. Let ($i\in \{1,2\}$) be two primitive embeddings of orders in the same imaginary quadratic field $K:=\mathbb {Q}\otimes \mathfrak {O}_1=\mathbb {Q}\otimes \mathfrak {O}_2$ of respective discriminants $\Delta _i$. Assume that $j_1(\mathfrak {O}_1)\ne j_2(\mathfrak {O}_2)$. Then $\Delta _1\Delta _2\ge p^2$.

Corollary 3.5

Let $(E,\iota )$ be a (primitively) $\mathfrak {O}$-oriented curve. Then

• (i) If $|\Delta _\mathfrak {O}|<p$, then $\iota$ and $\overline{\iota }: \alpha \longmapsto \iota (\overline{\alpha })$ are the only two (primitive) $\mathfrak {O}$-orientations of E.

• (ii) If $|\Delta _\mathfrak {O}|\ell <p$ and $\psi :(E,\iota )\longrightarrow (E',\iota ')$ is a descending $\ell$-isogeny, then $E'$ is not $\mathfrak {O}$-orientable.

Proof

(i) Let $\iota ':\mathfrak {O}\hookrightarrow {{\,\mathrm{{\text {End}}}\,}}(E)$ be another $\mathfrak {O}$-orientation of E. Since $|\Delta _\mathfrak {O}|<p$, we must have $\iota '(\mathfrak {O})=\iota (\mathfrak {O})$ by Theorem 3.4. Hence, $\iota '^{-1}\circ \iota$ is an automorphism of $\mathfrak {O}$, so it is either the identity or the complex conjugation. The result follows.

(ii) Suppose $E'$ is $\mathfrak {O}$ orientable and let $(E',\iota '')$ be an $\mathfrak {O}$-orientation. Let $\mathfrak {O}':=\mathbb {Z}+\ell \mathfrak {O}$. Then $\psi :(E,\iota )\longrightarrow (E',\iota ')$ being descending, $(E',\iota ')$ is an $\mathfrak {O}'$-orientation and $\iota '(\mathfrak {O}')\ne \iota ''(\mathfrak {O})$, so that $\Delta _{\mathfrak {O}'}\Delta _\mathfrak {O}\ge p^2$ by Theorem 3.4. But $\Delta _{\mathfrak {O}'}\Delta _\mathfrak {O}=\ell ^2\Delta _\mathfrak {O}^2<p^2$ by hypothesis. Contradiction. $\square$

Remark 3.6

Corollary 3.5 holds for any imaginary quadratic order $\mathfrak {O}$, not only the special form we consider in this section.

Assuming $p>|\Delta _{\mathfrak {O}}|\max _{1\le i\le r}\ell _i$, the orientation $\iota :\mathfrak {O}\hookrightarrow {{\,\mathrm{{\text {End}}}\,}}(E)$ is unique up to conjugation, the horizontal $\ell _1$-isogeny $\varphi _1:E_0\longrightarrow E_1$ given by the action of $\mathfrak {l}_1$ is uniquely determined, and it is the only $\ell _1$-isogeny with $\mathfrak {O}$-oriented codomain. In this case, $\varphi _1$ can be distinguished from other $\ell _1$-isogenies by an oracle query. Similarly for each further $i \in \{2,...,r\}$, the isogeny $\varphi _i:E_{i-1}\longrightarrow E_i$ given by the action of $[\mathfrak {l}_i]$ on $(E_{i-1},(\varphi _{i-1}\circ \cdots \circ \varphi _1)_*(\iota ))$ by computing each of the $\ell _i + 1$ isogenies and querying the oracle to find the one whose codomain $E_i$ is orientable by $\mathfrak {O}$. In particular, the isogeny $\varphi _r:E_{r-1}\longrightarrow E_r$ corresponding to the action of $\mathfrak {l}_r$ on $(E_{r-1},(\varphi _{r-1}\circ \cdots \circ \varphi _1)_*(\iota ))$ will have codomain $E_r \cong E_0$. Indeed,

$$\begin{aligned} (E_{r},(\varphi _{r}\circ \cdots \circ \varphi _1)_*(\iota ))=[\mathfrak {l}_1\cdots \mathfrak {l}_r]\cdot (E_0,\iota )=[\alpha \mathfrak {O}]\cdot (E_0,\iota )\cong (E_0,\iota ). \end{aligned}$$

Possibly post-composing with this isomorphism, we have an endomorphism $\varphi =\varphi _r\circ \cdots \circ \varphi _1\in {{\,\mathrm{{\text {End}}}\,}}(E)$ associated to the action of the ideal $\prod _{i=1}^r\mathfrak {l}_i=\alpha \mathfrak {O}$. It follows that $\varphi =\tau \circ \iota (\alpha )$ for some automorphism $\tau \in {{\,\textrm{Aut}\,}}(E)$. We may post-compose $\varphi$ by $\tau \in {{\,\textrm{Aut}\,}}(E)$ until the result has trace zero, as $\alpha$. The trace can be computed in polynomially many isogeny evaluations using Schoof’s algorithm [58, Sect. 5].

Remark 3.7

(Isomorphisms) Assuming we are working with elliptic curves in Weierstrass form, all isomorphism formulae are known. To find an isomorphism $\beta : E_r \longrightarrow E_0$, we check the codomain formula for each isomorphism from $E_r$ until $E_0$ is found.

There are additional automorphisms in the two special cases of $j=1728$ and $j=0$ [2, Figure 3.1, Sect. 6]. At each step $\varphi _i:E_{i-1}\rightarrow E_i$ where $j(E_i) = 0$ or 1728, we must decide whether or not to post-compose with these automorphisms. The automorphisms $[\pm 1]$ will not affect the resulting trace, but we must check one nontrivial automorphism for $j = 1728$ and two for $j = 0$. This can be done after the algorithm is completed, as the oracle calls will remain unaffected.

The additional running time of choosing isomorphisms can be bounded by a constant, so does not contribute to the overall complexity.

Example 3.8

Let $p = 83$ and $\mathfrak {O} = \mathbb {Z}[\sqrt{-21}]$, the ring of integers of $K = \mathbb {Q}(\sqrt{-21})$. We see p is inert in $\mathfrak {O}$ so K embeds into the quaternion algebra ${{\,\mathrm{{\text {End}}}\,}}^0(E)$ for any supersingular E over $\overline{\mathbb {F}_p}$. Now, let $E/\mathbb {F}_p^2$ be the $\mathfrak {O}$-oriented curve $y^2 = x^3 + x$, we find the orientation by finding an endomorphism $\omega$ with $N(\omega ) = 21 = 3 \cdot 7$ and $Tr(\omega ) = 0$. From E we pick a 3-isogeny to $y^2 = x^3 + 32x + 38\sqrt{-1}$, this is also $\mathfrak {O}$-oriented. Then we pick a horizontal 7-isogeny which has codomain $y^2 = x^3 + 26x$. This curve is isomorphic to E. By composing maps we get $\omega : E \longrightarrow E$. Finally we notice $\omega \ne - {\tilde{\omega }}$ so the endomorphism has a non-zero trace. But by post-composing with an automorphism $\iota$ on E, we get a trace-zero endomorphism of degree 21.

If $\alpha$ is a generator of $\mathfrak {O}$ ($d\not \equiv -1 \mod 4$), then $\varphi$ determines an $\mathfrak {O}$-orientation and we are done. Otherwise, $\mathbb {Z}[\alpha ]$ has index 2 in $\mathfrak {O}$ ($d\equiv -1 \mod 4$), and $\omega =(1+\alpha )/2$ generates $\mathfrak {O}$. Then $\varphi$ determines an imprimitive $\mathbb {Z}[\alpha ]$-orientation of E. This orientation cannot be primitive, otherwise, we would have $\Delta _\mathfrak {O}\Delta _{\mathbb {Z}[\alpha ]}\ge p^2$ i.e. $4\Delta _\mathfrak {O}^2\ge p^2$, which is a contradiction since we assumed that $p>|\Delta _{\mathfrak {O}}|\max _{1\le i\le r}\ell _i\ge 2|\Delta _{\mathfrak {O}}|$. It follows that $(\varphi +1)/2$ is well defined and induces an $\mathfrak {O}$-orientation on E: $\omega =(\alpha +1)/2\longmapsto (\varphi +1)/2$.

Remark 3.9

(Efficient representation) Knowing how to evaluate $\varphi$ (as the composition $\varphi _r\circ \cdots \circ \varphi _1$), we efficiently evaluate $(\varphi +1)/2$ as follows: if $P\in E(\mathbb {F}_{p^k})$, we find $P'\in E(\mathbb {F}_{p^{2k}})$ such that $[2]P'=P$ and compute $\varphi (P')+P'$. Assuming the $\ell _i$ are polynomial in $\log (d)$, the list of isogenies $(\varphi _r, \cdots , \varphi _1)$ defines an efficient representation of both $\varphi$ and $(\varphi +1)/2$.

We summarize all the steps to determine an $\mathfrak {O}$-orientation in Algorithm 3.1.

Theorem 3.10

Let $d:=\prod _{i=1}^r \ell _i$ be a product of small distinct primes, $\mathfrak {O}$ be the maximal order of $\mathbb {Q}(\sqrt{-d})$ and $p>|\Delta _\mathfrak {O}|\max _{1\le i\le r}\ell _i$. Then, over $\mathbb {F}_{p^2}$, Algorithm 3.1 reduces the $\mathfrak {O}$-Orienting Problem (Problem 2.4) to the Decisional $\mathfrak {O}$-Orienting Problem (Problem 2.3) in polynomial time in $\log (p)$ and $\max _{1\le i\le r}\ell _i$.

Proof

We justified above that this algorithm terminates and is correct. For all $i\in \{1,...,r\}$, this algorithm computes the $\ell _i+1$ curves which are $\ell _i$-isogenous to $E_{i-1}$, which costs ${\tilde{O}}(\ell _i^2\log (p))$ operations over $\mathbb {F}_{p^2}$ by Sect. 2.4. It calls the oracle $\ell _i+1$ times and computes one $\ell _i$-isogeny between $j(E_{i-1})$ and $j(E_{i})$, which costs on average ${\tilde{O}}(\ell _i^2\log (p))$ operations over $\mathbb {F}_{p^2}$ by Sect. 2.5. The number of isomorphisms $\beta :E_r\longrightarrow E_0$ is O(1). Using [58, Sect. 5], we compute the trace of $\beta \circ \varphi _r\circ \cdots \circ \varphi _1$ on line 18 of Algorithm 3.1 in polynomial time in $\log (p)$, $r=O(\log (p))$ and $\max _{1\le i\le r}\ell _i$. Operations over $\mathbb {F}_{p^2}$ have a polynomial cost in $\log (p)$ in terms of bit operations. Hence, the total cost is polynomial in $\log (p)$ and $\max _{1\le i\le r}\ell _i$. $\square$

4 Solving the $\mathfrak {O}$-orienting problem with a decision oracle

We remind the reader that all orientations discussed are primitive orientations. The oracle which we use to solve the Decisional $\mathfrak {O}$-Orienting Problem 2.3 is assumed to be perfect.

4.1 Description of the algorithms

Let $\mathfrak {O}$ be an imaginary quadratic order with general discriminant $\Delta _\mathfrak {O}$. Given access to an oracle for the Decisional $\mathfrak {O}$-Orienting Problem 2.3, we solve the $\mathfrak {O}$-Orienting Problem 2.4 finding a an $\mathfrak {O}$-orientation $\iota : \mathfrak {O}\hookrightarrow {{\,\mathrm{{\text {End}}}\,}}(E)$ of any given supersingular elliptic curve $E/\mathbb {F}_{p^2}$ if it exists, and returning Null otherwise.

The idea is similar the case of special discriminant considered in Sect. 3. We compute an endomorphism corresponding to a generator of $\mathfrak {O}$ as a chain of horizontal isogenies of small degrees. However, two difficulties arise. First, the canonical generator $\omega :=(s+\sqrt{\Delta _\mathfrak {O}})/2$ with $s:=\Delta _\mathfrak {O} \mod 2$ of $\mathfrak {O}$ is not smooth in general. We have to find another smooth generator $\theta$ of $\mathfrak {O}$. Second, if we denote $\varphi :=\iota (\theta )$ and decompose $\varphi :=\varphi _r\circ \cdots \circ \varphi _1$ as a product of horizontal isogenies of degrees $\ell _i, \cdots , \ell _r$ respectively, we may not be able to find the $\varphi _i$ simply by using the oracle as in Sect. 3. We are no longer guaranteed that $\ell _i \mid \Delta _\mathfrak {O}$, so there may be $1+(\Delta _\mathfrak {O}/\ell _i)=2$ horizontal isogenies of degree $\ell _i$ from a $\mathfrak {O}$-oriented elliptic curve. To search for $\varphi$, starting at root E we fill a binary tree whose nodes are $\mathfrak {O}$-oriented elliptic curves and edges are horizontal isogenies. We call such a tree an $\mathfrak {O}$-oriented $(\ell _1,\cdots , \ell _r)$-isogeny tree, see Definition 4.1. The endomorphism $\varphi$ is a branch of this tree with leaf E.

Definition 4.1

An $\mathfrak {O}$-oriented $(\ell _1,\cdots , \ell _r)$-isogeny tree is a binary tree of height r whose nodes are (primitively) $\mathfrak {O}$-oriented supersingular elliptic curves and such that every node $E_{i-1}$ of depth $i\in \{1,\cdots ,r\}$ has children that are horizontally $\ell _{i}$-isogenous to $E_{i-1}$.

To optimize the tree search, we propose a meet-in-the middle strategy where two half-depth such trees are computed starting at E instead of a single one:

• (1) Find a generator $\theta$ of $\mathfrak {O}$ of B-smooth norm $N(\theta ):=\prod _{i=1}^r\ell _i$.

• (2) Starting at E, compute $\mathfrak {O}$-oriented $(\ell _1,\cdots , \ell _{s})$-isogeny tree $\mathcal {T}_1$ and an $\mathfrak {O}$-oriented $(\ell _{s+1},\cdots , \ell _{r})$-isogeny tree $\mathcal {T}_2$ (with $s\simeq r/2$).

• (3) Find a matching leaf in $\mathcal {T}_1$ and $\mathcal {T}_2$.

• (4) Extract the corresponding endomorphism $\varphi \in {{\,\mathrm{{\text {End}}}\,}}(E)$.

• (5) Infer from $\varphi =\iota (\theta )$ an efficient representation of the canonical generator $\varphi _0:=\iota (\omega )$ (in the sense of Definition 2.8).

We explain each step in detail in the following paragraphs.

4.1.1 Finding a smooth norm generator

Let $\mathfrak {O}$ be an imaginary quadratic order and $\omega$ be a generator. We want to find another generator $\theta$ of $\mathfrak {O}$ with smooth norm $N(\theta )=\prod _{i=1}^r\ell _i$. The computation of $\varphi =\varphi _r\circ \cdots \circ \varphi _1$ associated to $\theta$ is exponential in the $\ell _i$ and r, so we require the $\ell _i$ and $2^r$ to be subexponential in $\log (|\Delta _\mathfrak {O}|)$. For technical reasons (see Lemma 4.3), $N(\theta )$ should also be non-square and coprime to $\Delta _\mathfrak {O}$. In summary, we look for a generator $\theta$ of ns-$(B,r_m,\Delta _\mathfrak {O})$-smooth norm, in the sense of Definition 4.2, with B and $2^{r_m}$ subexponential in $\log (|\Delta _\mathfrak {O}|)$.

Definition 4.2

An integer $N\in \mathbb {N}$ is $(B,r_m,d)$-smooth when its decomposition into prime factors $N=\prod _{i=1}^r\ell _i$ satisfies $r\le r_{m}$, $\ell _i\le B$, and $\ell _i\not \mid d$ for all $i\in \{1,\cdots , r\}$. We say that N is ns-$(B,r_m,d)$ -smooth when it is $(B,r_m,d)$-smooth and not a square.

We look for $\theta$ of ns-$(B,r_m,\Delta _\mathfrak {O})$-smooth norm the form $\theta :=a+\omega$ with $a\in \mathbb {Z}$ to be determined. There is no better known method to find a than to sample a at random and to test whether $N(a+\omega )$ is ns-$(B,r_m,\Delta _\mathfrak {O})$-smooth. To make sure $N(a+\omega )$ is close to $N(\omega )$, we sample $a\in \{-\lfloor \sqrt{N(\omega )}\rfloor , \cdots , \lfloor \sqrt{N(\omega )}\rfloor \}$. We have $N(\omega )=(|\Delta _{\mathfrak {O}}|+t^2)/4$ with $t:={{\,\textrm{Tr}\,}}(\omega )\in \{0,1\}$. It follows that:

$$\begin{aligned} \frac{|\Delta _{\mathfrak {O}}|}{4}=N(-t/2+\omega )\le N(a+\omega )\le N(-\sqrt{N(\omega )}+\omega )\le |\Delta _{\mathfrak {O}}| \end{aligned}$$

Since B is subexponential in $\log (|\Delta _\mathfrak {O}|)$, the optimal known way to test the B-smoothness of $N(a+\omega )$ is the method introduced in Sect. 2.7 using ECM with time complexity $L_B(1/2,\sqrt{2})$. Algorithm 4.1 presenting the search for $\theta =a+\omega$ follows.

4.1.2 Filling the $\mathfrak {O}$-oriented isogeny trees

Let $E/\mathbb {F}_{p^2}$ be an $\mathfrak {O}$-orientable elliptic curve and splitting primes $\ell _1,\dots ,\ell _s\le B$. We explain here how to fill $\mathcal {T}$, the $\mathfrak {O}$-oriented $(\ell _1,\dots ,\ell _s)$-isogeny tree starting at E.

We assume $p>B|\Delta _\mathfrak {O}|$ so any $\mathfrak {O}$-orientable curve admits a unique $\mathfrak {O}$-orientation up to conjugation by Corollary 3.5(i). Hence, every node of $\mathcal {T}$ can be represented by j-invariant (the root $E_0:=E$ included). If $E_{i-1}$ is a node of depth $i\in \{1,\cdots ,s\}$ of $\mathcal {T}$, its children $E_{i,1}$ and $E_{i,2}$ are the only two $\mathfrak {O}$-orientable curves that are $\ell _i$-isogenous to $E_i$, given by the action of ideals $\mathfrak {l}_i,$ $\overline{\mathfrak {l}_i}$ above $\ell _i$. As in Sect. 3, to find $E_{i,1}$ and $E_{i,2}$ we compute the codomain j-invariants of all degree-$\ell _i$ isogenies $E_i\longrightarrow E'$ and apply the decision oracle to see which are $\mathfrak {O}$-orientable. Determining such j-invariants can be done using modular polynomials in ${\tilde{O}}(\ell _i^2\log (p))$ operations over $\mathbb {F}_{p^2}$, as in Sect. 2.4. The tree filling algorithm (Algorithm 4.2) follows.

4.1.3 From a tree match to a generating endomorphism

Assume we have found $\theta$, a generator of $\mathfrak {O}$ with ns-$(B,r_m,\Delta _\mathfrak {O})$-smooth norm $N(\theta )=\prod _{i=1}^r\ell _i$. Let $\iota : \mathfrak {O}\hookrightarrow {{\,\mathrm{{\text {End}}}\,}}(E)$ denote the orientation with $\varphi :=\iota (\theta )$. Then, we may decompose $\varphi :=\varphi _r\circ \cdots \circ \varphi _1$, where $\varphi _i$ is an $\ell _i$-isogeny for all $i\in \{1,\cdots ,r\}$.

Lemma 4.3

Assuming $N(\theta )=\deg (\varphi )$ is coprime with $\Delta _\mathfrak {O}$, all the isogenies $\varphi _i$ in the decomposition of $\varphi$ are horizontal.

Proof

Let $\mathfrak {O}_1$ be the order associated to $\iota _1:=(\varphi _1)_*(\iota )$ i.e. such that $\iota _1(\mathfrak {O}_1)={{\,\mathrm{{\text {End}}}\,}}(\varphi _1(E))\cap \iota _1(K)$. Since $N(\theta )=\deg (\varphi )$ is coprime with $\Delta _\mathfrak {O}$, $\ell _1$ does not divide the conductor of $\mathfrak {O}$ so $\varphi _1$ is horizontal or descending by Proposition 3.1(i). It follows that $\mathfrak {O}_1\subseteq \mathfrak {O}$. Besides,

$$\begin{aligned} \iota _1(\theta )=\frac{1}{\ell _1}\varphi _1\circ \iota (\theta )\circ \widehat{\varphi _1}=\varphi _1\circ \varphi _r\circ \cdots \circ \varphi _2\in {{\,\mathrm{{\text {End}}}\,}}(\varphi _1(E)). \end{aligned}$$

Hence, $\theta \in \mathfrak {O}_1$ so $\mathfrak {O}\subseteq \mathfrak {O}$ since $\theta$ generates $\mathfrak {O}$. Consequently, $\mathfrak {O}_1=\mathfrak {O}$ and $\varphi _1$ is horizontal. We obtain easily by induction that $\varphi _2, \cdots , \varphi _r$ are also horizontal.

$\square$

Since the $\varphi _i$ are horizontal, we may use $\mathfrak {O}$-oriented isogeny trees to find these isogenies. Let $s:=\lfloor r/2\rfloor$, $\mathcal {T}_1$ the $\mathfrak {O}$-oriented $(\ell _1,\cdots , \ell _{s})$-isogeny tree starting at $E_0:=E$ and $\mathcal {T}_2$ the $\mathfrak {O}$-oriented $(\ell _{s+1},\cdots , \ell _{r})$-isogeny tree starting at E. Assume we have found a common leaf $E_s$ in $\mathcal {T}_1$ and $\mathcal {T}_2$. The branch of $\mathcal {T}_1$ of leaf $E_s$ is a chain of horizontal $\ell _i$-isogenies $\psi _i:E_{i-1}\longrightarrow E_i$ for $i\in \{1,\cdots , s\}$ and the branch of $\mathcal {T}_2$ of leaf $E_s$ (taken depth first) is a chain of horizontal $\ell _i$-isogenies $\psi _i:E_{i-1}\longrightarrow E_i$ for $i\in \{s+1,\cdots , r\}$, with $E_r=E_0=E$. The isogeny $\psi :=\psi _r\circ \cdots \circ \psi _1$ is a horizontal isogeny of degree $\prod _{i=1}^r \ell _i=N(\theta )$, but we do not know a priori if $\psi =\varphi =\iota (\theta )$.

Lemma 4.4

Let $(E_0,\iota )$ be an $\mathfrak {O}$-oriented supersingular elliptic curve and $\psi \in {{\,\mathrm{{\text {End}}}\,}}(E_0)$ a horizontal endomorphism of degree coprime to p. Then, there exists $\alpha \in \mathfrak {O}$ such that $\psi =\iota (\alpha )$.

Proof

Since $\psi$ is horizontal, $\psi _*(\iota )$ defines an $\mathfrak {O}$-orientation on $E_0$, like $\iota$. Since $|\Delta _{\mathfrak {O}}|<p$, by Theorem 3.4, we must have $\psi _*(\iota )(\mathfrak {O})=\iota (\mathfrak {O})$, so that $\psi _*(\iota )=\iota$ or $\psi _*(\iota )=\overline{\iota }$, where $\overline{\iota }(\alpha ):=\iota (\overline{\alpha })$ for all $\alpha \in K$.

If $\psi _*(\iota )=\iota$, $\psi$ commutes with $\iota (K)$ ($K:=\mathfrak {O}\otimes _\mathbb {Z}\mathbb {Q}$), so $\psi \in \iota (K)\cap {{\,\mathrm{{\text {End}}}\,}}(E_0)=\iota (\mathfrak {O})$ and $\psi =\iota (\alpha )$ for some $\alpha \in \mathfrak {O}$.

Suppose $\psi _*(\iota )=\overline{\iota }$. As Onuki proved in [46, Proposition 3.3 and Theorem 3.4], $(E,\overline{\iota })$ and $(E^{(p)},(\pi _p)_*(\iota ))$ are in the same orbit of the action of ${{\,\mathrm{{\text {Cl}}}\,}}(\mathfrak {O})$ on the set ${{\,\mathrm{{\text {SS}}}\,}}_{\mathfrak {O}}^{pr}(p)$ of (primitively) $\mathfrak {O}$-oriented supersingular elliptic curves over $\mathbb {F}_{p^2}$ ($\pi _p:E\longrightarrow E^{(p)}$ being the p-Frobenius isogeny). Hence, there exists an ideal $\mathfrak {b}\subset \mathfrak {O}$ of norm coprime with p, such that $(E,\overline{\iota })=\mathfrak {b}\cdot (E^{(p)},(\pi _p)_*(\iota ))$, so that $\psi _*(\iota )=\overline{\iota }=(\varphi _{\mathfrak {b}}\circ \pi _p)_*(\iota )$. Consequently, $\widehat{\pi _p}\circ \widehat{\varphi _{\mathfrak {b}}}\circ \psi$ commutes with $\iota (K)$, so there exists $\alpha \in \mathfrak {O}$ such that $\widehat{\pi _p}\circ \widehat{\varphi _{\mathfrak {b}}}\circ \psi =\iota (\alpha )$ and $p \mid N(\alpha )$. Since ${{\,\mathrm{{\text {SS}}}\,}}_{\mathfrak {O}}^{pr}(p)$ is not empty (it contains $E_0$), p is either inert or ramified in K by [46, Proposition 3.2]. The prime p cannot be ramified, otherwise we would have $p \mid \Delta _{\mathfrak {O}}$, so $|\Delta _{\mathfrak {O}}|\ge p$. If p is inert and $p \mid N(\alpha )$, then $p \mid \alpha$ so that $p^2 \mid N(\alpha )$ and $p \mid \deg (\psi )N(\mathfrak {b})$. Since $N(\mathfrak {b})$ is coprime with p, $p \mid \deg (\psi )$ which contradicts our assumption.

It follows that $\psi _*(\iota )=\iota$. $\square$

Lemma 4.5

Let $\theta :=a+\omega \in \mathfrak {O}$, with $a\in \mathbb {Z}$, $|a|\le \sqrt{N(\omega )}$. Assume $N(\theta )$ is not a square and $\Delta _{\mathfrak {O}}\ne -3, -4$. The only $\alpha \in \mathfrak {O}$ such that $N(\alpha )=N(\theta )$ are $\alpha = \pm \theta ,\pm \overline{\theta }$.

Proof

Let $\alpha :=b+c\omega \in \mathfrak {O}$ with $b,c\in \mathbb {Z}$ such that $N(\alpha )=N(\theta )$. Then

$$\begin{aligned} b^2+tbc+c^2n=N(\alpha )=N(\theta )=a^2+ta+n, \end{aligned}$$

(7)

with $t:={{\,\textrm{Tr}\,}}(\omega )\in \{0,1\}$ and $n:=N(\omega )=(t^2+|\Delta _{\mathfrak {O}}|)/4$.

If $c^2>1$, the minimum value of $b^2+tbc+c^2n$ is reached when $b=-ct/2$, so

$$\begin{aligned} b^2+tbc+c^2n\ge \left( n-\frac{t^2}{4}\right) c^2=\frac{|\Delta _{\mathfrak {O}}|c^2}{4}\ge |\Delta _{\mathfrak {O}}| . \end{aligned}$$

(8)

But, by (7) and since $|a|\le \sqrt{n}$, we have:

$$\begin{aligned} N(\theta )\le 2n+t\sqrt{n}<|\Delta _{\mathfrak {O}}| \end{aligned}$$

which contradicts (8).

So $c^2\le 1$ and $c\in \{0,\pm 1\}$. If $c=0$, then $N(\theta )$ is a square which is not possible. If $c=1$, then (7) becomes $(a-b)(a+b+t)=0$ and we have $b=a$ or $b=-a-t$. If $c=-1$, then (7) becomes $(a+b)(a-b+t)=0$ and we have $b=a$ or $b=a-t$. Hence, $(b,c)\in \{(a,1),(-a-t,1),(-a,-1),(t+a,-1)\}$ and $\alpha \in \{\pm \theta ,\pm \overline{\theta }\}$. $\square$

Remark 4.6

The cases of $\Delta _{\mathfrak {O}}=-3, -4$ are excluded from this lemma because in those cases, we have a very simple way to find the orientation:

If $\Delta _\mathfrak {O}=-3$, then $\mathfrak {O}=\mathbb {Z}[\zeta _3]$, with $\zeta _3:=(1+\sqrt{-3})/2$ so any elliptic curve E that is $\mathfrak {O}$-oriented contains an automorphism of order 3. By [60, Theorem III.10.1], we must have $j(E)=0$, so E is given by the Weierstrass equation $y^2=x^3+1$ (up to isomorphism), and $\zeta _3$ corresponds to the automorphism $(x,y)\in E\longmapsto (\xi _3 x, y)\in E$, where $\xi _3$ is a primitive third root of unity in $\mathbb {F}_{p^2}$.

Similarly, if $\Delta _\mathfrak {O}=-4$, then $\mathfrak {O}=\mathbb {Z}[i]$ so any elliptic curve E that is $\mathfrak {O}$-oriented contains an automorphism of order 4. By [60, Theorem III.10.1], we must have $j(E)=1728$, so E is given by the Weierstrass equation $y^2=x^3+x$ (up to isomorphism), and i corresponds to the automorphism $(x,y)\in E\longmapsto (x, a y)\in E$, where a is a square root of $-1$ in $\mathbb {F}_{p^2}$.

By Lemmas 4.4 and 4.5, we must have $\psi =\pm \iota (\theta )=\pm \varphi$ or $\psi =\pm \iota (\overline{\theta })=\pm {\widehat{\varphi }}$. The sign can be determined by computing ${{\,\textrm{Tr}\,}}(\psi )$ using a generalization of Schoof’s algorithm [58, Sect. 5] and comparing to ${{\,\textrm{Tr}\,}}(\theta )$. We recover $\iota$ or $\overline{\iota }: \mathfrak {O}\hookrightarrow {{\,\mathrm{{\text {End}}}\,}}(E)$ by mapping $\theta$ to $\pm \psi$.

However, the factors $\psi _i$ of $\psi$ have subexponential degree so they do not provide an efficient representation of $\psi$ (enabling to evaluate $\psi$ in polynomial time for instance). We apply Algorithm 2.1 to get an efficient representation of $\iota (\omega )$ or $\iota (\overline{\omega })=\pm \psi -[a]$. The search to decision reduction Algorithm 4.3 follows.

For efficiency, only j-invariants are stored in the trees and not the $\ell _i$-isogenies relating them so we use the method of Sect. 2.5 to recover them in time ${\tilde{O}}(\ell _i^2\log (p))$.

4.2 Complexity analysis

In the following, we shall count arithmetic operations over various rings ($\mathbb {Z}$, $\mathbb {F}_{p^2}$ and extensions of $\mathbb {F}_{p^2}$). To provide a unified way to present time complexity results, we shall count bit operations. We shall say an algorithm terminates in time O(f(n)) when it uses at most O(f(n)) bit operations. We shall denote by M(p) the maximal number of bit operations required for an operation over $\mathbb {F}_p$ (addition, substraction, multiplication, inversion or sampling of one element).

4.2.1 Complexity of the smooth norm search (Algorithm 4.1)

To estimate the complexity of Algorithm 4.1, we need to determine the probability that $N(a+\omega )$ is ns-$(B,r_m,\Delta _\mathfrak {O})$-smooth. We have proven results on the distribution of B-smooth integers among random integers but not for random values of quadratic integer polynomials. For that reason, we introduce the following heuristic assumption.

Heuristic 4.7

Let $f:=X^2-tX+N\in \mathbb {Z}[X]$, a following the uniform distribution in $\{-\lfloor \sqrt{N}\rfloor ,\cdots ,\lfloor \sqrt{N}\rfloor \}$, and b following the uniform distribution in $\{0,\cdots , N\}$. Then there exist constants $C>0$, $c>0$ such that for all $N\in \mathbb {Z}_{>0}$, $\log ^c(N)\le B\le N$, $\log (N)/log(B)\le r\le \log _2(N)$ and $d\le 4N$, we have:

$$\begin{aligned} \mathbb {P}(f(a) \hbox { is ns}-(B,r,d)-\hbox {smooth})\ge C\cdot \mathbb {P}(b \hbox { is ns}-(B,r,d)-\hbox {smooth}). \end{aligned}$$

This heuristic assumption is supported by an estimate on the probability for a given polynomial to take B-smooth values, which is very similar to proven estimates on the B-smoothness probability of integers [17, Theorem 1]. Such an estimate on polynomials has been proved in [42, Theorem 1.1] under a dual hypothesis on the number of prime values of polynomials when B is in a very tight range. It has been conjectured [29, Equation 1.20] that this result holds for broader values of B.

Lemma 4.8

Let $\Psi _{r}(x,y,d)$ denote the number of (y, r, d)-smooth integers $\le x$:

$$\begin{aligned} \Psi _r(x,y,d)=\#\left\{ n\le x\;\bigg |\; n=\prod _{i=1}^s\ell _i, \ s\le r \text{ and } \forall \, 1\le i\le s, \ \ell _i\le y \hbox { and } \ell _i\not \mid d \right\} . \end{aligned}$$

Then, if $r\ge \log (x)/\log (y)$,

$$\begin{aligned} \Psi _r(x,y,d)\ge \left( {\begin{array}{c}\pi (y)-\pi (z)-\omega _y(d)+\lfloor \frac{\log (x)}{\log (y)}\rfloor \\ \pi (y)-\pi (z)-\omega _y(d)\end{array}}\right) , \end{aligned}$$

with $z:=x^{1/r}$, $\pi (t)$ the number of prime numbers $\le t$ and $\omega _y(d)$ the number of distinct prime divisors $\le y$ of d.

Proof

The proof follows from [17, §2]. We have the following inequalities (following from set inclusions):

$$\begin{aligned} \Psi _r(x,y,d)&=\#\Bigg \{(\alpha _\ell )_{\begin{array}{c} \ell \le y\\ \ell \not \mid d \end{array}}\in \mathbb {N}^{\pi (y)-\omega _y(d)}\;\Bigg |\; \#\{ \ell \le y, \ell \not \mid d\mid \alpha _\ell \ne 0\}\le r\\&\qquad \hbox {and} \ \sum _{\ell \le y}\alpha _\ell \log (\ell )\le \log (x) \Bigg \}\\&\ge \#\left\{ (\alpha _\ell )_{\begin{array}{c} z<\ell \le y\\ \ell \not \mid d \end{array}}\in \mathbb {N}^{\pi (y)-\pi (z)-\omega _y(d)}\;\bigg |\; \sum _{\begin{array}{c} z<\ell \le y\\ \ell \not \mid d \end{array}}\alpha _\ell \log (\ell )\le \log (x) \right\} \\&\ge \#\left\{ (\alpha _\ell )_{\begin{array}{c} z<\ell \le y\\ \ell \not \mid d \end{array}}\in \mathbb {N}^{\pi (y)-\pi (z)-\omega _y(d)}\;\bigg |\; \sum _{\begin{array}{c} z<\ell \le y\\ \ell \not \mid d \end{array}}\alpha _\ell \le \left\lfloor \frac{\log (x)}{\log (y)} \right\rfloor \right\} . \end{aligned}$$

To conclude, we compute the cardinality of

$$\begin{aligned} S(k,n):=\left\{ (\alpha _1,\cdots ,\alpha _k)\in \mathbb {N}^k\;\bigg |\;\sum _{i=1}^k \alpha _i\le n\right\} \end{aligned}$$

for $k,n\in \mathbb {Z}_{>0}$ and apply it to the last set in the inequalities above. The set S(k, n) is in bijection with the subsets of k elements in $\{1,\cdots ,n+k\}$, via the maps:

$$\begin{aligned} \{s_1<\cdots <s_k\}\longmapsto (s_1-1,s_2-s_1-1,\cdots ,s_k-s_{k-1}-1)\\(\alpha _1,\cdots ,\alpha _k)\longmapsto \{\alpha _1+1,\alpha _1+\alpha _2+2,\cdots ,\alpha _1+\cdots +\alpha _k+k\}. \end{aligned}$$

It follows that

$$\begin{aligned} \#S(k,n)=\left( {\begin{array}{c}n+k\\ k\end{array}}\right) . \end{aligned}$$

$\square$

Lemma 4.9

Let $\psi (x,y)$ be the number of y-smooth numbers $\le x$. Assume that $\log (y)\ll \log (x)$ and $\log (y)\gg \log \log (x)$. Then

$$\begin{aligned} \log \left( \frac{\psi (x,y)}{x}\right) \sim -\frac{\log (x)\log \log (x)}{\log (y)}. \end{aligned}$$

Proof

It follows from [17, Theorem 1] that for all $2< y\le x$:

$$\begin{aligned} \log \psi (x,y)=\left( \log \left( 1+\frac{y}{\log (x)}\right) \frac{\log (x)}{\log (y)} +\log \left( 1+\frac{\log (x)}{y}\right) \frac{y}{\log (y)}\right) \nonumber \\ \cdot \left( 1+O\left( \frac{1}{\log (y)}\right) +O\left( \frac{1}{\log \log (x)}\right) +O\left( \left( 1+\frac{\log (x)}{\log (y)}\right) ^{-1}\right) \right) . \end{aligned}$$

(9)

Since $\log (y)\gg \log \log (x)$, we have $y\gg \log ^2(x)$, so that

$$\begin{aligned} \log \left( 1+\frac{y}{\log (x)}\right) \frac{\log (x)}{\log (y)}&=\left( \log \left( \frac{y}{\log (x)}\right) +\log \left( 1+\frac{\log (x)}{y}\right) \right) \frac{\log (x)}{\log (y)}\\&=\log (x)-\frac{\log (x)\log \log (x)}{\log (y)}+\frac{\log ^2(x)}{y\log (y)}(1+o(1))\\&=\log (x)-\frac{\log (x)\log \log (x)}{\log (y)}+o(1) \end{aligned}$$

and

$$\begin{aligned} \log \left( 1+\frac{\log (x)}{y}\right) \frac{y}{\log (y)}=\frac{\log (x)}{\log (y)}(1+o(1)). \end{aligned}$$

It follows finally by 9 that

$$\begin{aligned} \log \left( \frac{\psi (x,y)}{x}\right) \sim -\frac{\log (x)\log \log (x)}{\log (y)}. \end{aligned}$$

$\square$

Lemma 4.10

Let $\psi _r^*(x,y,d)$ be the number of ns-(y, r, d)-smooth integers $\le x$. Let $z:=x^{1/r}$ and $\varepsilon :=z/y$. Assume that $r\ge \log (x)/\log (y)$, $d=O(x)$, $\log (y)\ll \log (x)$, $\log (y)\gg \log \log (x)$ and $\log (1-\varepsilon )\ll \log \log (y)$. Then

$$\begin{aligned} \log \left( \frac{\psi _r^*(x,y,d)}{x}\right) \sim -\frac{\log (x)\log \log (x)}{\log (y)} \end{aligned}$$

as $x,y,r,d\longrightarrow +\infty$.

Proof

The number of squares smaller than x being bounded by $\sqrt{x}$, we have

$$\begin{aligned}\psi _r^*(x,y,d)\ge \psi _r(x,y,d)-\sqrt{x}.\end{aligned}$$

And by lemma 4.8,

$$\begin{aligned}\psi _r(x,y,d)\ge \left( {\begin{array}{c}n+k\\ k\end{array}}\right) \end{aligned}$$

with $k:=\pi (x)-\pi (z)-\omega _y(d)$, $n:=\lfloor \log (x)/\log (y)\rfloor$, so that

$$\begin{aligned} \log \psi _r(x,y,d)\ge \log \left( {\begin{array}{c}n+k\\ k\end{array}}\right) =(n+k)\log (n+k)-k\log (k)-n\log (n)\nonumber \\ +\frac{1}{2}\log (n+k)-\frac{1}{2}\log (k)-\frac{1}{2}\log (n)+O(1). \end{aligned}$$

(10)

We have $\pi (t)=t/\log (t)+O(t/\log (t)^2)$ as $t\longrightarrow +\infty$ and

$$\begin{aligned} \omega _y(d)=O(\log (d))=O(\log (x))=o(y/\log ^2(y)), \end{aligned}$$

since $y\gg \log ^\alpha (x)$ for all $\alpha >0$, because $\log (y)\gg \log \log (x)$. It follows that

$$\begin{aligned} k=\pi (x)-\pi (z)-\omega _y(d)=\frac{(1-\varepsilon )y}{\log (y)}+O\left( \frac{y}{\log (y)^2}\right) . \end{aligned}$$

Besides, we have $\log (1-\varepsilon )\ll \log \log (y)$, so

$$\begin{aligned} \log ((1-\varepsilon )\log (y))=\log (1-\varepsilon )+\log \log (y)\sim \log \log (y) \end{aligned}$$

so $\log ((1-\varepsilon )\log (y))\longrightarrow +\infty$ and $(1-\varepsilon )\log (y)\longrightarrow +\infty$ i.e. $1-\varepsilon \gg 1/\log (y)$. It follows that

$$\begin{aligned} k\sim \frac{(1-\varepsilon )y}{\log (y)}. \end{aligned}$$

Furthermore, $\log (y)\gg \log \log (x)$ so

$$\begin{aligned} \log \left( \frac{y}{\log ^2(x)}\right) =\log (y)-2\log \log (x)\sim \log (y) \end{aligned}$$

so $y/\log ^2(x)\longrightarrow +\infty$ and $y\gg \log ^2(x)$. It follows that

$$\begin{aligned} \frac{n^2}{k}\sim \frac{\log ^2(x)}{(1-\varepsilon )y\log (y)}=o\left( \frac{\log ^2(x)}{y}\right) =o(1), \end{aligned}$$

so 10 becomes

$$\begin{aligned} \log \psi _r(x,y,d)&\ge n\log \left( \frac{k}{n}\right) +n-\frac{1}{2}\log (n)+O(1)\\&=\frac{\log (x)}{\log (y)}\log \left( \frac{(1-\varepsilon )y}{\log (x)}\right) +\frac{\log (x)}{\log (y)}-\frac{1}{2}\log \left( \frac{\log (x)}{\log (y)}\right) +O(1)\\&=\log (x)-\frac{\log (x)\log \log (x)}{\log (y)}+o\left( \frac{\log (x)\log \log (y)}{\log (y)}\right) \\&\qquad (\hbox {since } \log (1-\varepsilon )\ll \log \log (y))\\&=\log (x)-\frac{\log (x)\log \log (x)}{\log (y)}(1+o(1)). \end{aligned}$$

It follows that

$$\begin{aligned} \frac{\psi _r(x,y,d)}{\sqrt{x}}&\ge \exp \left( \frac{1}{2}\log (x)-\frac{\log (x)\log \log (x)}{\log (y)}(1+o(1))\right) \\&=\exp \left( \frac{1}{2}\log (x)(1+o(1))\right) \longrightarrow +\infty , \end{aligned}$$

since $\log (y)\gg \log \log (x)$. Finally, we have

$$\begin{aligned} \log \left( \frac{\psi _r^*(x,y,d)}{x}\right)&=\log \left( \frac{\psi _r(x,y,d)}{x}\right) +\log \left( 1-\frac{\sqrt{x}}{\psi _r(x,y,d)}\right) \\&\ge -\frac{\log (x)\log \log (x)}{\log (y)}(1+o(1))+o(1). \end{aligned}$$

Besides, $\psi _r^*(x,y,d)\le \psi (x,y)$, so we conclude by Lemma 4.9. $\square$

Proposition 4.11

Let $\Delta :=|\Delta _{\mathfrak {O}}|$ and $\varepsilon :=\Delta ^{1/r_m}/B$. We assume that B is subexponential in $\log (\Delta )$, $\varepsilon <1$ and $\log (1-\varepsilon )\ll \log \log (B)$. Then Algorithm 4.1 has expected time complexity (in bit operations)

$$\begin{aligned} T_{FS}(\Delta ,B,r_m)= \exp \Bigg ((1+o(1))\frac{\log (\Delta )\log \log (\Delta )}{\log (B)}\\ +(\sqrt{2}+o(1))\sqrt{\log (B)\log \log (B)}\Bigg ), \end{aligned}$$

assuming the ECM method has the complexity conjectured in [38, Conjecture 2.10].

Proof

By Heuristic 4.7 (since $\varepsilon <1$ i.e. $r_m\ge \log (\Delta )/\log (B)$), the probability to find an ns-$(B,r_m,\Delta )$-smooth value of $N(a+\omega )$ stisfies

$$\begin{aligned} \mathbb {P}(B, r_m,\Delta )\ge C\cdot \frac{\psi _{r_m}^*(N(\omega ),B,\Delta )}{N(\omega )}, \end{aligned}$$

where $C>0$ is a constant. Since $N(\omega )=(\Delta +t^2)/2$ with $t:={{\,\textrm{Tr}\,}}(\omega )=\Delta \mod 2$ and B is subexponential in $\log (\Delta )$, we have $\Delta =O(N(\omega ))$, $N(\omega )=O(\Delta )$, $\log (B)\ll \log (N(\omega ))$ and $\log (B)\gg \log \log (N(\omega ))$. We also have $r_m\ge \log (\Delta )/\log (B)$ and $\log (1-\varepsilon )\ll \log \log (B)$, so we may apply Lemma 4.10:

$$\begin{aligned} \log \left( \frac{\psi _{r_m}^*(N(\omega ),B,\Delta )}{N(\omega )}\right)&\sim -\frac{\log (N(\omega ))\log \log (N(\omega ))}{\log (B)}\\&\sim -\frac{\log (\Delta )\log \log (\Delta )}{\log (B)}(1+o(1)). \end{aligned}$$

Hence, taking into account the ECM method complexity, Algorithm 4.1 terminates in time

$$\begin{aligned} T_{FS}(\Delta , B, r_m)&=\frac{L_B(1/2,\sqrt{2}){{\,\mathrm{{\text {polylog}}}\,}}(\Delta )}{\mathbb {P}(B, r_m, \Delta )}=\exp \Bigg ((1+o(1))\frac{\log (\Delta )\log \log (\Delta )}{\log (B)}\\&\qquad +(\sqrt{2}+o(1))\sqrt{\log (B)\log \log (B)}\Bigg ). \end{aligned}$$

$\square$

4.2.2 Complexity of the tree filling algorithm (Algorithm 4.2)

Proposition 4.12

With inputs $B>0$, an imaginary quadratic order $\mathfrak {O}$ with $|\Delta _\mathfrak {O}|B<p$, primes $\ell _1,\cdots ,\ell _s\le B$ splitting in $\mathfrak {O}$ and an oracle for Problem 2.3 running in constant time, Algorithm 4.2 runs in time

$$\begin{aligned} O\left( 2^sB^2{{\,\mathrm{{\text {polylog}}}\,}}(B)\log (p)M(p)\right) , \end{aligned}$$

where M(p) is the time complexity of operations over $\mathbb {F}_p$. It also uses $O(2^s\log (p))$ bits of memory.

Proof

Filling-in tree $\mathcal {T}$ in Algorithm 4.2 costs for all $1\le i\le s$, $2^{i-1}$ calls to and the computation of $2^{i-1}$ sets of j-invariants $\ell _i$-isogenous to the same elliptic curve. Each call to costs O(1) and each j-invariants computation costs $O(\ell _i^2{{\,\mathrm{{\text {polylog}}}\,}}(\ell _i)\log (p))$ operations over $\mathbb {F}_{p^2}$ by Sect. 2.4. Arithmetic operations over $\mathbb {F}_{p^2}$ cost O(M(p)). Hence, the total cost of filling tree $\mathcal {T}$ is

$$\begin{aligned} T_{tree}(s,B,p)&=\sum _{i=1}^{s}2^{i-1}O\left( \ell _i^2{{\,\mathrm{{\text {polylog}}}\,}}(\ell _i)\log (p)M(p)\right) \\&=\sum _{i=1}^s 2^{i-1}O(B^2{{\,\mathrm{{\text {polylog}}}\,}}(B)\log (p)M(p))\\&=O\left( 2^sB^2{{\,\mathrm{{\text {polylog}}}\,}}(B)\log (p)M(p)\right) . \end{aligned}$$

The memory used by Algorithm 4.2 is the size of tree $\mathcal {T}$, which contains $\sum _{i=1}^s 2^{i-1}=2^s-1$ j-invariants defined over $\mathbb {F}_{p^2}$. Each j-invariant takes $2\log (p)$ bits to store, so the algorithm uses $O(2^s\log (p))$ bits of memory. $\square$

4.2.3 Complexity of the search to decision reduction algorithm (Algorithm 4.3)

Theorem 4.13

Let $\Delta :=|\Delta _{\mathfrak {O}}|$. Assume Heuristic 4.7 and that the ECM method has the complexity conjectured in [38, Conjecture 2.10]. Then, with smoothness parameters

$$\begin{aligned} B:=L_\Delta \left( \frac{1}{2},\frac{\sqrt{2}}{2}\right) , \quad r_m:=\lceil \sqrt{\frac{2\log (\Delta )}{\log \log (\Delta )}}\rceil +1 \quad \hbox {and} \quad D:=O(\log (p)) \end{aligned}$$

and provided $B\Delta <p$, Algorithm 4.3 terminates in time

$$\begin{aligned} L_\Delta (1/2,\sqrt{2})\log (p)M(p). \end{aligned}$$

It also requires

$$\begin{aligned} O\left( 2^{\sqrt{2\log (\Delta )/\log \log (\Delta )}}\log (p)\right) \end{aligned}$$

bits of memory.

Proof

We already have proved the termination of Algorithm 4.3 when $B\Delta <p$. This is a consequence of Lemma 4.3, Lemma 4.4 and Heuristic 4.7 (which prove that and terminate).

On the whole, the total time complexity (in bit operations) of Algorithm 4.3 is

$$\begin{aligned} T(B,\Delta ,r_m,p)=T_{FS}+2T_{tree}+T_{iso}+T_{trace}+T_{rep}, \end{aligned}$$

where:

• $T_{FS}$ is the execution time of (Algorithm 4.1), given by Proposition 4.11:

  $$\begin{aligned} T_{FS}(\Delta ,B,r_m)&= \exp \Bigg ((1+o(1))\frac{\log (\Delta )\log \log (\Delta )}{\log (B)}\\&\quad +(\sqrt{2}+o(1))\sqrt{\log (B)\log \log (B)}\Bigg ). \end{aligned}$$

• $T_{tree}$ is the execution time of (Algorithm 4.2), given by Proposition 4.12:

  $$\begin{aligned} T_{tree}(B,s,p)=O\left( 2^sB^2{{\,\mathrm{{\text {polylog}}}\,}}(B)\log (p)M(p)\right) . \end{aligned}$$

  with $s=r_m/2+O(1)$.

• $T_{iso}$ is the time taken in lines 9 and 10 of Algorithm 4.3 to recover the chain of $\ell _i$-isogenies $\psi _i:E_{i-1}\longrightarrow E_i$, given the sequence of j-invariants $j(E_0)=j(E),j(E_1),\cdots ,j(E_r)=j(E)$. By Sect. 2.5, recovering an $\ell _i$-isogeny from the j-invariants of its domain and codomain costs $O(\ell _i^2{{\,\mathrm{{\text {polylog}}}\,}}(\ell _i)\log (p))$ operations over $\mathbb {F}_{p^2}$. Hence, we have

  $$\begin{aligned} T_{iso}=O(r_m B^2{{\,\mathrm{{\text {polylog}}}\,}}(B)\log (p)M(p)) \end{aligned}$$

• $T_{trace}$ is the time needed to compute the trace of $\psi =\psi _r\circ \cdots \circ \psi _1$. We use Schoof’s algorithm [58, Sect. 5]. Namely, we look for primes $p_1, \cdots , p_t$ such that $\prod _{i=1}^t p_i>4\sqrt{\deg (\psi )}$ and evaluate $\psi$ on $E[p_i]$ to find $\tau _i\in \mathbb {Z}/p_i\mathbb {Z}$ such that $\psi ^2-[\tau _i]\psi +[\deg (\psi )]$ is zero on $E[p_i]$ and recover ${{\,\textrm{Tr}\,}}(\psi )$ by solving ${{\,\textrm{Tr}\,}}(\psi )\equiv \tau _i \mod p_i$ for all $i\in \{1,\cdots , t\}$ via Chinese remainder theorem. Since $\deg (\psi )=N(\theta )\le \Delta$, we can choose $t=O(\log (\Delta ))$ and $p_i=O(\log (\Delta ))$. Hence, the dominant cost is the evaluation via $\psi$ of $O(\log (\Delta ))$ points all defined over an extension of degree $O(\log (\Delta ))$ of $\mathbb {F}_{p^2}$ (by Lemma 2.12). This cost amounts to

  $$\begin{aligned} T_{trace}(B,r_m,\Delta ,p)=O(r_mB\log ^3(\Delta )M(p)). \end{aligned}$$

• $T_{rep}$ is the running time of (Algorithm 2.1). Since $\deg ([\epsilon ]\circ \psi -[a])=N(\omega )\le (\Delta +1)/4$, we can find a D-powersmooth number coprime with $\deg ([\epsilon ]\circ \psi -[a])$ when $D=O(\log (\Delta ))$ (line 1 of Algorithm 2.1). Hence, by Proposition 2.13, the dominant cost of the call to is given by $O(\log (\Delta ))$ evaluations of $\psi$ on points defined over an extension of degree $O(\log (\Delta ))$ of $\mathbb {F}_{p}$, which amounts to

  $$\begin{aligned} T_{rep}(\Delta ,B,r_m,p)=O(r_m B\log ^3(\Delta )M(p)). \end{aligned}$$

It follows that:

$$\begin{aligned} T(B,r_m,\Delta ,p)&=T_{FS}+2T_{tree}+T_{iso}+T_{trace}+T_{rep}\\&=\exp \Bigg ((1+o(1))\frac{\log (\Delta )\log \log (\Delta )}{\log (B)}\\&\qquad +(\sqrt{2}+o(1))\sqrt{\log (B)\log \log (B)}\Bigg )\\&\qquad +M(p)\log (p)\exp \left( \frac{\log (2)r_m}{2}+2\log (B)\right) \end{aligned}$$

But by Proposition 4.11, we have $r_m=\log (\Delta )/\log (B\varepsilon )$ with $\log (1-\varepsilon )\ll \log \log (B)$. We can impose that $\varepsilon \longrightarrow 0$, so that $\log (1-\varepsilon )\ll \log \log (B)$ and that $\log (\varepsilon )\ll \log (B)$, so that $r_m\sim \log (\Delta )/\log (B)$. Heuristically, the quantity $T(\Delta ,B,r_m,p)$ is minimal when the arguments of the two exponentials are close, i.e. when

$$\begin{aligned} \frac{\log (\Delta )\log \log (\Delta )}{\log (B)}\simeq 2\log (B), \end{aligned}$$

the other terms being negligible. Hence, we choose

$$\begin{aligned} B=\exp \left( \frac{\sqrt{2}}{2}\sqrt{\log (\Delta )\log \log (\Delta )}\right) =L_\Delta \left( \frac{1}{2},\frac{\sqrt{2}}{2}\right) , \end{aligned}$$

so that

$$\begin{aligned} T(B,r_m,\Delta ,p)=M(p)\log (p)L_\Delta \left( \frac{1}{2},\sqrt{2}\right) . \end{aligned}$$

and

$$\begin{aligned} r_m&=\sqrt{\frac{2\log (\Delta )}{\log \log (\Delta )}}\left( 1+\frac{\sqrt{2}\log (\varepsilon )}{\sqrt{\log (\Delta )\log \log (\Delta )}}\right) ^{-1}\\&=\sqrt{\frac{2\log (\Delta )}{\log \log (\Delta )}}-\frac{2\log (\varepsilon )}{\log \log (\Delta )}. \end{aligned}$$

Hence, we can set $r_m:=\lceil \sqrt{2\log (\Delta )/\log \log (\Delta )}\rceil +1$, so that $\log (\varepsilon )=O(\log \log (\Delta ))=o(\log (B))$.

The space complexity is dominated by the trees $\mathcal {T}_1$ and $\mathcal {T}_2$, so Algorithm 4.3 uses

$$\begin{aligned} O(2^{r_m/2}\log (p))=O\left( 2^{\sqrt{2\log (\Delta )/\log \log (\Delta )}}\log (p)\right) \end{aligned}$$

bits of memory by Proposition 4.12. $\square$

Corollary 4.14

Given an imaginary quadratic order $\mathfrak {O}$ of discriminant $\Delta _{\mathfrak {O}}$ and a prime $p>L_{|\Delta _\mathfrak {O}|}(1/2,\sqrt{2}/2)|\Delta _{\mathfrak {O}}|$, then, over $\mathbb {F}_{p^2}$ the $\mathfrak {O}$-orienting Problem (Problem 2.4) reduces to the Decisional $\mathfrak {O}$-orienting Problem (Problem 2.3) in time

$$\begin{aligned} L_{|\Delta _\mathfrak {O}|}(1/2,\sqrt{2})\log (p)M(p), \end{aligned}$$

using

$$\begin{aligned} O\left( 2^{\sqrt{2\log (|\Delta _\mathfrak {O}|)/\log \log (|\Delta _\mathfrak {O}|)}}\log (p)\right) \end{aligned}$$

bits of memory.

5 $\mathfrak {O}$-orienting problem for quaternion orders

Isogeny problems can often be translated to quaternion problems via the Deuring correspondence, and in many cases, the quaternion problems are easier to solve. In this section we consider the quaternion analogue of the $\mathfrak {O}$-Orienting Problem as it was stated earlier:

Problem 2.6

(Quaternion Order Embedding Problem) Given a maximal quaternion order $\mathcal {O} \subset B_{p, \infty }$ and an imaginary quadratic order $\mathfrak {O}$ which embeds into $\mathcal {O}$, find the embedding.

Similarly to the curve setting, we define a primitive $\mathfrak {O}$-embedding of $\mathcal {O}$ to be an embedding $\iota : \mathfrak {O} \hookrightarrow \mathcal {O}$ which cannot be extended to a superorder of $\mathfrak {O}$, also known as an optimal embedding [62, Chapter 30]. We also address this problem for primitive embeddings.

In this section, we present a general algorithm and analyse its complexity, noting special cases. For complexity analysis we assume an efficient factorization oracle exists, however, we provide a practical alternative for running the algorithm without such an oracle. For embedding small discriminant quadratic orders $\mathfrak {O}$, our algorithm improves the state of the art by being efficient up to ${{\,\textrm{disc}\,}}(\mathfrak {O}) = O(p)$.

Before moving on to the actual algorithms we give a brief technical overview of the main idea. First, we compute a short prime norm N ($\approx \sqrt{p}$) connecting ideal between a quaternion order $\mathcal {O'}$ isomorphic to $\mathcal {O}$ and a special extremal order. Our goal is to compute an element of prescribed trace and norm in $\mathcal {O'}$ and then one can easily construct an element with said trace and norm in $\mathcal {O}$ as well. For simplicity assume that the prescribed trace is 0. The trace 0 part of $\mathcal {O'}$ is a rank 3 lattice and one can compute the Hermite Normal Form (HNF) of this lattice. This means that one has a basis of the form $e_{11}i+e_{12}j+e_{13}k,e_{22}j+e_{23}k,e_{33}k$ and even though $e_{ij}$ are not likely to be integers, their denominator is a divisor of 2N. When looking for an element of trace 0 and norm smaller than p the coefficients of this element with respect to this HNF basis will have a very specific structure. Namely, the coefficient of $e_{11}i+e_{12}j+e_{13}k$ will be smaller than p in absolute value and thus can be easily determined by looking at the norm modulo p. Then one only has to work out the two other coefficients which is equivalent to solving a binary quadratic form where the quadratic part is positive definite. This can then essentially be reduced to Cornacchia’s algorithm [57]. We can extend this to filter out imprimitive solutions.

5.1 Finding general embeddings

First, we present an algorithm for finding embeddings, and in the next section, we use this to define primitive embeddings. Suppose we are given a maximal quaternion order $\mathcal {O} \subset B_{p, \infty }$ in terms of a $\mathbb {Z}$-basis, and an imaginary quadratic order $\mathfrak {O} = \mathbb {Z}[\omega ]$, by generator $\omega$ of reduced trace t and reduced norm d.

We start with an observation: suppose an embedding $\iota : \mathbb {Z}[\omega ] \hookrightarrow \mathcal {O}$ exists and let $\alpha = \iota (\omega )$. Since $\omega ^2 - t \omega + d = 0$ we must also have $\alpha ^2 - t \alpha + d = 0$. Hence $\alpha$ also has trace t and norm d. Finding any element $\alpha$ of norm d and trace t is enough to define an embedding $\iota$, solving Problem 2.6. This is the approach we take in Algorithm 5.1, finding $\alpha \in \mathcal {O}$ of a given norm and trace. We make the assumption $p \ne 2$ and conventionally use 1, i, j, k as a basis of $B_{p, \infty }$ with $i^2 = -q$ and $j^2 = -p$. If $p\equiv 3\pmod {4}$ we take $q = 1$. If $p\equiv 5\pmod {8}$ we take $q = 2$. If $p\equiv 1\pmod {8}$ we take q to be a prime $q\equiv 3\pmod {4}$ such that p is not a quadratic residue modulo q. While $p \equiv 3 \mod 4$ is the most relevant for isogeny-based cryptography, we consider general p. We fix a maximal order $\mathcal {O}_0$ in the following way:

Proposition 5.1

[48, Proposition 5.2] The following definitions give a maximal order in $B_{p, \infty }$ for any $p \ne 2$:

$$\begin{aligned} \mathcal {O}_0 = {\left\{ \begin{array}{ll} \mathbb {Z}[\frac{1+j}{2}, \frac{i+k}{2}, j, k] &{} \text {if } p \equiv 3 \mod 4 \\ \mathbb {Z}[\frac{1+j+k}{2}, \frac{i+2j+k}{4}, j, k] &{} \text {if } p \equiv 5 \mod 8 \\ \mathbb {Z}[\frac{1+i}{2}, \frac{i + ck}{q}, \frac{j+k}{2}, k] &{} \text {if } p \equiv 1 \mod 8 \\ \end{array}\right. } \end{aligned}$$

where c is an integer such that q divides $c^2 p + 1$ where q and c exist by [21, Proposition 1].

Our algorithm will work with the basis of $\mathcal {O}$ in column-style Hermite normal form (HNF). We denote the basis vectors $e_0, e_1, e_2, e_3$. Then we can write $\mathcal {O}$ as:

$$\begin{aligned} \begin{aligned} \mathcal {O} = \langle&e_{00} + e_{01}i + e_{02}j + e_{03}k, \\&e_{11}i + e_{12}j + e_{13}k, \\&e_{22}j + e_{23}k, \\&e_{33}k \rangle _{\mathbb {Z}} \end{aligned} \end{aligned}$$

(11)

with coefficients $e_{mn} \in \mathbb {Q}$. For example, see the orders in Proposition 5.1 above. We know the basis is full rank, so $e_{nn} \ne 0$ for $n=0,1,2,3$, and we prove some additional properties:

Lemma 5.2

Given the basis $(e_{mn})$ of a maximal order $\mathcal {O} \subset B_{p, \infty }$ in column-style HNF as above, the following properties hold:

• (1) $e_{mn} \ge 0$ for all n, m

• (2) The denominators of each rational entry $e_{mn}$, when expressed in simplest form, divide $K \cdot N(I)$ where $K = 2, 4$ or 2q depending on whether $p \equiv 3 \mod 4$, or $\equiv 5 \mod 8$ or $\equiv 1 \mod 8$ respectively

• (3) $e_{00} = \frac{1}{2}$

• (4) $e_{22} e_{33} \le N(I)$

• (5) $e_{01} = 0$ or $e_{01} = 1/ (2 K e_{22}e_{33})$ where K is defined in (2)

where $I = I({\mathcal {O}}_0, \mathcal {O}) :=N\mathcal {O}_0 \mathcal {O}$ is the connecting left-ideal from $\mathcal {O}_0$ to $\mathcal {O}$ and $N :=[\mathcal {O}: \mathcal {O}_0 \,\cap \,\mathcal {O}]$.

Proof

We can prove the statements as follows:

• (1) Requirement of HNF.

• (2) As defined, I is the connecting ideal between $\mathcal {O}_0$ and $\mathcal {O}$. The ideal I is contained in both $\mathcal {O}_0$ and $\mathcal {O}$ and $N(I)\cdot \mathcal {O} = I {\bar{I}} \subseteq \mathcal {O}_0$ [62, Proposition 16.6.15]. Therefore the largest denominator of all $e_{mn}$s is at most N(I) times the largest denominator of $\mathcal {O}_0$ as given in Proposition 5.1.

• (3) The trace of any element must be integral hence $2 e_{00} \in \mathbb {Z}$. We must also have $1 \in \mathcal {O}$ hence 1 can be written as a linear combinations of the basis $(e_i)$, where where taking the trace gives $k_0 e_{00} = 1$ for some $k_0 \in \mathbb {Z}$. This implies either $e_{00} = \frac{1}{2}$ or 1 and ${{\,\textrm{Tr}\,}}(\mathcal {O}) = \mathbb {Z}$ or $2 \mathbb {Z}$ respectively. The (non-reduced) discriminant of any maximal order in $B_{p, \infty }$ is $p^2$, so by definition $p^2 = \det ({{\,\textrm{Tr}\,}}(e_m e_n)) \in {{\,\textrm{Tr}\,}}(\mathcal {O})$, but p is odd, so $p^2 \not \in 2 \mathbb {Z}$ and we must have $e_{00} = \frac{1}{2}$.

• (4) As $\mathcal {O}$ and $\mathcal {O}_0$ are maximal, they both have the same discriminant. Hence the change of basis matrix must have determinant 1 [62, Lemma 15.2.5], which means $\prod e_{nn} = \prod f_{nn} = \frac{1}{2\cdot K}$, where $(f)_n$ is the basis of $\mathcal {O}_0$ specified in Proposition 5.1. Then using (3) we have $e_{11} = 1/(Ke_{22}e_{33})$. The result follows from (2).

• (5) $1 \in \mathcal {O}$ so there is some $n \in \mathbb {Z}$ such that $\frac{1}{e_{00}} e_{01} - n e_{11} = 0$. From above $e_{00} = \frac{1}{2}$, and $e_{11} = \frac{1}{Ke_{22}e_{33}}$ so $2 e_{01} = \frac{n}{K e_{22}e_{33}}$. But to be in HNF we must have already reduced $e_{01}$ as much as possible hence $n = 0$ or 1.

$\square$

Further, we will also use the following lemma to bound the denominators (taken from [16, Lemma 5.2.2]):

Lemma 5.3

Let $\mathcal {O} \subset B_{p, \infty }$ be a maximal order with connecting ideal $I = I({\mathcal {O}}_0, \mathcal {O})$, then there exists an equivalent ideal $J \sim I$ with $N(J) \le \frac{2\sqrt{2}}{\pi } \sqrt{p}$

We now describe the algorithm. We address arbitrary trace in Remark 5.4 and Algorithm 5.1 has no restrictions on the trace. However, for simplicity, we first describe the algorithm under the assumption that the trace of $\omega$ is zero.

Step 1:

: Compute HNF Put the basis of $\mathcal {O}$ into column-style Hermite normal form (HNF). In general, we can replace the order $\mathcal {O}$ by an isomorphic order $\mathcal {O'}$, having denominator bounded by $N :=K \cdot N(I') = O(\sqrt{p})$, where $I'$ is an ideal equivalent to the connecting $(\mathcal {O}_0, \mathcal {O})$-ideal I, and where K is defined in (2) of Proposition 5.2, by taking the ideal from Lemma 5.3. We return to this in Sect. 5.3, but for now, by passing to the isomorphic order “closest” to $\mathcal {O}_0$, we assume that N is of size $O(\sqrt{p})$

Step 2:

: Fix trace To find a trace zero element $\alpha$ of norm d, we may write an arbitrary element in the following form:

$$\begin{aligned} \alpha = \alpha _0 e_0 + \alpha _1 e_1 + \alpha _2 e_2 + \alpha _3 e_3. \end{aligned}$$

Note that since we are working in Hermite Normal Form only $e_0$ contributes to the trace of $\alpha$ so we set $\alpha _0 = 0$ to get ${{\,\textrm{Tr}\,}}(\alpha ) = 0$. For the condition on the norm, consider the case $p \equiv 3 \mod 4$ for simplicity. However, note that this generalizes for any prime $p \ne 2$. Then we have the rational ternary quadratic form:

$$\begin{aligned} (\alpha _1 e_{11})^2 + p(\alpha _1 e_{12} + \alpha _2 e_{22})^2 + p(\alpha _1 e_{13} + \alpha _2 e_{23} + \alpha _3 e_{33})^2 = {{\,\textrm{nrd}\,}}(\alpha ) = d. \end{aligned}$$

Step 3:

: Find $\pmb {\alpha _1 \mod p}$ Since $\alpha _1$ controls the coefficient of i it is the only term without a factor of p. Hence working modulo p removes terms containing $\alpha _2$ and $\alpha _3$, and we can find $\alpha _1 \equiv r \mod p$.

$$\begin{aligned} r_{\pm } :=\frac{\pm \sqrt{d}}{e_{11}} \mod p \end{aligned}$$

Fix the least positive residue class representative $r = r_{+}$, as we can execute the remainder of the algorithm a second time on $r_{-}$ if necessary. Then substitute $\alpha _1 = r + k p$ giving a rational ternary quadratic form in k, $\alpha _2$ and $\alpha _3$.

Step 4:

: A binary quadratic form As defined in Step 1, we may multiply by the denominator $N^2$ to obtain integral coefficients. Rearranging we have:

$$\begin{aligned} p N^2 ( \gamma _1^2 + \gamma _2^2 ) = N^2(d - \alpha _1^2 e_{11}^2) \end{aligned}$$

where

$$\begin{aligned} \gamma _1 = \alpha _1 e_{12} + \alpha _2 e_{22}, \qquad \gamma _2 = \alpha _1 e_{13} + \alpha _2 e_{23} + \alpha _3 e_{33}. \end{aligned}$$

Let $v :=N^2(\gamma _1^2 + \gamma _2^2)$ and notice $v \ge 0$. From the right-hand side above we see its value depends on k.

$$\begin{aligned} v = \frac{N^2 (d - (r + kp)^2 e_{11}^2)}{p} \end{aligned}$$

Clearly, v decreases as k increases. Without loss of generality, we can assume $k \ge 0$, and since $v \ge 0$ we get an upper bound on k. We can iterate over this range of k which is precisely

$$\begin{aligned} k=0,..., \left\lfloor \frac{\sqrt{d}}{p e_{11}} - \frac{r}{p} \right\rfloor \end{aligned}$$

where for each iteration, we compute v using the above equation, and with k fixed are left with the integral binary quadratic form $v = N^2(\gamma _1^2 + \gamma _2^2)$.

Step 5:

: Cornacchia’s Algorithm Writing the above form as $\beta _1^2 + \beta _2^2 = v$ we solve for integral pairs $(\beta _1, \beta _2)$ using Cornacchia’s algorithm. For a valid solution we can write it in the form:

$$\begin{aligned} \beta _1&= N \gamma _1 = N\alpha _1 e_{12} + N\alpha _2 e_{22} \\ \beta _2&= N \gamma _2 = N\alpha _1 e_{13} + N\alpha _2 e_{23} + N\alpha _3 e_{33} \end{aligned}$$

and solve for $\alpha _2$ and $\alpha _3$

$$\begin{aligned} \alpha _2 = \frac{\beta _1 - N \alpha _1 e_{12}}{N e_{22}}, \qquad \alpha _3 = \frac{\beta _2 - N \alpha _1 e_{13} - N \alpha _2 e_{23}}{N e_{33}}. \end{aligned}$$

Finally, we must check $\alpha _2, \alpha _3 \in \mathbb {Z}$. If this is true we have a valid solution $\alpha = \alpha _1 e_1 + \alpha _2 e_2 + \alpha _3 e_3$. If not we continue trying the next solution to Cornacchia’s, or move on to the next iteration of k in Step 4. If no solutions are found it means $\mathbb {Z}[\omega ]$ does not embed into $\mathcal {O}$.

Remark 5.4

(Arbitrary trace t) Suppose the element we are searching for does not have trace zero. We can always reduce the problem to finding an element of trace zero. Suppose $t \in 2 \mathbb {Z}$, then since $\mathcal {O}$ is a ring we have $1 \in \mathcal {O}$ so $\alpha - t/2 \in \mathcal {O}$ has trace zero and norm $d - t^2/4 \in \mathbb {Z}$. We can search for this trace zero element and then use this to find $\alpha$. Similarly if t is odd, we have a trace zero element $2\alpha - t \in \mathcal {O}$ of norm $4d - t^2$. Once this is found, we translate back, divide by 2 and check if $\alpha \in \mathcal {O}$ in Step 5 of the algorithm. If not, we continue searching.

Note for t odd, this is not optimal as the scaling increases d by a factor of 4, and hence the number of iterations of k by a factor of 2, which asymptotically makes no difference, but in practice, can double the running time. Instead we can avoid this by incorporating additional constant terms for the non-zero trace. These details are included in Algorithm 5.1, which we use for our implementation.

The complete algorithm for arbitrary trace is summarised in Algorithm 5.1. Additionally, we describe a few further generalisations and improvements:

Remark 5.5

Algorithm 5.1...

• results in an embedding, but this does not necessarily define a primitive $\mathbb {Z}[\omega ]$-embedding. This is discussed in Sect. 5.5.

• can be adapted to work with any prime $p\ne 2$, not specifically $p \equiv 3 \mod 4$. For general, $B_{p, \infty } = \left( \frac{-q,-p}{\mathbb {Q}}\right)$, q appears in the equations for r, v and the maximum k, and the binary quadratic form to solve is $\beta _1^2 + q\beta _2^2 = v$ instead of the sum of two squares. Cornacchia’s still works since for $B_{p, \infty }$, q and p are always coprime.

• can be adapted to non-maximal orders. The value N gains the index of the order within a maximal order as a factor.

• is more efficient iterating from largest k to smallest, as this minimizes the values of v used in Cornacchia’s.

• can be improved by using a congruence condition to rule out some cases where Cornacchia’s does not have any solutions, before executing Cornacchia’s. In the case $p \equiv 3 \mod 4$, we test for solutions by noting v can be written as the sum of two squares if and only if, in its prime factorization, every prime which is 3 mod 4 occurs an even number of times. For arbitrary p, a similar necessary but not sufficient congruence condition can test the splitting of v to rule out some cases.

5.2 Complexity analysis of Algorithm 5.1

In this section we give results on the asymptotic complexity of Algorithm 5.1, in particular giving average-case results and a probabilistic worst-case result. We start by giving a worst-case running time. Note that there are three reasons why Cornacchia’s algorithm may not be efficient at finding all solutions to $\beta _1^2 + q \beta _2^2 = v$:

• (1) It requires a factorization of v. To this end, we assume we have an efficient factorization oracle such as Shor’s algorithm. See Sect. 5.3 later on for a practical alternative to using a factorization oracle.

• (2) Cornacchia’s algorithm typically only refers to finding primitive solutions where $gcd(\beta _1, \beta _2)=1$. To also find imprimitive solutions we must run Cornacchia’s on $\beta _1^2 + q \beta _2^2 = v/g^2$ for every square $g^2 \mid v$ and scale up the solutions $(g\beta _1, g\beta _2)$. The number of squares dividing v can be subexponential in v. However, we can say the probability of this for random v is very small, in fact asymptotically there is $\frac{\pi ^2}{6} \sim 61\%$ chance v is square-free.

• (3) While just solving for primitive solutions, we must iterate over all the solutions Cornacchia gives. Internally Cornacchia must iterate over all solutions x to the equation $x^2 \equiv -q \mod v$, where the number of solutions can be exponential in v if v has a large number of distinct prime factors. For example, experimentally with $p \equiv 3 \mod 4$ and $d \sim p$ we get some integers $v \sim p$ where if v has lots of distinct prime factors, there can be as many as $v^{0.15} \sim p^{0.15}$ solutions which is exponential. We resolve this issue by bounding the number of factors of v by the following probability estimate known as the fundamental theorem of probabilistic number theory:

Lemma 5.6

(Erdős–Kac theorem [24]) For a positive integer n, the number of distinct prime factors of n follows the standard normal distribution with mean $\log \log n$ and standard deviation $\sqrt{\log \log n}$ as $n \rightarrow \infty$.

This gives us the following result:

Theorem 5.7

Let $0.5 \le P < 1$. Assuming the heuristic that v is distributed like random integers and hence the number of distinct prime factors follows Lemma 5.6, and given an efficient Algorithm 5.1 is within

with probability P. With $N = 2 N(I) = O(\sqrt{p})$ (Lemma 5.3) this is

where X is the total size of the inputs, and T(P) is a value large enough such that the asymptotic probability that a random number has less than T(P) perfect square divisors is larger than P. We define $\mathcal {F}$ as the inverse cumulative distribution function of the standard normal distribution where a sample is less than $\mathcal {F}(P)$ with probability P. For example, for $P = 0.95$ we have $\mathcal {F}\left( \frac{P+1}{2}\right) < 2$ and $T\left( \frac{P+1}{2}\right) \sim 4$.

Proof

Steps 1–3 of the algorithm are efficient as polynomial time algorithms exist for computing Hermite normal form [30] [14, Chapter VII], and fixing $\alpha _0$ and solving $\alpha _1$ modulo p is efficient. In Step 4 a worst-case input will result in iterating k over it’s full range of values which is $O( \frac{1}{p e_{11}} \sqrt{d - (\alpha _0 e_{00})^2} )$, where the trace is fixed through $\alpha _0 = \frac{t}{2 e_{00}}$ so $(\alpha _0 e_{00})^2 = \frac{t^2}{4}$. And by Proposition 5.2 we have $\frac{1}{e_{11}} \le N$. Then for each iteration over k, Cornacchia’s algorithm is used in Step 5. To be efficient at finding primitive solutions we have to bound the number of distinct prime factors of v, by Lemma 5.6 with probability $\frac{P+1}{2}$, v is less than $\mathcal {F}(\frac{P+1}{2})$ standard deviations above the mean,

$$\begin{aligned} \text {Number of factors of } v \le \log \log (v) + \mathcal {F}\left( \frac{P+1}{2}\right) \sqrt{\log \log (v)} \end{aligned}$$

hence it is certainly true that

$$\begin{aligned} \text{ Number } \text{ of } \text{ factors } \text{ of } v \le (\mathcal {F}\left( \frac{P+1}{2}\right) +\log \log (v). \end{aligned}$$

Then it follows that the number of square roots found in Cornacchia’s algorithm is less than $O(2^{(\mathcal {F}(\frac{P+1}{2}) + 1) \log \log (v)}) = O(\log (v)^{(\mathcal {F}(\frac{P+1}{2}) + 1)})$, so we can bound the running time of Cornacchia by $O(\log (v)^{(\mathcal {F}(\frac{P+1}{2}) + 1)}) \cdot polylog(v)$, and clearly for each v we have $v \le N^2 d < O(pd)$ and hence $polylog(v) = polylog(X)$. The final consideration is for finding imprimitive solutions using Cornacchia’s algorithm which requires repeating for every square dividing v. By definition this is at most $T(\frac{P+1}{2})$ repetitions with probability $\frac{P+1}{2}$. The probability both this condition and v having the correct number of factors is at least $\frac{P+1}{2} + \frac{P+1}{2} -1 = P$. $\square$

Now we give a result for the average-case running time:

Theorem 5.8

Making the following assumptions, regarding iterating over k:

• Each $v_k$ is distributed like random integers and hence the expected number of distinct prime factors is $\log \log (v_k)$ by Lemma 5.6, and there is a high probability it only has a few square divisors.

• Additionally, the probability each $v_k$ is the sum of two squares is independent and at least the probability a random integer less than $(Nd)^2$ is the sum of two squares.

• The first solution to Cornacchia’s algorithm has $\beta _1, \beta _2$ uniformly distributed modulo $e_{22} N$ and $e_{33} N$ respectively.

Then given an efficient factorization oracle, in the case $p \equiv 3 \mod 4$, the average-case running time of Algorithm 5.1 is $O(\min \{ N^3, \lceil \frac{N}{p}\sqrt{d - \frac{t^2}{4}}\rceil \} \times polylog(X))$ and substituting $N = O(\sqrt{p})$ from Lemma 5.3 it is $O(\min \{ p \sqrt{p}, \lceil \frac{1}{\sqrt{p}}\sqrt{d - \frac{t^2}{4}}\rceil \} \times polylog(X))$ where X is the total size of all inputs.

We use the following result of Landau [35]:

Lemma 5.9

The number of integers representable as the sum of two squares from from 0 to $n \in \mathbb {N}$ is the limit $C \frac{n}{\sqrt{\log n}}$ as $n \rightarrow \infty$, where $C \approx 0.764$ is the Landau-Ramanujan constant. Hence for sufficiently large n, the number of integers representable is greater than $\frac{1}{2} \frac{n}{\sqrt{\log n}}$. (In fact, experimentally this appears true for all $n \ge 0$).

Proof of Theorem 5.8

It’s clear the running time is the product of the number of iterations over k, and the running time of Cornacchia’s, because all other operations are polynomial time. In the case $p \equiv 3 \mod 4$ we are solving the sum of two squares, hence by Lemma 5.9, and using the first assumption, we expect (for sufficiently large d) less than $2 \sqrt{\log ((N d)^2)}$ iterations until we find a k where Cornacchia’s gives at least one solution.

Now recall that finding one solution to Cornacchia’s algorithm is not necessarily enough, since we need to satisfy the conditions $\alpha _2, \alpha _3 \in \mathbb {Z}$. This amounts to checking:

$$\begin{aligned} \beta _1 - N \alpha _0 e_{02} - N \alpha _1 e_{12}&\equiv 0 \mod e_{22} N \\ \beta _2 - N \alpha _0 e_{03} - N \alpha _1 e_{13} - N \alpha _2 e_{23}&\equiv 0 \mod e_{33} N. \end{aligned}$$

Therefore, by the second assumption we expect to have an integral solution after $e_{22} N \times e_{33} N$ solutions from Cornacchia’s. Noting that $e_{22} e_{33} \le N/2$ from Proposition 5.2, that’s $N^3 /2$ solutions. In total we expect $O(N^3 \sqrt{\log (N d)})$ iterations of k. This is bounded above by the maximum number of iterations from Theorem 5.7. Finally, Cornacchia’s algorithm uses the efficient factorization oracle to factorize each $v_k$ and on average $v_k$ is expected to have $\log \log (v_k)$ distinct prime factors by the first assumption, hence internally Cornacchia’s computes at most $2^{\log \log (v_k)} = \log (v_k)$ square roots, which is efficient. Then to find imprimitive solutions, we only repeat Cornacchia’s a constant number of times as the expected number of squares dividing v is very small. Overall this takes time polylog in each $v_k \le N^2 d = O(pd)$, so this term can be incorporated into polylog(X). $\square$

From this we observe the following:

Remark 5.10

(Efficient for orders close to $\mathcal {O}_0$) Given an efficient factorization oracle, consider the algorithm applied to the order $\mathcal {O}_0$. Here we have $N=2$, hence the algorithm is efficient; the average-case running time is polylog(X). For orders close to $\mathcal {O}_0$, such as a curve l-isogenous to the curve with j-invariant 1728, we gain a factor of l in N, hence for small l the algorithm is still efficient. However with each step from $\mathcal {O}_0$, N gains a factor of the degree of the isogeny, so it gets exponentially harder the further you walk, until we reach the point $N \sim \sqrt{p}$.

For completeness, we now consider the case of arbitrary primes $p \ne 2$. Then the quaternion algebra containing order $\mathcal {O}$ is $B_{p, \infty } = \left( \frac{-q, -p}{\mathbb {Q}} \right)$ where q is either 2 or a prime $q \equiv 3 \mod 4$ with Legendre symbol $\left( \frac{q}{p} \right) = -1$.

By the same argument as Theorem 5.7, with high probability P the worst-case running time is within

which is the same as before except the additional factor of $\frac{1}{\sqrt{q}}$ appears requiring more iterations over k. Then from Lemma 5.2 part (2), for a different value of K, we get $N = 2q N(I) = O(q\sqrt{p})$. Applying this along with Lemma 5.3 gives:

Typically, q is treated as a constant, as for random primes, q will typically be of negligible size, so asymptotically the complexity is the same. However, q is actually unbounded; you can construct a prime such that the minimum value for q is larger than any given threshold. Hence we treat q as a variable in our analysis.

We have a similar variation on the average time complexity, however, the proof is more complex:

Theorem 5.11

Given an efficient factorization oracle, for arbitrary $p \ne 2$, making the same assumptions as in Theorem 5.8 (replacing sum of two squares with $x^2 + q y^2$), the average-case running time of Algorithm 5.1 is

where C is a special function generalising the Landau-Ramanujan constant, and X is the total size of all inputs.

Proof

The proof is the same as Theorem 5.8, except instead of solving the sum of two squares, we are solving $x^2 + q y^2 = v_k$ using Cornacchia’s algorithm. This means in the proof we cannot use Lemma 5.9 on the sum of two squares. However, this result generalises.

Bernays proved that the number of integers between 0 and n represented by a binary quadratic form f(x, y) converges to $C(\Delta _f) \frac{n}{\sqrt{\log n}}$ as $n \rightarrow \infty$, where $\Delta _f$ is the discriminant of the form f and $C(\Delta _f)$ is a constant depending on $\Delta _f$ [5]. In our case $f(x,y) = x^2 + q y^2$, hence $\Delta _f = -4q$.

Additionally the bound $e_{22} e_{33} \le \frac{N}{2}$ from the proof of Theorem 5.8 becomes $e_{22} e_{33} \le \frac{N}{2q}$ by Proposition 5.2. And we have $N = 2q N(I) = q \cdot O(1) \sqrt{p}$. $\square$

For an explicit formula for $C(-4q)$, see the results of Moree and Osburn [43], and for a summary of results on $C(\cdot )$ see the work of Brink, Moree and Osburn [8].

Next, we note how the complexity changes in other contexts:

Remark 5.12

(Suborders) Suppose you are given a quaternion order $\mathcal {O} \subset B_{p, \infty }$ which is not necessarily maximal. As stated in Remark 5.5, Algorithm 5.1 still works. The complexity is the same with the subtlety that N is multiplied by the index of the suborder within a maximal order.

Remark 5.13

(p-extremal orders) Suppose $\mathcal {O}$ is a p-extremal order, and has suborder $R + jR \subseteq \mathcal {O}$ and we are trying to find an embedding into this suborder. For $\omega$ a generator of R and $\alpha = \alpha _0 + \alpha _1 \omega$, $\alpha ' = \alpha _0' + \alpha _1' \omega \in R$, the norm equation is:

$$\begin{aligned} nrd(\alpha + j \alpha ') = f(\alpha _0, \alpha _1) + p f(\alpha _0', \alpha _1') = d \end{aligned}$$

where f is a binary quadratic form of discriminant disc(R). The approach in the KLPT algorithm [34, Sect. 3.2] randomly samples $\alpha _0'$ and $\alpha _1'$ until $d - p f(\alpha _0', \alpha _1') = q$ is a prime which is split in R where the ideal factors of (q) are principal, and hence its generator gives a solution to $q = f(\alpha _0, \alpha _1)$. Assuming sampled integers q satisfy the distribution of primes less than d, this takes roughly $2 h(R) \log (d)$ iterations. Note that we require d to be large enough for the set $d - p f(x, y)$ to contain enough primes.

Our algorithm is very similar but works in reverse. Also assuming d is sufficiently large ($d>p^{2+\epsilon }$), we sample k until $q = \frac{d - f(\alpha _0, \alpha _1)}{p} \in \mathbb {Z}$ has a solution to the equation $q = f(\alpha _0', \alpha _1')$. By the same argument, if we wait until q is a prime, split in R, where the ideals of norm q are principal, we are guaranteed a solution, so we also expect $2 h(R) \log (d)$ iterations, which is efficient assuming R has a small class number.

Finally note we do not necessarily need a factorization oracle using the technique presented in the next section.

5.3 Rerandomization and small discriminant

Consider the special case of small discriminant orders $\mathbb {Z}[\omega ]$ in Algorithm 5.1. Previously, the best known efficient algorithm, as stated in [66] was simply to look for small vectors in $\mathcal {O}$. This works for $|{{\,\textrm{disc}\,}}(\mathbb {Z}[\omega ])| < 2\sqrt{p} - 1$ and is stated below. Note that an alternative approach that heuristically works for $|{{\,\textrm{disc}\,}}(\mathbb {Z}[\omega ])|<p^{0.8}$ is outlined in [4].

Proposition 5.14

[66, Proposition 6] Assume that $|{{\,\textrm{disc}\,}}(\mathbb {Z}[\omega ])|<2\sqrt{p}-1$. Then, there is a probabilistic polynomial time algorithm that solves Problem 2.6.

Under certain heuristics, we can rerandomize Algorithm 5.1 by considering isomorphic orders, potentially in different representations of $B_{p, \infty }$, to avoid factoring and bound the denominator $N < O(\sqrt{p})$. The result of this is a corollary (Corollary 5.16) which gives a heuristic polynomial time algorithm for solving Problem 2.6 for ${{\,\textrm{disc}\,}}(\mathbb {Z}[\omega ])$ in O(p), or deciding that no solution exists.

The first step is to bound the number of values to try Cornacchia on in Algorithm 5.1. By Cornacchia’s algorithm, here we mean finding all solutions to $x^2 + qy^2 = v$, not just primitive ones.

Lemma 5.15

Fix a positive integer M. As in the context of Algorithm 5.1, take $d = {{\,\textrm{nrd}\,}}(\omega )$, N as the common denominator of the HNF basis, and k the variable we iterate over. Suppose we have

$$\begin{aligned} d \le \frac{q p^2 (\frac{M}{2}-1)^2}{N^2}, \end{aligned}$$

then Algorithm 5.1 performs at most M executions of Cornacchia’s algorithm, each with some additional polynomial time arithmetic, and all other operations are polynomial time.

Proof

As stated in proof of Theorem 5.7, all computations performed in the algorithm, excluding the iterations over k, can be achieved in polynomial time. For each iteration over k we compute a value v using basic arithmetic operations, and naively use Cornacchia’s algorithm only once. Iterating over k happens twice, once using $r_{+}$ and once using $r_{-}$. Therefore it is enough to show for each $r \in \{r_{+}, r_{-}\}$ we perform at most $\lfloor M/2 \rfloor$ iterations over k. As in proof of Theorem 5.7, and taking into account general p, we can consider the maximum number of iterations over k for each r. It is sufficient to show this is bounded by M/2,

$$\begin{aligned} \left\lfloor \frac{1}{p e_{11}}\left( \sqrt{\frac{d - (\alpha _0 e_{00})^2}{q}} - \alpha _0 e_{01} - r e_{11} \right) \right\rfloor + 1 \le \frac{M}{2} \end{aligned}$$

which since $e_{01} \ge 0$ by Proposition 5.2, is certainly true if

$$\begin{aligned} \frac{1}{p e_{11}} \cdot \sqrt{\frac{d}{q}} + 1 \le \frac{M}{2} \end{aligned}$$

and noting $e_{11} \ge \frac{1}{N}$, we see this is true by its equivalence to the condition in statement of the Lemma,

$$\begin{aligned} d \le \frac{q p^2 (\frac{M}{2}-1)^2}{N^2}. \end{aligned}$$

$\square$

From this, we obtain a result about ${{\,\textrm{disc}\,}}(\mathbb {Z}[\omega ])$ since we can translate generator $\omega$ to either $\sqrt{-{{\,\textrm{disc}\,}}(\mathbb {Z}[\omega ])}/2$ or $(1+\sqrt{-{{\,\textrm{disc}\,}}(\mathbb {Z}[\omega ])})/2$ hence $N(\omega )=d\le (|{{\,\textrm{disc}\,}}(\mathbb {Z}[\omega ])|+1)/4$. Recalling that we can bound N, the denominator of $\mathcal {O}$, to $O(\sqrt{p})$, we see that Lemma 5.15 says that when ${{\,\textrm{disc}\,}}(\mathbb {Z}[\omega ])$ is in O(p), and assuming q in O(1), the only potentially expensive part in Algorithm 5.1 is Cornacchia on a constant number of instances.

The general idea of our rerandomized version is then the common technique of only running Cornacchia on “good” instances. However, if the discriminant is small, then the embedding is with large probability unique, hence we might end up discarding the correct solution. Therefore, we need to rerandomize until all O(1) Cornacchia instances are good before the algorithm can be sure that no embedding exists.

We define a “good” instance to be $x^2 + qy^2 = v_k$ where $v_k$ can be factorized in polynomial time, has $O(\log \log (N^2 d))$ distinct prime factors, and $O(\log (N^2 d))$ square divisors. The set of prime numbers satisfies these conditions, and with the heuristic that the events of each $v_k$ being prime are independent and follow from the density of primes, we expect at most some constant multiple of $\log (N^2 d)^{C}$ iterations until C of the Cornacchia instances are primes. With $C = O(1)$ instances from the small discriminant condition, this is efficient.

Now we discuss how we rerandomize the order. Let $\mathcal {O}_0 \subseteq B_{p, \infty }$ be a maximal order with negligible denominator K (e.g. the “standard” maximal order from Proposition 5.1). As has been pointed out, any maximal order $\mathcal {O} \subseteq B_{p, \infty }$ will have denominator bounded by KN, where N is the norm of the connecting ideal from $\mathcal {O}_0$ to $\mathcal {O}$. Hence we can consider any equivalent ideal $J = I\gamma$ of small norm, and instead solve the problem in the isomorphic order $\mathcal {O}_R(J) = \gamma ^{-1}\mathcal {O}\gamma$, before transferring the solution back to $\mathcal {O}$.

As we rerandomize, we might need to try many distinct $J \sim I$ of small norm. Heuristically, for random orders $\mathcal {O}$, we can expect there to be an abundance of small, equivalent ideals (i.e. with N(J) in $O(\sqrt{p})$). The problem is that this heuristic fails completely if there exists some equivalent ideal $I'$ with ${{\,\textrm{nrd}\,}}(I') \ll \sqrt{p}$, i.e. in the case that $\mathcal {O}$ is too “close” to $\mathcal {O}_0$. We can fix this by considering other maximal orders $\mathcal {O}_0'$ with negligible denominator in other representations of $B_{p, \infty }$.

Specifically, we can generate representations $B_i = (-q_i, -p \mid \mathbb {Q})$, where we take $q_i$ to be the smallest primes satisfying

$$\begin{aligned} q_i \equiv 3 \pmod {4}, \quad \left( \frac{-q_i}{p}\right) = -1. \end{aligned}$$

These quaternion algebras are indeed ramified at p and $\infty$ [62, Proposition 14.2.7]. In each of these representations, regardless of congruence conditions on p, we can take a standard choice of maximal order $\mathcal {O}_{0,i}$ with denominator $2q_i$ as

$$\begin{aligned} \mathcal {O}_{0,i} :=\mathbb {Z} \oplus \mathbb {Z}\frac{1 + i}{2} \oplus \mathbb {Z}j \oplus \mathbb {Z}\frac{(1+i)j}{2q_i}, \end{aligned}$$

[62, Exercise 15.5]. Heuristically, these choices of maximal orders of the different quaternion algebra representations are “independent” in the sense that there is no reason that these should be close to each other, so it is enough to try a small, fixed number of such orders, as (heuristically), the probability that $\mathcal {O}$ is close to all $\mathcal {O}_{0, i}$ is negligible.

Finally, the explicit isomorphisms $B_i \cong B_j$ are also easy to find and compute, using [25, Lemma 10]. The whole algorithm is summarized in Algorithm 5.2.

This gives the following corollary:

Corollary 5.16

For arbitrary $p \ne 2$, given a maximal order $\mathcal {O} \subseteq B_{p, \infty }$, and a quadratic order $\mathfrak {O}$ with $|{{\,\textrm{disc}\,}}(\mathfrak {O})|$ in O(p), Algorithm 5.2 computes an $\mathfrak {O}$-embedding of $\mathcal {O}$, or decides that none exists. Under the heuristics discussed above the algorithm terminates in probabilistic polynomial time in $\log (p)$.

Proof

By Lemma 5.15, and the subsequent discussion, the only potentially expensive step of running Algorithm 5.1 are the (constant number of) Cornacchia instances. By only running the Cornacchia on prime instances (or more generally, “good” instances), we expect to have to run Algorithm 5.1 at most $O(\log (p)^{O(1)})$ times by the prime number theorem, and the fact that v is in O(p). Further, it is clear that if all O(1) values to try Cornacchia on is prime, and a solution is still not found, a solution cannot exist, hence the algorithm can conclude that no solution exists. $\square$

5.4 From embeddings to primitive embeddings

Algorithms 5.1 and 5.2 find all possible embeddings $\iota : \mathbb {Z}[\omega ] \hookrightarrow \mathcal {O}$. Every embedding gives a primitive $\mathbb {Z}[\omega ']$-embedding where $\mathbb {Z}[\omega ] \subseteq \mathbb {Z}[\omega ']$. We separate between these two cases, and call those embeddings which give primitive superorder embeddings $\mathbb {Z}[\omega ] \subsetneq \mathbb {Z}[\omega ']$ imprimitive embeddings. Finding primitive embeddings solves Problem 2.6, and we consider this question in Sect. 5.5. In this section, we focus on the imprimitive embeddings. First, we explain how to differentiate between primitive and imprimitive embeddings.

For any element $\alpha \in \mathcal {O}$, write ${{\tilde{\alpha }}}$ for its class in the lattice $\mathcal {O}/\mathbb {Z}$. Consider the discriminant form

$$\begin{aligned} \Delta : \mathcal {O}/\mathbb {Z}&\longrightarrow \mathbb {Q}\\ {\tilde{\alpha }}&\longmapsto {{\,\textrm{Tr}\,}}(\alpha ) - 4 {{\,\textrm{nrd}\,}}(\alpha ), \end{aligned}$$

an integral quadratic form of rank 3. It does not depend on the choice of a representative $\alpha$ of the class ${{\tilde{\alpha }}}$, and we also write $\Delta (\alpha )$. Let d be an integer. We say that a solution $\alpha \in \mathcal {O}$ of $\Delta (\alpha ) = d$ is primitive if ${\tilde{\alpha }}$ is a primitive element of the lattice $\mathcal {O}/\mathbb {Z}$, i.e., it is not of the form ${\tilde{\alpha }} = b{\tilde{\beta }}$ for some element ${{\tilde{\beta }}} \in \mathcal {O}/\mathbb {Z}$ and integer $b > 1$. We now show primitive solutions correspond directly to primitive embeddings.

Lemma 5.17

An element $\alpha \in \mathcal {O}$ is a primitive solution of $\Delta (\alpha ) = d$ if and only if $\mathbb {Z}[\alpha ] \subseteq \mathcal {O}$ is a primitive embedding.

Proof

Suppose $\alpha$ is an imprimitive solution of $\Delta (\alpha ) = d$, i.e., there are integers a and $b>1$ such that $(\alpha - a)/b \in \mathcal {O}$. Then, $\mathbb {Z}[\alpha ] \subsetneq \mathbb {Z}[(\alpha - a)/b] \subseteq \mathcal {O}$, hence $\mathbb {Z}[\alpha ] \subseteq \mathcal {O}$ is not primitive. Conversely, suppose $\mathbb {Z}[\alpha ] \subseteq \mathcal {O}$ is not primitive, so there exists $\beta \in (\mathbb {Q}[\alpha ] \cap \mathcal {O}){\setminus } \mathbb {Z}[\alpha ]$. There exists integers a and $b>1$ such that $\alpha = a + b\beta$. In particular, ${{\tilde{\alpha }}} = b{{\tilde{\beta }}}$, so $\alpha$ is not a primitive solution. $\square$

Now we know primitive embeddings come from primitive solutions, we can determine if an embedding is primitive, and if not extend it to its superorder, very fast using a $\gcd$ computation:

Lemma 5.18

Given a maximal order $\mathcal {O}$ with basis $e_0, e_1, e_2, e_3$ and an element $\alpha \in \mathcal {O}$ of trace $t={{\,\textrm{Tr}\,}}(\omega )$ and norm $d={{\,\textrm{nrd}\,}}(\omega )$, there is a polynomial time algorithm which:

• determines whether embedding $\iota$ defined by extending $\omega \mapsto \alpha$ is a primitive or imprimitive embedding of $\mathbb {Z}[\omega ]$,

• and if it’s imprimitive outputs $(a, b, \alpha ')$ defining a superorder $\mathbb {Z}[\frac{\omega - a}{b}] \supsetneq \mathbb {Z}[\omega ]$ which $\iota$ can be extended to, through the map $\frac{\omega - a}{b} \mapsto \alpha '$.

Proof

First convert the upper diagonal basis $e_0, e_1, e_2, e_3$ of $\mathcal {O}$ into an lower diagonal basis $f_0, f_1, f_2, f_3$,

$$\begin{aligned} \begin{aligned} \mathcal {O} = \langle f_0, f_1, f_2, f_3 \rangle _{\mathbb {Z}} = \langle&f_{00}, \\&f_{10} + f_{11}i, \\&f_{20}i + f_{21}i + f_{22}j, \\&f_{30} + f_{31}i + f_{32}j + f_{33}k \rangle _{\mathbb {Z}} \end{aligned}\end{aligned}$$

(12)

and compute a function applying the change of basis transformation taking coefficients of $e_i$s onto coefficients of $f_i$s. This is polynomial time as a variant of the Hermite normal form algorithm, and can be seen as precomputation. Then given a solution $\alpha$, we change the basis to obtain:

$$\begin{aligned} \alpha = \gamma _0 f_0 + \gamma _1 f_1 + \gamma _2 f_2 + \gamma _3 f_3 \end{aligned}$$

Since $\mathcal {O}$ is a ring we have $1 \in \mathcal {O}$, and every norm is integral so it cannot contain a rational number less than one. Hence $f_{0} = f_{00} = 1$. For $\alpha$ to be a primitive solution there should be no $a,b \in \mathbb {Z}$ with $b > 1$ such that $\alpha - a = b\tau$ where $\tau \in \mathcal {O}$. Equivalently, for any a, when expressing $\alpha - a$ in terms of $f_i$s, the coefficients should not all be divisible by any $b > 1$. Note that we have:

$$\begin{aligned} \alpha - a= (\gamma _0 - a) f_0 + \gamma _1 f_1 + \gamma _2 f_2 + \gamma _3 f_3 \end{aligned}$$

where $a = \gamma _0$ can be chosen, setting the first coefficient to zero. Then the solution is primitive if and only if $\gamma _1, \gamma _2, \gamma _3$ share no factor. This can be checked with a $\gcd$ computation. Note that if the solution is imprimitive, so we have $gcd(\gamma _1, \gamma _2, \gamma _3) = b > 1$, we return $(\gamma _0, b, \frac{\gamma _1 f_1 + \gamma _2 f_2 + \gamma _3 f_3}{b})$ defining an embedding giving a primitive $\mathbb {Z}[\frac{\omega - \gamma _1}{b}]$-embedding. $\square$

Algorithm 5.3 is a concise version of this. Hence using Algorithm 5.1 for the embedding we find we always get a primitive $\mathbb {Z}[\omega ']$-embedding without affecting asymptotic time complexities. Furthermore, if we iterate over the full range of k in Algorithm 5.1, we find all embeddings and hence all primtive superorder embeddings.

5.5 Finding primitive $\pmb {\mathbb {Z}[\omega ]}$-embeddings-solving problem 2.6

To solve Problem 2.6 we must find primitive $\mathbb {Z}[\omega ]$-embeddings. We have Algorithm 5.1 for finding embeddings, and we have Algorithm 5.3 which can check if an embedding is primitive.

To combine them, we modify Algorithm 5.1 to include the pre-computation of basis $f_i$ and the change of basis transformation at the start, then when each solution is found, check if it is primitive using Algorithm 5.3 and only stop searching if it is. The worst-case running time doesn’t change, since finding a primitive embedding takes at most as long as finding all embeddings. However, the average-case running time does increase, heuristically by the total number of solutions divided by the number of primitive solutions. We now provide a further heuristic argument that this ratio can be bounded from above.

Let $f(\gamma _1, \gamma _2, \gamma _3) = \lambda$ be the solution to the ternary quadratic norm form of the order defined in 12 with fixed trace. We have shown the solution is primitive if and only if $gcd(\gamma _1, \gamma _2, \gamma _3) = gcd(|\gamma _1|, |\gamma _2|, |\gamma _3|) = 1$.

Consider the rational solution $x_1 = x_2 = x_3 \in \mathbb {Q}$ such that $f(x_1, x_1, x_1) = w x_1^2 = \lambda$ where $w \in \mathbb {Q}$ is the sum of coefficients in the form. Then $|x_1| = |\sqrt{\lambda / w}|$ and since the norm form is positive definite, this means if one variable were to increase, another must decrease in absolute value. Therefore $\min (|\gamma _1|, |\gamma _2|, |\gamma _3|) \le \lfloor \sqrt{\lambda / w} \rfloor$.

Now reconsider our integral solution. Suppose the solution is not primitive so $gcd(|\gamma _1|, |\gamma _2|, |\gamma _3|) \ne 1$, then there is a prime number $\ge 2$ that divides all three numbers. This prime factor must be in the set $S = \{2, 3, 5,...,\lfloor \sqrt{\lambda / w} \rfloor \} \cap \{\text {Primes } p\}$ as it must divide the smallest of these three numbers.

Heuristically, we assume that $\gamma _1, \gamma _2, \gamma _3$ are distributed like random numbers in the sense that some $q \in S$ divides one of them with uniformly random probability 1/q. And assume independence of the probabilities of different factors $q_1, q_2 \in S$ occurring. Then the probability 2 divides all three numbers is $1/2^3$, the probability 3 divides them is $1/3^3$, and the probability any $q \in S$ divides them is $1/q^3$. Combined, the probability a number in S divides all three is:

$$\begin{aligned} \mathbb {P}[(\gamma _1, \gamma _2, \gamma _3) \text { imprimitive}] = \sum _{\text {primes } q\in S} \frac{1}{q^3} \le \sum _{\text {all primes } q \in \mathbb {N}} \frac{1}{q^3} = P(3) \le 0.175 \end{aligned}$$

where P(3) is the prime zeta function at 3. Hence the probability a random solution is primitive is over $80\%$ so if we find 5 independent solutions we would expect at least one to be primitive. Therefore assuming the heuristics above, if we modify algorithm 5.1 to ignore imprimitive solutions, the average running time should only increase by at most a factor of 5.

Note that this argument makes some strong assumptions. Experimentally for some parameters, we see the probability the first solution found is primitive is around $80\%$, but on other parameter choices, it is considerably lower. With all the parameters we tested, we found the probability is always over $50\%$, which still suggests the average running time is only worsened by a small factor, but it gives reason to doubt these assumptions. In particular, consider independence. The existence of embeddings comes with symmetry hence we may find two solutions where there is only a change of sign in the defining formulae. This means the probability q divides the coefficients of one solution might have a strong dependence on whether q divides the coefficients of the second solution. We leave a more complete analysis of the probability of finding primitive embeddings for future research.

Singular points on the curve \(\Phi _\ell (X,Y)=0\)

If K is a field, we denote by ${\tilde{X}}_0(\ell ,K)$ the curve $\Phi _\ell (X,Y)=0$ over the field K. We also define:

$$\begin{aligned} S_0(\ell ,\mathbb {F}_{p^2}):=\Bigg \{j\in \mathbb {F}_{p^2} \hbox { supersingular } \Bigg | \ \exists j'\in \mathbb {F}_{p^2}, \ \ \Phi _\ell (j,j')=\frac{\partial \Phi _\ell }{\partial X}(j,j') \\ =\frac{\partial \Phi _\ell }{\partial Y}(j,j')=0\Bigg \}. \end{aligned}$$

Lemma A.1

Assume that $2\ell <p$. Then, $\# S_0(\ell ,\mathbb {F}_{p^2})=O(\ell ^{3+o(1)})$.

Proof

Let $j(E)\in S_0(\ell ,\mathbb {F}_{p^2})$ and $j(E')\in \mathbb {F}_{p^2}$ such that $(j(E),j(E'))$ is a singular point of ${\tilde{X}}_0(\ell ,\mathbb {F}_{p^2})$ i.e. such that

$$\begin{aligned} \Phi _\ell (j(E),j(E'))=\frac{\partial \Phi _\ell }{\partial X}(j(E),j(E'))=\frac{\partial \Phi _\ell }{\partial Y}(j(E),j(E'))=0. \end{aligned}$$

Schoof proved in [58, Sect. 7] that there exists a lift $(j({\widetilde{E}}),j(\widetilde{E'}))$ over $\mathbb {C}$ of $(j(E),j(E'))$ that is also a singular point of the curve ${\tilde{X}}_0(\ell ,\mathbb {C})$. Schoof deduced that there exists two $\ell$-isogenies $\phi , \psi : {\widetilde{E}}\longrightarrow \widetilde{E'}$ over $\mathbb {C}$ that are not equal up to pre or post composition by an isomorphism, so that $\varphi :={\widehat{\psi }}\circ \phi$ is a cyclic endomorphism of ${\widetilde{E}}$ of degree $\ell ^2$. Hence, $\varphi$ is non-scalar and ${{\,\mathrm{{\text {End}}}\,}}({\widetilde{E}})$ is isomorphic to an imaginary quadratic order $\mathfrak {O}$ and $\mathbb {Z}[\varphi ]$ is mapped to a suborder of $\mathfrak {O}$ via this isomorphism. It follows that ${{\,\textrm{disc}\,}}(\mathfrak {O}) \mid {{\,\textrm{disc}\,}}(\mathbb {Z}[\varphi ])$. But

$$\begin{aligned} {{\,\textrm{disc}\,}}(\mathbb {Z}[\varphi ])={{\,\textrm{Tr}\,}}(\varphi )^2-4\deg (\varphi )={{\,\textrm{Tr}\,}}(\varphi )^2-4\ell ^2. \end{aligned}$$

Since $\mathfrak {O}$ is imaginary quadratic, so is $\mathbb {Z}[\varphi ]$ and ${{\,\textrm{disc}\,}}(\mathbb {Z}[\varphi ])<0$, so that $|{{\,\textrm{disc}\,}}(\mathfrak {O})|\le |{{\,\textrm{disc}\,}}(\mathbb {Z}[\varphi ])|\le 4\ell ^2$.

Since $2\ell <p$, p does not divide the conductor of $\mathfrak {O}$ (otherwise, $p^2 \mid {{\,\textrm{disc}\,}}(\mathfrak {O})$ so $p^2\le 4\ell ^2$). It follows by [46, Lemma 3.1] (generalizing [36, Chapter 13, Theorem 12]), that E is (primitively) $\mathfrak {O}$-oriented. Besides, by [46, Proposition 3.3 and Theorem 3.4] there are at most $2\#{{\,\mathrm{{\text {Cl}}}\,}}(\mathfrak {O})$ j-invariants of supersingular $\mathfrak {O}$-oriented curves. By Siegel’s theorem [59], we have $\#{{\,\mathrm{{\text {Cl}}}\,}}(\mathfrak {O})=O(|{{\,\textrm{disc}\,}}(\mathfrak {O})|^{1/2+o(1)})=O(\ell ^{1+o(1)})$, so there are at most $O(\ell )$ j-invariants of $\mathfrak {O}$-oriented supersingular elliptic curves. Taking into account all possible imaginary quadratic orders $\mathfrak {O}$ of discriminant $|{{\,\textrm{disc}\,}}(\mathfrak {O})|\le 4\ell ^2$, we conclude that j(E) lies in a set of cardinality $O(\ell ^{3+o(1)})$, which completes the proof. $\square$

Lemma A.2

Assume that $\log (\ell )\ll \log (p)$. Then, computing an $\ell$-isogeny between two $\ell$-isogenous supersingular j-invariants chosen uniformly at random takes on average ${\tilde{O}}(\ell ^2\log (p))$ operations over $\mathbb {F}_{p^2}$.

Proof

As discussed in Sect. 2.5, the average number of operations over $\mathbb {F}_{p^2}$ to compute an $\ell$-isogeny between supersingular $\ell$-isogenous j-invariants is:

$$\begin{aligned} N&:=(1-\mathbb {P}((j(E),j(E')) \hbox {singular})-\mathbb {P}(j(E) \hbox { or } j(E')=0,1728)){\tilde{O}}(\ell ^2\log (p))\\&\qquad + (\mathbb {P}((j(E),j(E')) \hbox { singular})+\mathbb {P}(j(E) \hbox { or } j(E')=0,1728))O(\ell ^{7/2}) \end{aligned}$$

Since there are $\sim p/12$ supersingular j-invariants by [60, Theorem V.4.1.c], we obtain by Lemma A.1:

$$\begin{aligned} \mathbb {P}((j(E),j(E')) \hbox { singular})\le \mathbb {P}(j(E)\in S_0(\ell ,\mathbb {F}_{p^2}))=O\left( \frac{\ell ^{3+o(1)}}{p}\right) \end{aligned}$$

Besides, $\mathbb {P}(j(E) \hbox { or } j(E')=0,1728)=O(1/p)$. Since $\log (\ell )\ll \log (p)$, we have $\ell ^{13/2+o(1)}\ll p$ so the dominant term of N is ${\tilde{O}}(\ell ^2\log (p))$ and all other terms are negligible. Hence, $N={\tilde{O}}(\ell ^2\log (p))$ and the proof is complete. $\square$