The phase-space architecture in extrasolar systems with two planets in orbits of high mutual inclination

Abstract

We revisit the secular 3D planetary three-body problem, aiming to provide a unified formalism representing all basic phenomena in the phase space as the mutual inclination between the planetary orbits increases. We propose a ‘book-keeping’ technique allowing to decompose the Hamiltonian as $\mathcal {H}_\textrm{sec}=\mathcal {H}_\textrm{planar}+\mathcal {H}_\textrm{space}$, with $\mathcal {H}_\textrm{space}$ collecting all terms depending on the planets mutual inclination $i_\textrm{mut}$. We numerically compare several models obtained by multipole (Legendre) or Laplace–Lagrange expansions of $\mathcal {H}_\textrm{sec}$, aiming to define suitable truncation orders for these models. We explore the transition, as $i_\textrm{mut}$ increases, from a ‘planar-like’ to a ‘Lidov–Kozai’ regime. Using a numerical example far from hierarchical limits, we find that the structure of the phase portraits of the (integrable) planar case is reproduced to a large extent also in the 3D case. A semi-analytical criterion allows to estimate the level of $i_\textrm{mut}$ up to which the dynamics remains nearly integrable. We propose a normal form method to compute the basic periodic orbits (apsidal corotation orbits A and B) in this regime. We explore the sequence of saddle-node and pitchfork bifurcations by which the A and B families are connected to the highly inclined periodic orbits of the Lidov–Kozai regime. Finally, we perform a numerical study of phase portraits for different planetary mass and distance ratios and qualitatively describe the approach to the corresponding hierarchical limits.

1 Introduction

In the present paper we revisit the planetary three-body problem in Poincaré heliocentric canonical variables governed by the Hamiltonian

$$\begin{aligned} \mathcal {H}({\varvec{r}_2},{\varvec{r}_3},{\varvec{p}_2},{\varvec{p}_3})=\frac{{\varvec{p}_2}^2}{2\, m_2}-\frac{\mathcal {G}\, m_0\, m_2}{r_2}+\frac{{\varvec{p}_3}^2}{2 \,m_3}-\frac{\mathcal {G}\, m_0\, m_3}{r_3}+\frac{({\varvec{p}_2}+{\varvec{p}_3})^2}{2\, m_0}-\frac{\mathcal {G}\, m_2\, m_3}{|{\varvec{r}_2}-{\varvec{r}_3}|}, \nonumber \\ \end{aligned}$$

(1)

where $m_0 \gg m_2,\, m_3\,$. Thus, $m_0$ represents the mass of a star, and $m_{i}\,$, ${\varvec{p}_i}\,$, ${\varvec{r}_{i}}\,$, $i=2,3$ the masses, barycentric momenta and heliocentric position vectors of two planets orbiting the star, at distances $r_i=|{\varvec{r}_i}|\,$.

Our focus is on a systematic study of how the structure of the phase space of the above Hamiltonian system is altered as the planetary masses and distances and, most importantly, the mutual inclination between the two planets’ trajectories is varied. Answering this question in the framework of the three-body problem is a key step towards understanding the phase-space architecture in planetary systems with two or more planets in orbits with high mutual inclination. A well-studied case of the latter, used in all our numerical studies below, is the $\upsilon$-Andromedae system. The choice of this system is motivated by the fact that it has been proposed (see references below) as an archetype example of exoplanetary system with substantially 3D dynamics. It is a double star system with four planets (b, c, d and e) orbiting one of the stars. Since $m_b \ll m_c, m_d$ the motion of the innermost planet b can be modeled to a good approximation via a restricted four-body problem, with planets c, d providing the main perturbations (this motivates the use of the indices $2\,$, 3 in the Hamiltonian (1) for the masses and canonical variables of the planets); the long-term motion of the innermost planet $(m_1,{\varvec{r}}_1,{\varvec{p}}_1)$ is considered in a separate work (Mastroianni and Locatelli 2023). More detailed models can include planet e as well as the second star. At any rate, all the above models require providing first a good analytical model for the orbits of the giant planets c and d, whose masses, as estimated by observations, are larger than $10\,M_{J}\,$ (see, e.g., Deitrick et al. 2015; McArthur et al. 2010).

The available data for the orbital parameters of an extrasolar system are typically affected by wide observational error bars. As discussed below, one important problem with the uncertainties in the observations stems from the fact that even small changes in a system’s estimated parameters, consistent with the observations, may imply a drastic change in the type of orbital state in which the observed system is assumed to have been settled. We will argue that this sensitive dependence on available parameter estimates affects mostly those predictions referring to the secular (long-term) evolution of the orbital state, i.e., the variations of the planets’ eccentricity and inclination vectors which take place in timescales of the order of $\sim 10^2-10^4$ orbital periods. Characterizing the whole variety of possible stable secular orbital states of exoplanetary systems can be relevant also in the interpretation of short-in-time observations: most importantly, it can serve the purpose of constraining observational uncertainties on the basis of stability considerations, i.e., indicating which subdomains in parameter space favor the long-term stability of the planetary orbits (Petit et al. 2018; Stalport et al. 2022; Volpi et al. 2018)

Our study in the present paper focuses on one phenomenon, whose role appears central to the aim of classifying and characterizing the variety of possible secular orbital states in 3D planetary systems: this is the chain of bifurcations of periodic orbits which mark the transition, as the planets’ mutual inclination increases, from the apsidal corotation to the Lidov–Kozai regime.

As is well known, the apsidal corotation (AC) states (see, for example, Laughlin et al. 2002; Beaugé et al. 2003; Lee and Peale 2003) provide the backbone of the phase space of secular motions for two planets in coplanar orbits. Formally, the apsidal corotations are the continuation, in the nonlinear regime, of the linear (Laplace–Lagrange) normal mode solutions of the planar two-planets secular Hamiltonian. Physically, they represent periodic orbits along which the pericenters of the two planets constantly precess by always remaining either anti-aligned (state A) or aligned (state B). Periodic orbits having the same property in the exact (non-averaged with respect to fast angles) Hamiltonian can also be found Hadjidemetriou (1975), Hadjidemetriou (2006) once the dynamics is regarded in a frame rotating with angular velocity equal to the (common) precession rate of the apsides.

Given their importance in the planar case, a natural question regards how the family of apsidal corotation (AC) orbits is continued, as well as what is the dynamical role played by the AC periodic, and nearby quasi-periodic, orbits, when we pass from the coplanar to the 3D planetary orbital configuration. Both questions have been addressed in the literature using diverse formalisms (see Beaugé et al. 2012 and references therein). Owing to reasons explained in Sect. 2, in our present study we employ a formalism, in which, by applying a book-keeping technique already at the level of Jacobi’s reduction of the nodes (Jacobi 1842), we arrive at a natural decomposition of the 3D secular Hamiltonian of a system with fixed Angular Momentum Deficit (AMD; see Laskar and Robutel 1995) as

$$\begin{aligned} \mathcal {H}_\textrm{sec}=\mathcal {H}_\textrm{planar}({\textbf{X}},{\textbf{Y}})+\mathcal {H}_\textrm{space}({\textbf{X}},{\textbf{Y}};\textrm{AMD})~~. \end{aligned}$$

(2)

In Eq. (2), $({\textbf{X}},{\textbf{Y}})$ are Poincaré canonical variables for the two planets ($(X_2,Y_2)$, $(X_3,Y_3)$ are approximately proportional to the planets’ eccentricity vectors). The Angular Momentum Deficit is defined by

$$\begin{aligned} \textrm{AMD}=L_2+L_3-L_z \end{aligned}$$

where $L_2$, $L_3$ are the angular momenta of the circular orbits at semi-major axes $a_2,a_3$ equal to those of the two planets, and $L_z$ the modulus of the total angular momentum normal to the system’s Laplace plane.

The methodological benefits from working with a decomposition of the secular Hamiltonian as in Eq. (2) stem from the following remarks (see Sect. 2 and 3 for details):

• (i) $\mathcal {H}_\textrm{planar}({\textbf{X}},{\textbf{Y}})$ is an integrable Hamiltonian, the quantity $J=(X_2^2+Y_2^2+X_3^2+Y_3^2)/4$ being a second integral independent of the energy.

• (ii) For every permissible value of the energy $\mathcal {E}$, all the orbits $({\textbf{X}}(t),{\textbf{Y}}(t))$ under Hamilton’s equations with the Hamiltonian $\mathcal {H}_\textrm{planar}$ are confined to the Laplace plane, i.e., they have zero mutual inclination at the given (and fixed in advance) level of AMD.

• (iii) The Hamiltonian $\mathcal {H}_\textrm{planar}$ admits two periodic orbits (called below the ‘modes’ A and B) which correspond to the anti-aligned and aligned apsidal corotation states for the two planets in the planar case.

• (iv) The Hamiltonian $\mathcal {H}_\textrm{space}$ can be further decomposed as $\mathcal {H}_\textrm{space}=\mathcal {H}_{0,\textrm{space}}+\mathcal {H}_{1,\textrm{space}}$, where $\mathcal {H}_{0,\textrm{space}}$ also admits $J=(X_2^2+Y_2^2+X_3^2+Y_3^2)/4$ as a second integral. Then, the Hamiltonian

  $$\begin{aligned} \mathcal {H}_\textrm{int}=\mathcal {H}_\textrm{planar}+\mathcal {H}_{0,\textrm{space}} \end{aligned}$$

  (3)

  is integrable, and it has a formal structure similar to $\mathcal {H}_\textrm{planar}$. In particular, the existence of periodic orbits of the type A and B can be demonstrated for the Hamiltonian $\mathcal {H}_\textrm{int}$ using the integrability property, in the same way as for $\mathcal {H}_\textrm{planar}$.

• (v) Focusing, now, on the full secular Hamiltonian

  $$\begin{aligned} \mathcal {H}_\textrm{sec}=\mathcal {H}_\textrm{int}+\mathcal {H}_{1,space} \end{aligned}$$

  (4)

  a Birkhoff-like normal form construction (Sect. 3) allows to demonstrate (up to an exponentially small error) that the periodic orbits A, B (accompanied by neighboring quasi-periodic orbits) continue to exist in the full 3D regime. Hence, we call these orbits the 3D apsidal corotation states.

• (vi) We parameterize the level of non-coplanarity (mutual inclination $i_2+i_3$) of the system in terms of the (constant) energy level of the full Hamiltonian $\mathcal {E}=\mathcal {H}_\textrm{sec}$. One way to regard the connection between the value of $\mathcal {E}$ and the level of non-coplanarity of the orbits is by noting that the energy grows in absolute value nearly as a quadratic function of the planetary eccentricities, i.e., nearly proportionally to a linear combination of $e_2^2$ and $e_3^2$. Restricted to a suitably chosen Poincaré surface of section (see Sect. 2), the constant energy condition $\mathcal {E}=\mathcal {H}_\textrm{sec}$ yields an ellipsoid-like surface. Also (see Sect. 3), for fixed $\textrm{AMD}$, the condition of constant mutual inclination yields the contour of a function also quadratic in the orbital eccentricities. Then, for any fixed value of the energy $\mathcal {E}$, there are two values of the mutual inclination $i_\textrm{mut}=i_2+i_3$, namely $i_\textrm{mut}^{\min }(\mathcal {E})$, $i_\textrm{mut}^{\max }(\mathcal {E})$ such that, the corresponding contours of constant mutual inclination come tangent to the ellipsoidal surface of constant energy $\mathcal {E}=\mathcal {H}_\textrm{sec}$. As a consequence, for any value of the mutual inclination in the interval $i_\textrm{mut}^{\min }(\mathcal {E})\le i_\textrm{mut}\le i_\textrm{mut}^{\max }(\mathcal {E})$ there is an energetically permissible domain of motions. Both $i_\textrm{mut}^{\min }(\mathcal {E})$ and $i_\textrm{mut}^{\max }(\mathcal {E})$ are monotonically increasing functions of the energy. Thus, selecting a particular level value of the energy $\mathcal {E}$ fixes the overall level of mutual inclinations allowed at the energy $\mathcal {E}$. The minimum possible energy $\mathcal {E}_{\min }$ is defined by the condition that the level surface $\mathcal {H}_\textrm{sec}=\mathcal {E}_{\min }$ is tangent to the level surface of minimum possible mutual inclination $i_\textrm{mut}=0$ (planar case), while all other points of the surface $i_\textrm{mut}=0\,$, except for the points of tangency, are in the interior of the surface $\mathcal {H}_{\text {sec}}=\mathcal {E}_{\min }$. Solving for these conditions allows to specify the value of $\mathcal {E}_{\min }$.

In conclusion, the level of mutual inclination for all orbits increases in general with the quantity $\delta \mathcal {E}=\mathcal {E}-\mathcal {E}_{\min }$, where $\mathcal {E}_{\min }\le \mathcal {E}\le 0$. Thus, the lowermost value of $\delta \mathcal {E}$ is $\delta \mathcal {E}_{\min }=0$, while the highermost limit is $\delta \mathcal {E}_{\max }=-\mathcal {E}_{\min }$, i.e., $\mathcal {E}=0$. At this latter limit, the available phase space shrinks to a point with $e_2(t)=e_3(t)=0$. This means a unique possible orbital configuration of two mutually inclined circular planetary orbits. We call this the limiting trajectory of the Lidov–Kozai regime. Actually, in Sect. 3 we show that, for negative energies $\mathcal {E}$ close to zero, the phase space acquires a structure reminiscent to the one of the non-integrable Lidov–Kozai case of the restricted three-body problem (i.e., non-intersecting trajectories examined in a higher than quadrupolar development of the disturbing function, see Libert et al. (2011)). The typical behavior of the trajectories in the Lidov–Kozai regime is to (quasi-)periodically exchange eccentricity with mutual inclination. However, such a coarse illustration of the dynamics is rather simplistic in the case of non-hierarchical (in distance ratios or masses) two-planet systems; in reality, the dynamics around the central Lidov–Kozai periodic orbit is highly unstable and the corresponding phase space turns to exhibit strong chaos.

In summary, our focus in the present paper is on describing, with sufficient detail, the observed transition in the structure of the phase space, as the parameter $\delta \mathcal {E}$ increases from its lowermost limit, corresponding to a nearly planar orbital configuration, to the highermost limit, corresponding to nearly circular orbits with a high degree of mutual inclination. As expected in the study of any dynamical system, structural changes in the phase space are associated with the birth, bifurcations and stability evolution of the most important periodic orbits of the system. We already mentioned that these are the ACs, in the nearly planar limit, and the Lidov–Kozai orbits, in the maximum mutual inclination limit. We follow the evolution and the connections between these periodic orbits, as well as other ones emerging in half the way between the two limiting regimes. We work with analytical estimates, as well as with a numerical example inspired by the $\upsilon$-Andromedae system, which is reasonably far from any hierarchical limit. On the other hand, Sect. 4 contains results from a numerical investigation referring to different choices for the mass and semi-major axis ratios, representing every one of all possible cases of hierarchical models that could arise in the problem under consideration. As a rough guide, the phenomena discussed below should cover most cases of interest in the range of mass ratios $1/10\le m_2/m_3 \le 10\,$, distance ratios $1/7\le r_2/r_3\le 1/3$ and mutual inclination $0\le i_\textrm{mut}\le 45^\circ \,$.

The paper is organized as follows. Section 2 describes all the steps leading to the finally adopted secular Hamiltonian model. This includes the introduction of a suitable ‘book-keeping’, the chain of canonical transformations leading to the Hamiltonian expressed in Poincaré canonical variables, as well as a number of precision tests about the order of the multipole truncation, comparison with the Laplace–Lagrange series, etc. Section 3 analyzes the dynamics and the phase portraits under the secular Hamiltonian model computed as in Sect. 2. Sections 3.2.1 and 3.2.2 discuss the phase-space properties of the integrable approximations $\mathcal {H}_\textrm{planar}$ and $\mathcal {H}_\textrm{int}$, and introduce the basic terminology for the orbits of the apsidal corotation type (modes A and B). Section 3.2.3 deals, instead, with an analysis of the phase-space structure under the full (non-integrable) Hamiltonian (2). This includes the semi-analytical method (via a normal form) used to demonstrate the continuation of the A and B modes as well as to produce the time series yielding the evolution of the eccentricity vectors for both planets along these modes. Section 3.3 discusses numerical results on the form of the surfaces of section as the energy $\mathcal {E}$ (or the parameter $\delta \mathcal {E}$) increases, showing the sequence of bifurcations that connect the apsidal corotation with the Lidov–Kozai regime. Section 3.4 contains results related to the ‘Kozai mechanism’, i.e., the transition from linear stability to instability for the inclined circular orbit of one of the two planets. Numerical results are compared to an analytical approximation obtained in the framework of the quadrupolar approximation. Section 4 contains the investigation of how the phase portraits change in cases with different distance or mass ratios of the planets, covering various hierarchical models ($r_2/r_3=1/7,\,1/3,\,$ $m_2/m_3=1/10,\,1/3,\,1,\,3,\,10\,$). Section 5 summarizes of the main conclusions from the present study.

2 Hamiltonian model

2.1 On the formalism

The spectrum of methods and formalisms used in the study of the secular planetary three-body problem is nearly as wide as the literature itself on the subject. Different proposals distinguish between cases in which the problem is considered hierarchical (e.g., in the distances $r_2/r_3 \ll 1\,$; see Brouwer 1959; Ford et al. 2000; Migaszewski and Goździewski 2011), or non-hierarchical (see Henrard and Libert 2004; Naoz et al. 2013). The choice of variables and/or proposed representation of the phase space of the system often reflects whether the focus is on phenomena related to apsidal-corotations (see Beaugé et al. 2003; Laughlin et al. 2002; Lee and Peale 2003), or to the Kozai instability (see Kozai 1962; Libert and Delsate 2012; Libert and Tsiganis 2009; Lithwick and Naoz 2011). Different formalisms also stem from:

• (i) The type of Hamiltonian expansion: this can be performed as the usual (Laplace–Lagrange) series expansion in powers of the planets’ eccentricities and inclinations (see, for example, Murray and Dermott 1999; Libert and Henrard 2007) or as a Legendre multipolar expansion (see Naoz 2016; Libert and Delsate 2012 and references therein). Although the two types of expansions are equivalent in the limit of infinite order of the expansion, different truncations (even with the same type of series) can lead to quantitatively, and even qualitatively, different results. Libert and Henrard (2007) discuss the question of the correct order of truncation in the framework of the Laplace–Lagrange expansion. Migaszewski and Goździewski (2011), Naoz et al. (2013), Naoz (2016) discuss various truncated multipolar models for spatial hierarchical systems. To our knowledge, there is no literature on a comparison between the results obtained by truncated models with the two types of expansion for non-hierarchical spatial systems.

• (ii) Method of averaging: different secular models are obtained by a different choice of method for averaging the Hamiltonian with respect to the fast angles. Such methods include: a) averaging “by scissors”, i.e., by just dropping-off the Hamiltonian all fast periodic terms (see, for example, Libert and Henrard 2007; Henrard and Libert 2004). b) “Closed-form” averaging (e.g., Migaszewski and Goździewski 2011). This method has the benefit of avoiding expansions in the orbital eccentricities (whose convergence becomes limited due to the limit in the series inversion of Kepler’s equation, see, for example Szeto and Lambeck 1982). However, closed-form averaging can only be performed after a multipolar expansion of the Hamiltonian. Thus, the method is particularly suited for systems hierarchical with respect to the planetary distances, while its precision in the case of non-hierarchical systems is an open issue (see some results in Sect. 4). c) Numerical computation (e.g., by Gauss’ method) of the quadratures involved in the averaging (see Thomas and Morbidelli 1996; Migaszewski and Goździewski 2008). This method is particularly suited for systems with intersecting trajectories, owing to theorems (see, for example, Gronchi and Milani 1998) establishing the continuity of the secular equations of motion at the points of intersection, which are singularities of the integrand functions appearing in the quadratures. d) Elimination of the mean anomalies from the Hamiltonian via a canonical transformation (see Brouwer 1959; Naoz et al. 2013). From the theoretical point of view, the use of a canonical transformation is imperative when precision of second order in the planetary masses is sought for. It should be stressed that while $\mathcal {O}(m^2)$ expansions are straightforward to obtain in the framework of the Laplace–Lagrange series (see Locatelli and Giorgilli 2000), their counterpart in the form of closed-form series is an open question. In fact, closed-form averaging in the framework of the three-body problem requires the use of some ‘relegation’ technique (see Palacián et al. 2006), or alternatives as those recently proposed in Cavallari and Efthymiopoulos (2022).

• (iii) Choice of coordinates: Several sets of variables have been proposed to visualize and study the phase space of secular motions. Examples are: $(e_2\,\sin (\omega _2),\,e_3\,\sin (\omega _3))$ (see, for instance, Libert and Henrard 2007; Migaszewski and Goździewski 2011; Libert and Tsiganis 2009); $(e_j\cos (\Delta \varpi ),\,e_j\sin (\Delta \varpi ))\,$, $j=2,\,3\,$ (see Michtchenko and Malhotra 2004); $(e_2\,\cos (2\omega _2),\,e_3\cos (\Delta \omega ))$ (see Libert and Delsate 2012); $(e_2\,\cos (\Delta \varpi ),\,e_3\,\cos (2\,\omega _2)\,)$ (see Migaszewski and Goździewski 2011; Michtchenko et al. 2006); $(e_j\,\cos (\omega _j),\,e_j\,\sin (\omega _j))\,$ (see Thomas and Morbidelli 1996). Among the motivations behind the choice of a particular set of variables are: a) the treatment of singular cases (e.g., $e_2=0$, $e_3=0$, in which the longitude of the perihelium is no longer defined), b) the possibility to include all the main families of possible orbital states in one plot. As a characteristic example of the latter, see figure 3 of Michtchenko et al. (2006); one can remark, however, the complexity involved in properly deciphering the information given in that figure, which is evident from the accompanying caption.

Our own choice regarding the points (i) to (iii) above is: as regards the choices of the type of Hamiltonian expansion and the method of averaging, we base most of our results on a closed-form averaging of a multipolar expansion (in powers of the distance ratio $r_2/r_3<1$) of the Hamiltonian (1). Besides its compact form, convenient for analytic studies (as, for example, in Sect. 3.4), such a model, truncated at a sufficiently high multipole order, circumvents the problem of slow convergence of the Laplace–Lagrange series for highly eccentric orbits, without compromising precision even far from the hierarchical limit (e.g., for $r_2/r_3\sim 0.3\,$). Indeed, the closed-form averaging is based on a multipolar expansion, which produces a smaller number of terms in the Hamiltonian with respect to the Laplace–Lagrange expansion, rendering the procedure faster. Moreover, the closed-form averaging allows to avoid the singularity of Kepler’s equation. The rational dependence of the Hamiltonian on the quantity $\sqrt{1-e_3^2}$ implies that any expansion in the eccentricities performed after the closed-form averaging is convergent in the entire domain $|e_2 | < 1$, $|e_3 | < 1\,$. However, we have to recall that multipolar expansions have singularities for values of the eccentricities $e_2$ and $e_3$ for which the distance of the apocenter of the inner planet is equal to the distance of the pericenter of the outer planet, i.e., such that $a_2 (1+e_2) = a_3 (1- e_3)\,$. In order to specify the suitable order of multipole truncation, in Sect. 2 we make a comparison of the phase portraits obtained via the secular Hamiltonian arrived at by the above method versus those obtained by a scissors’ averaging of the Laplace–Lagrange series truncated at order 10 in the eccentricities. Also, in obtaining the final Hamiltonian we introduce a book-keeping procedure for the Jacobi reduction of the nodes, which, as mentioned already, leads to a convenient decomposition of the Hamiltonian as in Eq. (2). As regards the choice of coordinates, we illustrate all phase portraits using the usual Poincaré surface of section $(e_2\cos \omega _2,e_2\sin \omega _2)$ every time when $\omega _3=\pi$, $\dot{\omega _3}\ge 0$. The sequence of canonical transformations leading to such a representation of phase portraits is described in Sect. 2. Owing to the conservation of angular momentum, one can easily see (Cushman and Bates 1997) that the phase space of the integrable models $\mathcal {H}_\textrm{planar}$ and $\mathcal {H}_\textrm{int}$ is the sphere $S^2$ (instead of ${\mathbb {R}}^4$, as generically true for Hamiltonian systems with two degrees of freedom). Furthermore (see Sect. 3.2.2), some points of the sphere ‘inflate’ to curves in the Poincaré surface of section defined as above. Finally, using an appropriate set of variables whose Poisson algebra admits the angular momentum as a Casimir, the two modes A and B are seen to be separated by a meridian circle in the sphere $S^2$, which, nevertheless, does not correspond to a dynamical separatrix (since the integrable model contains no unstable periodic orbits). We devote some effort to carefully describe these phenomena, which are often found to generate confusion in the literature when use is made of some of the components of the planets’ eccentricity vectors to describe the structure of the problem’s phase space.

2.2 Averaged Hamiltonian

We focus mostly on the properties of a secular model $\mathcal {H}_\textrm{sec}$ for the Hamiltonian (1) obtained by performing averaging with respect to the fast angles just ‘by scissors’. We denote by $(a,e,i,M,\omega ,\Omega )$ the Keplerian elements of a body (semi-major axis, eccentricity, inclination, mean anomaly, argument of the periastron, argument of the nodes), and by $\lambda =M+\omega +\Omega$, $\varpi =\omega +\Omega$ the mean longitude and longitude of the periastron, respectively. A secular model can be obtained after averaging $\mathcal {H}$ with respect to the fast angles $M_2\,$, $M_3\,$:

$$\begin{aligned} \begin{aligned} \mathcal {H}_\textrm{sec}&={1\over 4\pi ^2} \int _{0}^{2\pi }\!\!\!\int _{0}^{2\pi } \mathcal {H}({\varvec{r}_2},{\varvec{r}_3},{\varvec{p}_2},{\varvec{p}_3}) \,\textrm{d}M_{2}\,\textrm{d}M_{3} \\&= {1\over 4\pi ^2}\int _{0}^{2\pi }\!\!\!\int _{0}^{2\pi } \left( \left( {1\over m_0}+{1\over m_2}\right) \frac{{\varvec{p}_2}^2}{2}-\frac{\mathcal {G}\, m_0\, m_2}{r_2} \right) \,\textrm{d}M_{2}\,\textrm{d}M_{3}\\&\quad +{1\over 4\pi ^2}\int _{0}^{2\pi }\!\!\!\int _{0}^{2\pi } \left( \left( {1\over m_0}+{1\over m_3}\right) \frac{{\varvec{p}_3}^2}{2}-\frac{\mathcal {G}\, m_0\, m_3}{r_3} \right) \,\textrm{d}M_{2}\,\textrm{d}M_{3}\\&\quad +{1\over 4\pi ^2}\int _{0}^{2\pi }\!\!\!\int _{0}^{2\pi } \frac{{\varvec{p}_2}\cdot {\varvec{p}_3}}{ m_0}\textrm{d}M_{2}\,\textrm{d}M_{3} -{1\over 4\pi ^2}\int _{0}^{2\pi }\!\!\!\int _{0}^{2\pi }\frac{\mathcal {G}\, m_2\, m_3}{|{\varvec{r}_2}-{\varvec{r}_3}|}\,\textrm{d}M_{2}\,\textrm{d}M_{3}. \end{aligned} \end{aligned}$$

(5)

We have

$$\begin{aligned} \begin{aligned}&{1\over 4\pi ^2}\int _{0}^{2\pi }\!\!\!\int _{0}^{2\pi } \left( \left( {1\over m_0}+{1\over m_2}\right) \frac{{\varvec{p}_2}^2}{2}-\frac{\mathcal {G}\, m_0\, m_2}{r_2} \right) \,\textrm{d}M_{2}\,\textrm{d}M_{3}\\&\qquad +{1\over 4\pi ^2}\int _{0}^{2\pi }\!\!\!\int _{0}^{2\pi } \left( \left( {1\over m_0}+{1\over m_3}\right) \frac{{\varvec{p}_3}^2}{2}-\frac{\mathcal {G}\, m_0\, m_3}{r_3} \right) \,\textrm{d}M_{2}\,\textrm{d}M_{3} \\&\quad =-{\mathcal {G} m_0 m_2\over 2a_2}-{\mathcal {G} m_0 m_3\over 2a_3}+ {\mathcal {G} m_2^2\over 2a_2}+{\mathcal {G} m_3^2\over 2a_3}~~. \end{aligned} \end{aligned}$$

(6)

The indirect part of the disturbing function, depending on the product ${\varvec{p}_2}\cdot {\varvec{p}_3}$, yields zero average

$$\begin{aligned} {1\over 4\pi ^2}\int _{0}^{2\pi }\!\!\!\int _{0}^{2\pi }\frac{{\varvec{p}_2}\cdot {\varvec{p}_3}}{ m_0}\,\textrm{d}M_{2}\,\textrm{d}M_{3} ={1\over 4\pi ^2} {\mathcal {G}\,m_2 m_3\over \sqrt{a_2^3\,a_3^3}} \int _{0}^{2\pi }\!\!{\textrm{d}{\varvec{r}_2}\over \textrm{d}M_2} \,\textrm{d}M_{2} \int _{0}^{2\pi }\!\!{\textrm{d}{\varvec{r}_3}\over \textrm{d}M_3} \,\textrm{d}M_{3}=0. \nonumber \\ \end{aligned}$$

(7)

To compute the average of the direct part $\mathcal {G}m_2m_3/|{\varvec{r}_2}-{\varvec{r}_3}|$, we assume a dynamical regime of the planetary system in which the distance ratio $r_2/r_3$ (where $r_{i}=|{\varvec{r}}_{i}|\,$, $i=2,3$) remains always smaller than unity. Then, the quantity $\mathcal {G}m_2m_3/|{\varvec{r}_2}-{\varvec{r}_3}|$ admits a convergent Legendre multipolar expansion in powers of the quantity $r_2/r_3<1$:

$$\begin{aligned}&-{1\over 4\pi ^2}\int _{0}^{2\pi }\!\!\!\int _{0}^{2\pi }\frac{\mathcal {G}\, m_2\, m_3}{|{\varvec{r}_2}-{\varvec{r}_3}|} \,\textrm{d}M_{2}\,\textrm{d}M_{3}\, \\&\quad =-{\mathcal {G}\, m_2\, m_3\over 4\pi ^2} \int _{0}^{2\pi }\!\!\!\int _{0}^{2\pi }\!\!\left( \frac{1}{r_{3}} +\frac{{\varvec{r}}_{2}\cdot {\varvec{r}}_{3}}{r_{3}^3}-\frac{1}{2}\frac{r_2^2}{r_3^3}+\frac{3}{2}\frac{({\varvec{r}}_{2}\cdot {\varvec{r}}_{3})^2}{r_{3}^5} +\ldots \right) \,\textrm{d}M_{2}\,\textrm{d}M_{3}\, . \end{aligned}$$

We have:

$$\begin{aligned}&-{\mathcal {G}\, m_2\, m_3\over 4\pi ^2} \int _{0}^{2\pi }\!\!\!\int _{0}^{2\pi }\frac{1}{r_{3}} \textrm{d}M_1\textrm{d}M_2=-\frac{\mathcal {G}m_2 m_3}{a_3}\, ,{} & {} \int _{0}^{2\pi }\!\!\!\int _{0}^{2\pi }\frac{{\varvec{r}}_{2}\cdot {\varvec{r}}_{3}}{r_{3}^3}\,\textrm{d}M_{2}\,\textrm{d}M_{3}=0\,. \end{aligned}$$

We then need to compute

$$\begin{aligned} \mathcal {R}_\textrm{sec}= {1\over 4\pi ^2} \int _{0}^{2\pi }\int _{0}^{2\pi }-\frac{\mathcal {G}\, m_2\, m_3}{r_3}\left( -\frac{1}{2}\frac{r_2^2}{r_3^2}+\frac{3}{2}\frac{({\varvec{r}}_{2}\cdot {\varvec{r}}_{3})^2}{r_{3}^4} +\ldots \right) \,\textrm{d}M_{2}\,\textrm{d}M_{3}. \end{aligned}$$

(8)

Keeping only the lowest order term in the integrand of (8), proportional to $(r_2/r_3)^2\,$, leads to the so-called quadrupole approximation; the next truncation (up to terms proportional to $(r_2/r_3)^3\,$) is the octupole approximation, etc. The integrals of any multipole approximation can be computed in so-called closed form (i.e., without expansions in the eccentricities), by avoiding completely the series reversion of Kepler’s equation, using, instead, the change of variables $M_2\rightarrow u_2$ (eccentric anomaly), $M_3\rightarrow f_3\,$ (true anomaly). We have

$$\begin{aligned} \begin{aligned}&\textrm{d}M_2=(1-e_2\cos u_2)\,\textrm{d}u_2,&\quad&\textrm{d}M_3={r_3^2\over a_3^2\sqrt{1-e_3^2}} \, \textrm{d}f_3,\\&r_2=a_2(1-e_2\cos (u_2)),&\quad&\frac{1}{r_3}=\frac{1+e_3\cos (f_3)}{a_3\,(1-e_3^2)}. \end{aligned} \end{aligned}$$

(9)

Replacing the above expressions in (8) and performing all trigonometric reductions, we find a trigonometric polynomial series containing only terms of the form $\cos (K_2u_2+K_3f_3+\ldots )$, with $K_2,K_3$ integers. This implies that the average can be computed by just scissor-cutting all the terms in the integrand of  (8) for which $|K_2|+|K_3|\ne 0$. This leads to a closed-form expression for the secular Hamiltonian

$$\begin{aligned} \begin{aligned} \mathcal {H}_\textrm{sec}&=-{\mathcal {G} m_0 m_2\over 2a_2}-{\mathcal {G} m_0 m_3\over 2a_3}+{\mathcal {G} m_2^2\over 2a_2}+{\mathcal {G} m_3^2\over 2a_3}-{\mathcal {G} m_2 m_3\over a_3}\\&\quad +\mathcal {R}_\textrm{sec}(a_2,a_3,e_2,e_3,i_2,i_3,\omega _2,\omega _3,\Omega _2-\Omega _3)~~. \end{aligned} \end{aligned}$$

(10)

The following are some relevant remarks regarding the Hamiltonian (10):

Remark 1

The averaging (8) yields a valid secular model only when the system is assumed to be far from any low-order mean-motion resonance. By ‘low’ it is implied that no resonance condition of the form

$$\begin{aligned} |K_2n_2+K_3n_3|<\mathcal {O}\left( (\mathcal {G}m_0\mu )^{1/2}/a_2^{3/2}\right) ~~, \end{aligned}$$

should be satisfied, with $K_2,\,K_3$ integers, $\mu =\max (m_2/m_0,m_3/m_0)$, $n_2=\left( \mathcal {G}(m_0+m_2)\right.$$\left. /a_2^3\right) ^{1/2}$, $n_3=\left( \mathcal {G}(m_0+m_3)/a_3^3\right) ^{1/2}$, for an order $|K_2|+|K_3|$ inferior or equal to the order $N_P$ of the multipole expansion in Eq. (8). Formally, such a requirement reflects the fact that the averaging by scissors serves to substitute the complete procedure of first-order averaging, which involves a canonical transformation to properly eliminate from the Hamiltonian the fast angles $M_2,\,M_3$. Such a transformation can be defined in closed form (see, for example, Cavallari and Efthymiopoulos 2022), but it involves divisors of the form $K_2n_2+K_3n_3$ which become very small near any mean motion resonance of order smaller or equal to $N_P$.

Remark 2

The function $\mathcal {R}_\textrm{sec}$ is trigonometric polynomial in the angles $\omega _2,\,\omega _3$, and $\Omega _2-\Omega _3$. The dependence on $\Omega _2,\,\Omega _3$ only by the difference $\Omega _2-\Omega _3$ is a consequence of the fact that the sum of the angular momenta $H_2+H_3$ perpendicularly to the system’s Laplace plane is an invariant of the motion of the complete Hamiltonian (1), which is preserved also in the secular Hamiltonian (10).

Remark 3

The function $\mathcal {R}_\textrm{sec}$ is polynomial in the quantities $a_2\,$, $e_2\,$, $e_3\,$, $\sin i_2\,$, $\sin i_3\,$, $\cos i_2\,$, $\cos i_3\,$, while it is rational in the quantities $a_3,\, \eta _3=\sqrt{1-e_3^2}$. As regards the dependence on $\sin i_2\,$, $\sin i_3\,$, $\cos i_2\,$, $\cos i_3\,$, this follows the symmetry that $\mathcal {R}_\textrm{sec}$ can only depend on the quantity $\cos (i_2+i_3)$, where $i_2+i_3=i_\textrm{mut}$ is the mutual inclination of the planetary trajectories. This symmetry is exploited when performing Jacobi reduction of the nodes in the Hamiltonian $\mathcal {H}_\textrm{sec}$ (see next subsection). On the other hand, the rational dependence on the quantity $\sqrt{1-e_3^2}$ implies that any expansion in the eccentricities performed after the closed-form averaging is convergent in the entire domain $|e_2|<1,|e_3|<1$. The possibility to perform an a posteriori expansion of the Hamiltonian $\mathcal {H}_\textrm{sec}$ in the planetary eccentricities proves useful in the theoretical analysis of the dynamical properties of such a Hamiltonian (see Sect. 3).

2.3 Book-keeping and Jacobi reduction of the nodes

Consider the set of Delaunay canonical variables:

$$\begin{aligned}&L_j=m_j\sqrt{\mathcal {G}\,m_0\,a_j}\, ,{} & {} l_j=M_j\, ,\nonumber \\&G_j=L_j\sqrt{1-e^2_j}\, ,{} & {} g_j=\omega _j\, , \nonumber \\&H_j=G_j\cos (i_j)\, ,{} & {} h_j=\Omega _j\, \end{aligned}$$

(11)

with $j=2,3$. Since $a_j=a_j(L_j)$, $e_j=e_j(L_j,G_j)$, $i_j=i_j(L_j,G_j,H_j)$, substituting the corresponding expressions in the Hamiltonian $\mathcal {H}_\textrm{sec}$ (Eq. (10)) leads to a Hamiltonian model with six degrees of freedom

$$\begin{aligned} \mathcal {H}_\textrm{sec}\equiv \mathcal {H}_\textrm{sec}(L_2,L_3,G_2,G_3,H_2,H_3,{g_2,g_3,h_2-h_3})~~. \end{aligned}$$

(12)

Since the angles $M_2,M_3$ are ignorable, the Delaunay momenta $L_2,L_3$ are integrals of motion under the Hamiltonian flow of $\mathcal {H}_\textrm{sec}$. There are two more integrals of motion in involution with $L_2,L_3$, corresponding to the two components of the total angular momentum, normal and parallel to the Laplace plane. For the normal component we have $L_z=H_2+H_3=C=const$, while for the parallel component we have $L_{\parallel }=G_2\sin (i_2(L_2,G_2,H_2))-G_3\sin (i_3(L_3,G_3,H_3))=0$. Thus, with an appropriate reduction, we can reduce the system to a Hamiltonian with two degrees of freedom. This reduction, implemented on the invariant manifold $L_z=C$, $L_\parallel =0$ is called the Jacobi reduction of the nodes (Jacobi 1842) and it exploits one more invariance of the system: since $L_\parallel =0$ we necessarily have $\Omega _2-\Omega _3=\pi =$const.

In our analytical treatment of the Hamiltonian $\mathcal {H}_\textrm{sec}$ it turns convenient to perform the Jacobi reduction of the nodes by simultaneously introducing two ‘book-keeping symbols’, $\varepsilon$, $\eta$ (Efthymiopoulos 2012), both with numerical values equal to 1, whose role is the following: i) $\varepsilon$ keeps track of the order of a certain term in the planetary eccentricities and inclinations, ii) $\eta$ separates Hamiltonian terms which depend on powers of the quantity $\cos (i_2+i_3)$ from those terms which do not depend on the mutual inclination $i_\textrm{mut}=i_2+i_3$.

More specifically, we Jacobi-reduce the Hamiltonian $\mathcal {H}_\textrm{sec}$ by the following steps:

Step 1: Canonical transformation. Similarly as in Libert and Henrard (2007), we introduce the canonical transformation

$$\begin{aligned}&\Lambda _2=L_2\, ,{} & {} \lambda _2=M_2+\omega _2+\Omega _2\, ,\nonumber \\&\Lambda _3=L_3\, ,{} & {} \lambda _3=M_3+\omega _3+\Omega _3\, ,\nonumber \\&W_2=L_2-G_2\, ,{} & {} w_2=-\omega _2\, , \nonumber \\&W_3=L_3-G_3\, ,{} & {} w_3=-\omega _3\, , \nonumber \\&R_2=L_2-H_2\, ,{} & {} \theta _{r2}=\Omega _3-\Omega _2\, , \nonumber \\&R_3=L_2+L_3-H_2-H_3=\textrm{AMD}\, ,{} & {} \theta _{r3}=-\Omega _3\, . \end{aligned}$$

(13)

Inverting the transformation (13) and substituting the result into the Hamiltonian (12), the secular Hamiltonian obtains the form:

$$\begin{aligned} \mathcal {H}_{\sec }\equiv \mathcal {H}_\textrm{sec}(\Lambda _2,\Lambda _3,W_2,W_3,R_2,R_3,w_2,w_3,\theta _{r2})~~. \end{aligned}$$

(14)

Step 2: Jacobi reduction with book-keeping. Hamilton’s equations for the Hamiltonian (14) yield ${\dot{\Lambda }}_2={\dot{\Lambda }}_3=0$ (constancy of the semi-major axes $a_2$, $a_3$), as well as ${\dot{R}}_3=0$ (constancy of $L_z$, implying the constancy of the AMD). The crucial point stems from the following invariance property of the Hamiltonian (14):

$$\begin{aligned} {\dot{\theta }}_{r2}= {\partial \mathcal {H}_\textrm{sec}(\Lambda _2,\Lambda _3,W_2,W_3,R_2,R_3=\textrm{AMD},w_2,w_3,\theta _{r2}=\pi )\over \partial R_2}=0 \end{aligned}$$

(15)

corresponding to the invariance in time of the relation $\theta _{r2}=\Omega _3-\Omega _2=\pi$ for all trajectories. Equation (15), however, implies that when the substitution $\Omega _3=\Omega _2+\pi$ is made in the Hamiltonian (14), the resulting expression becomes independent of $R_2$, and the reduced set of Hamilton’s equations

$$\begin{aligned}&{\dot{w}}_2={\partial \mathcal {H}_\textrm{sec}(\Lambda _2,\Lambda _3,W_2,W_3,R_3=\textrm{AMD},w_2,w_3,\theta _{r2}=\pi ) \over \partial W_2} \, , \nonumber \\&{\dot{w}}_3={\partial \mathcal {H}_\textrm{sec}(\Lambda _2,\Lambda _3,W_2,W_3,R_3=\textrm{AMD},w_2,w_3,\theta _{r2}=\pi ) \over \partial W_3} \, , \nonumber \\&{\dot{W}}_2=-{\partial \mathcal {H}_\textrm{sec}(\Lambda _2,\Lambda _3,W_2,W_3,R_3=\textrm{AMD},w_2,w_3,\theta _{r2}=\pi ) \over \partial w_2} \, , \nonumber \\&{\dot{W}}_3=-{\partial \mathcal {H}_\textrm{sec}(\Lambda _2,\Lambda _3,W_2,W_3,R_3=\textrm{AMD},w_2,w_3,\theta _{r2}=\pi ) \over \partial w_3} \, , \end{aligned}$$

(16)

remains valid. Note that $\Lambda _2,\,\Lambda _3$ in the above secular equations of motion are constant, and can be effectively treated as parameters, depending on the (also constant) parameters $a_2,a_3$. Also, the terms

$$\begin{aligned} -{\mathcal {G} m_0 m_2\over 2a_2}-{\mathcal {G} m_0 m_3\over 2a_3}+{\mathcal {G} m_2^2\over 2a_2}+{\mathcal {G} m_3^2\over 2a_3}-{\mathcal {G} m_2 m_3\over a_3} \end{aligned}$$

appearing in Eq. (10), do not contribute to the secular equations of motion, and can be omitted from further analysis, by just renaming $\mathcal {R}_\textrm{sec}\rightarrow \mathcal {H}_\textrm{sec}$. Finally, Eqs. (16) allow to compute the secular evolution of only the eccentricity vectors of the two planets. To obtain the evolution in inclination, instead, we use the following relations, obtained directly from the conservation of the angular momentum:

$$\begin{aligned} \cos (i_2)={\Lambda _2^2(1-e_2^2)-\Lambda _3^2(1-e_3^2)+L_z^2\over 2L_z\Lambda _2\sqrt{1-e_2^2}},~~~ \cos (i_3)={L_z-\Lambda _2\sqrt{1-e_2^2}\cos (i_2)\over \Lambda _3\sqrt{1-e_3^2}}~~. \nonumber \\ \end{aligned}$$

(17)

In the practical implementation of step 2, we perform the substitutions of the angle variables $\Omega _3=\Omega _2+\pi$, as well as $\omega _2=-w_2$, $\omega _3=-w_3$ in the Hamiltonian, but leave implicit the latter’s dependence on the action variables $(\Lambda _2,\Lambda _3,W_2,W_3,R_2,R_3)$ through the elements $(a_2,a_3,e_2,e_3,i_2,i_3)$. Owing to the non-dependence of the Hamiltonian on $R_2$ after the substitution $\Omega _3=\Omega _2+\pi$, we have that the Hamiltonian depends on the inclinations only through the trigonometric combination $\cos (i_2+i_3)=\cos (i_2)\cos (i_3)-\sin (i_2)\sin (i_3)\,$. Taking into account that

$$\begin{aligned} \left( 1- \cos (i_2)\cos (i_3)\right) =\mathcal {O}(\varepsilon ^2), \qquad \, \sin (i_2)\sin (i_3)=\mathcal {O}(\varepsilon ^2)~~, \end{aligned}$$

where the attribution of $\mathcal {O}(\varepsilon ^2)$ to an expression stands for ‘second order in the eccentricities and inclinations’, we then introduce the following book-keeping control identities:

$$\begin{aligned}&\cos (i_2)\cos (i_3)\!=\!\varepsilon ^2\eta \left( \cos (i_2)\cos (i_3)-1\right) +1\, ,&\!\!\!\!&\sin (i_2)\sin (i_3)\!=\!\varepsilon ^2\eta \sin (i_2)\sin (i_3)\, . \end{aligned}$$

(18)

We also use the substitution rule

$$\begin{aligned} \begin{aligned}&\sin (i_2)\sin (i_3)=\\&\quad \cos (i_2)\cos (i_3)+\frac{\Lambda _2\sqrt{1-e_2^2}}{2\,\Lambda _3\sqrt{1-e_3^2}} +\frac{\Lambda _3\sqrt{1-e_3^2}}{2\,\Lambda _2\sqrt{1-e_2^2}} -\frac{L_z^2}{2\,\Lambda _2\sqrt{1-e_2^2}\,\Lambda _3\sqrt{1-e_3^2}}. \end{aligned} \end{aligned}$$

(19)

Substituting the above expressions into the Hamiltonian, and truncating the resulting expression up to a preselected maximum order in book-keeping $N_\textrm{bk}$, by symmetry the terms in equal powers of the products $\sin (i_2)\sin (i_3)$ and $\cos (i_2)\cos (i_3)$ are opposite, and thus, they are canceled. Hence, the Hamiltonian resumes the form:

$$\begin{aligned} \mathcal {H}_\textrm{sec}=\sum _{s_1=0}^{N_\textrm{bk}/2}\eta ^{s_1}\varepsilon ^{2s_1}\mathcal {H}_{\textrm{sec},s_1}(e_2,e_3,w_2,w_3;a_2,a_3,L_z)~~. \end{aligned}$$

(20)

We finally restore the values $\varepsilon =1$ and $\eta =1$ of the book-keeping parameters, and write the truncated (up to book-keeping order $N_\textrm{bk}$) Hamiltonian as:

$$\begin{aligned} \mathcal {H}_\textrm{sec}=\mathcal {H}_\textrm{planar}+\mathcal {H}_\textrm{space} \end{aligned}$$

(21)

where

$$\begin{aligned} \begin{aligned}&\mathcal {H}_\textrm{planar}= \mathcal {H}_{\textrm{sec},0}(e_2,e_3,w_2-w_3;a_2,a_3), \\&\mathcal {H}_\textrm{space}= \sum _{s_1=1}^{N_\textrm{bk}/2}\mathcal {H}_{\textrm{sec},s_1}(e_2,e_3,w_2,w_3;a_2,a_3,L_z)~. \end{aligned} \end{aligned}$$

(22)

Note that the term $\mathcal {H}_\textrm{planar}$ depends only on the difference $w_2-w_3$, since, in the planar case, the sum $W_2+W_3$, which represents the total angular momentum deficit for planar orbits, is a conserved quantity.

Step 3: Expansion in the orbital eccentricities. We re-introduce the book-keeping:

$$\begin{aligned} e_2\rightarrow \varepsilon e_2,~~e_3\rightarrow \varepsilon e_3,~~ L_z \rightarrow \Lambda _2+\Lambda _3-\varepsilon ^2\textrm{AMD}. \end{aligned}$$

(23)

Substituting the above expressions in the Hamiltonian (22), and expanding the resulting expressions in powers of the book-keeping parameter $\varepsilon$, apart from a term depending only on the constants $a_2,\,a_3$ and $\textrm{AMD}$, we arrive at the truncated Hamiltonian:

$$\begin{aligned} \begin{aligned}&\mathcal {H}_\textrm{sec}=\mathcal {H}_\textrm{planar}+\mathcal {H}_\textrm{space} \\&\quad =\!\sum _{s_2=1}^{N_\textrm{bk}/2}\!\!\varepsilon ^{2 s_{2}} h_{s_{2}}(e_2,e_3, w_2-w_3;a_2,a_3) +\!\!\sum _{s_2=1}^{N_\textrm{bk}/2}\!\!\varepsilon ^{2 s_{2}}{\tilde{h}}_{s_2}(e_2,e_3,w_2,w_3;a_2,a_3,\textrm{AMD}). \end{aligned}\nonumber \\ \end{aligned}$$

(24)

Step 4: introduction of Poincaré variables: The Hamiltonian (24) is polynomial in the eccentricities $e_2,\,e_3$. It was already stressed that the series expansion with respect to the quantities $\eta _2=\sqrt{1-e_2^2}$ and $\eta _3=\sqrt{1-e_3^2}$ introduces no divergence in the polydisc $|e_2|<1$, $|e_3|<1$. Besides, the Hamiltonian satisfies the D’Alembert rules in the eccentricities: every trigonometric term in it is of the form $h_{\ell _2,\ell _3,k_2,k_3}(a_2,a_3)(\textrm{AMD})^{\ell _1}e_2^{\ell _2}e_3^{\ell _3}{\cos }(k_2w_2+k_3w_3)$, with $\ell _1$ positive integer, and the integers $\ell _2,\ell _3,k_2,k_3$ satisfying i) $\ell _2,\ell _3>0$, ii) $\ell _2+\ell _3\ge |k_2|+|k_3|$, and iii) $mod(\ell _2,2)=mod(k_2,2)$, $mod(\ell _3,2)=mod(k_3,2)$. Rules (i) to (iii) imply, now, that the Hamiltonian is polynomial in the Poincaré canonical variables stemming from the variables $(w_j,W_j)$, $j=2,3$ through the canonical transformation

$$\begin{aligned}&X_j= -\sqrt{2 \,W_j}\cos (w_j)\, ,{} & {} Y_j=\sqrt{2\,W_j}\sin (w_j)\, ,{} & {} j=2,3~~. \end{aligned}$$

(25)

To obtain a truncated polynomial series of the Hamiltonian in the variables $(X_j,Y_j)$, we substitute into the Hamiltonian (24) the expressions

$$\begin{aligned} \sin (w_j)={Y_j\over \sqrt{2W_j}},~~~\cos (w_j)=-{X_j\over \sqrt{2W_j}},~~~e_j=\sqrt{{2W_j\over \Lambda _j}-\varepsilon ^2\left( {W_j\over \Lambda _j}\right) ^2},~~j=2,3~ \end{aligned}$$

and expand the result in powers of the book-keeping parameter $\varepsilon$, up to the truncation order $N_\textrm{bk}$. This leads to an expression which no longer contains trigonometric functions of the angles $w_j$, while it still contains integer powers of the actions $W_j$. We then substitute $W_j\rightarrow (X_j^2+Y_j^2)/2$, and, finally, set back $\varepsilon =1$. In this way we arrive at the final secular model:

$$\begin{aligned} \begin{aligned}&\mathcal {H}_\textrm{sec}(X_2, X_3, Y_2, Y_3; \textrm{AMD})\\&\quad =\mathcal {H}_\textrm{planar}(X_2, X_3, Y_2, Y_3)+\mathcal {H}_\textrm{space}(X_2, X_3, Y_2, Y_3;\textrm{AMD}) \\&\quad =\!\!\!\! \sum _{\ell \in {\mathbb {N}}^4,|\ell |=2}^{N_\textrm{bk}} \mathcal {K}_{\textrm{planar},\ell }X_2^{\ell _1}\,X_3^{\ell _2}\,Y_2^{\ell _3}\,Y_3^{\ell _4} + \!\!\!\!\sum _{\ell \in {\mathbb {N}}^4,|\ell |=2}^{N_\textrm{bk}} \mathcal {K}_{\textrm{space},\ell }(\textrm{AMD})X_2^{\ell _1}\,X_3^{\ell _2}\,Y_2^{\ell _3}\,Y_3^{\ell _4}. \end{aligned}\nonumber \\ \end{aligned}$$

(26)

Note that, by symmetry, the value of the disturbing function (and hence of the secular Hamiltonian) remains invariant by the rotation of both planets’ argument of the pericenter by $\pi$, hence the secular Hamiltonian is necessarily even in the planetary eccentricities. This implies that the Hamiltonian (26) contains only even powers of the Poincaré variables $(X_j,Y_j)\,$, i.e., $\ell _1+\ell _2$ and $\ell _3+\ell _4$ are even. In particular, Hamilton’s equations take the form:

$$\begin{aligned} \begin{aligned}&\dot{X_i}=\frac{\partial \mathcal {H}_\textrm{sec}}{\partial Y_i} =Y_2\,F_{X_i,2}(X_2, X_3, Y_2, Y_3)+Y_3\,F_{X_i,3}(X_2, X_3, Y_2, Y_3),\\&\dot{Y_i}=-\frac{\partial \mathcal {H}_\textrm{sec}}{\partial X_i} =X_2\,F_{Y_i,2}(X_2, X_3, Y_2, Y_3)+X_3\,F_{Y_i,3}(X_2, X_3, Y_2, Y_3), \end{aligned} \end{aligned}$$

(27)

where the polynomials $F_{X_i,2}$, $F_{X_i,3}$, $F_{Y_i,2}$, $F_{Y_i,3}\,$, $i=2,\,3,$ are even, starting with constant terms.

2.4 Poincaré surface of section: precision tests

We will now discuss several precision tests, based on the method of comparison of phase portraits, which aim to establish which is the minimum multipole order, as well as minimum order in the eccentricities of the Hamiltonian $\mathcal {H}_\textrm{sec}$, such that the Hamiltonian, truncated at the above orders, represents with sufficient precision the dynamics at the timescales of the secular system produced by averaging of the original Hamiltonian (1).

In all the numerical examples which follow, we refer to the mass, periods and $\textrm{AMD}$ parameters as estimated for the $\upsilon$-Andromedae system. We adopt the value $\mathcal {G}=4 \pi ^2~AU^3/(yr^2 M_\odot )$ for Newton’s gravity constant, as well as the mass parameters $m_0 = 1.31 M_\odot$, $m_2 = 13.98 M_J$, $m_3 = 10.25 M_J$ ($M_\odot =$ mass of the Sun, $M_J=0.0009546, M_\odot =$ mass of Jupiter), and the semi-major axes $a_2 = 0.829$ AU, $a_3= 2.53$ AU (according to Table 13 of McArthur et al. 2010). For what concerns the constants $L_z$ or $\textrm{AMD}$, we estimate their values adopting for the two planets’ eccentricities, inclinations and arguments of periastron and of the nodes the values proposed in Table 1 of Deitrick et al. (2015), namely, $e_2 = 0.2445$, $e_3 = 0.316$, $i_2 = 11.347^\circ$, $i_3 = 25.609^\circ$, $\omega _2 = 247.629^\circ$, $\omega _3 = 252.991^\circ$, $\Omega _2 = 248.181^\circ$, $\Omega _3 = 11.425^\circ$. As emphasized in the introduction, there are great uncertainties in the observation as regards, in particular, the estimates on the planetary eccentricity and inclination vectors. However, starting with parameters as above, we can have a representative value for the system’s $\textrm{AMD}$, which then serves to analyze the secular orbital dynamics under different assumptions for the planets’ initial conditions. The value of the $\textrm{AMD}$ is estimated from the above data, by first computing the angular momentum vector with respect to a heliocentric frame of reference whose x-axis points to the observer, then computing the rotation matrix connecting the observer’s frame with Laplace’s reference frame, and finally rotating the positions and velocities of both planets to the Laplace frame of reference and re-calculating their inclinations and arguments of periastron and of the nodes. This yields the values $i_2 = 18.4748^\circ$, $i_3 = 14.6462^\circ$, $\omega _2 = 289.049^\circ$, $\omega _3 = 235.464^\circ$, $\Omega _2 = 217.318^\circ$, $\Omega _3 = 37.3176^\circ$. Then, we obtain $L_z=0.183101~AU^2 M_\odot /yr$, which, together with $\Lambda _2=0.0873819$, $\Lambda _3=0.111923$, leads to $\textrm{AMD}=\Lambda _2+\Lambda _3-L_z=0.0162044~AU^2 M_\odot /yr$.

The complete expression of the Hamiltonian (26) up to terms of order 4 in the eccentricity is provided as a supplementary online material.

2.4.1 Poincaré surface of section: definitions

To visualize phase portraits, use is made of a Poincaré surface of section $\mathcal {P}(\mathcal {E};\textrm{AMD})$ defined by the relations:

$$\begin{aligned} \begin{aligned} \mathcal {P}(\mathcal {E};\textrm{AMD})= \bigg \{&(X_2,Y_2,X_3,Y_3)\in {\mathbb {R}}^4:~ \mathcal {H}_\textrm{sec}(X_2,Y_2,X_3, Y_3=0;\textrm{AMD})=\mathcal {E},\\&Y_3=0,~{\dot{Y}}_3=-{\partial \mathcal {H}_\textrm{sec}(X_2,Y_2,X_3,Y_3=0;\textrm{AMD})\over \partial X_3}\ge 0,\\&\cos (i_{\max })\le \cos (i_\textrm{mut})(X_2,Y_2,X_3,Y_3=0;\textrm{AMD})\le 1\bigg \}, \end{aligned} \end{aligned}$$

(28)

where the mutual inclination $i_\textrm{mut}=i_2+i_3$ for fixed $\textrm{AMD}$, or angular momentum $L_z=\Lambda _2+\Lambda _3-\textrm{AMD}\,$, is given (consistently with Eq. (12) of Robutel 1995) by

$$\begin{aligned} \begin{aligned} \cos (i_\textrm{mut})&= \frac{L_z^2-\Lambda _2^2-\Lambda _3^2+\Lambda _2^2\,e_2^2+\Lambda _3^2\,e_3^2}{2\Lambda _2\Lambda _3\sqrt{1-e_2^2}\sqrt{1-e_3^2}}={\frac{L_z^2-G_2^2-G_3^2}{2 G_2 G_3}}\\&= \frac{L_z^2-\Lambda _2^2\left( 1-{X_2^2+Y_2^2\over 2\Lambda _2}\right) ^2 -\Lambda _3^2\left( 1-{X_3^2+Y_3^2\over 2\Lambda _3}\right) ^2}{2\Lambda _2\Lambda _3\left( 1-{X_2^2+Y_2^2\over 2\Lambda _2}\right) \left( 1-{X_3^2+Y_3^2\over 2\Lambda _3}\right) } \end{aligned} \end{aligned}$$

(29)

and the maximum possible mutual inclination consistent with the given $\textrm{AMD}$ is

$$\begin{aligned} i_{\max }= \cos ^{-1}\left( \frac{L_z^2-\Lambda _2^2-\Lambda _3^2}{2\Lambda _2\Lambda _3}\right) ~~. \end{aligned}$$

(30)

The phase portrait corresponding to the Poincaré surface of section at a fixed level of energy $\mathcal {E}$ is obtained numerically, by choosing several initial conditions $(X_2,Y_2)\in \mathcal {D}(\mathcal {E})\subset {\mathbb {R}}^2$, where $\mathcal {D}(\mathcal {E})$ is the domain of permissible initial conditions consistent with the definition of the surface of section as in Eq. (28). For each initial condition, we then iterate the corresponding orbit under the Hamiltonian $\mathcal {H}_\textrm{sec}$, and plot the consequent points $(X_2,Y_2)$, or, equivalently $(e_2\cos \omega _2,e_2\sin \omega _2)$, with

$$\begin{aligned} e_2\!=\!\left( \!1-\left( \!1-\frac{(X_2^2+Y_2^2)}{2\Lambda _2}\!\right) ^2\right) ^{1/2}\!\!\!\!\!\!\!,\quad \omega _2\!=\!\textrm{sgn}\!\left( \!\frac{-Y_2}{\sqrt{X_2^2+Y_2^2}}\!\right) \!\arccos \!\left( \!\frac{-X_2}{\sqrt{X_2^2+Y_2^2}}\!\right) \nonumber \\ \end{aligned}$$

(31)

every time the orbit intersects the surface of section. Note that the conditions $Y_3=0,{\dot{Y}}_3\ge 0$ imply i) that the Poincaré mapping defined by the successive iterates is symplectic, and ii) that the section definition corresponds physically to instants when the orbit of the outer planet crosses the pericenter $\omega _3=\pi$. The symplecticity of the Poincaré mapping, along with the straightforward physical interpretation, is the main motive for the choice of variables and for the visualization of phase portraits via the definition of the Poincaré section as above.

The domain $\mathcal {D}(\mathcal {E})$ is non-null in a range of energies $\mathcal {E}_{\min }\le \mathcal {E}<0$. The energy $\mathcal {E}_{\min }$ is computed as follows: consider the surface $\mathcal {I}_0$ of all possible points satisfying the section condition $Y_3=0$ as well as the lowermost limit of possible mutual inclination $i_\textrm{mut}=0$. The surface $\mathcal {I}_0$ is the sphere given by

$$\begin{aligned} \mathcal {I}_0=\bigg \{X_3^2+X_2^2+Y_2^2=2\textrm{AMD}\bigg \}~~. \end{aligned}$$

(32)

We then find the point $(X_{2,0},Y_{2,0})$ on the surface $\mathcal {I}_0$ where the energy is minimum by looking to the solutions of the system of equations

$$\begin{aligned} ~&~&{\partial \mathcal {H}_\textrm{sec}(X_2,Y_2,X_3^2=2\textrm{AMD}-X_2^2-Y_2^2,Y_3=0; \textrm{AMD})\over \partial X_2}=0 \nonumber \\ ~&~&{\partial \mathcal {H}_\textrm{sec}(X_2,Y_2,X_3^2=2\textrm{AMD}-X_2^2-Y_2^2,Y_3=0; \textrm{AMD})\over \partial Y_2}=0~, \end{aligned}$$

(33)

satisfying also the section condition

$$\begin{aligned} {\dot{Y}}_3=\left( -{\partial \mathcal {H}_\textrm{sec}(X_2,Y_2,X_3,Y_3=0;\textrm{AMD})\over \partial X_3}\right) _{X_3^2=2\textrm{AMD}-X_2^2-Y_2^2}\ge 0~~. \end{aligned}$$

(34)

We find two solutions $(X_{2,0},Y_{2,0}=0)\,$ and $(X_{2,1},Y_{2,1}=0)$ of the system of equations (33) and (34); however, only $(X_{2,0},Y_{2,0}=0)\,$ corresponds to the minimum value of the energy, i.e., $\mathcal {H}_{\text {sec}}(X_2=X_{2,0}, Y_2=Y_{2,0}=0, X_3= X_{3,0}=\sqrt{2\textrm{AMD}- X_{2,0}^2}, Y_3=0 )=\mathcal {E}_{\min }$, as verified by the Hessian matrix

$$\begin{aligned} \begin{pmatrix} &{}\frac{\partial ^2 \mathcal {H}_\text {sec}(X_3=\sqrt{2\text {AMD}- X_{2}^2- Y_{2}^2},Y_3=0)}{\partial X_2^2}\quad &{} \frac{\partial ^2 \mathcal {H}_\text {sec}(X_3=\sqrt{2\text {AMD}- X_{2}^2- Y_{2}^2},Y_3=0)}{\partial Y_2\partial X_2}\\ {} &{}\frac{\partial ^2 \mathcal {H}_\text {sec}(X_3=\sqrt{2\text {AMD}- X_{2}^2- Y_{2}^2},Y_3=0)}{\partial X_2\partial Y_2} \quad &{} \frac{\partial ^2 \mathcal {H}_\text {sec}(X_3=\sqrt{2\text {AMD}- X_{2}^2- Y_{2}^2},Y_3=0)}{\partial Y_2^2} \end{pmatrix}_{\!\!\!(X_2=X_{2,0}, Y_2=Y_{2,0})} \end{aligned}$$

which is positive definite. On the other hand, the solution $(X_{2,1},\,Y_{2,1}=0,\,X_{3,1}=\sqrt{2\textrm{AMD}- X_{2,1}^2})$ corresponds to the energy $\mathcal {E}_{2,3}$, and yield a negative definite Hessian matrix. Thus, $\mathcal {E}_{2,3}$ corresponds to the maximum energy for which we can have co-planar orbits (see discussion below).

The uniqueness of the solution realizing the minimal value of the energy can be understood by the following argument: for every fixed value of the energy $\mathcal {E}$, sufficiently close to the origin the restriction of the manifold of constant energy to the section $Y_3=0$

$$\begin{aligned} \mathcal {M}(\mathcal {E})=\bigg \{(X_2,Y_2,X_3)\in {\mathbb {R}}^3: \mathcal {H}_\textrm{sec}(X_2,X_3,Y_2,Y_3=0;\textrm{AMD})=\mathcal {E}\bigg \} \end{aligned}$$

(35)

yields a surface. We will show in the next section that, for energies smaller than a suitably defined threshold, the surface $\mathcal {M}(\mathcal {E})$ is close to the surface of an integrable model

$$\begin{aligned} \mathcal {M}_\textrm{int}(\mathcal {E})=\bigg \{(X_2,Y_2,X_3)\in {\mathbb {R}}^3: \mathcal {H}_\textrm{int}(X_2,X_3,Y_2,Y_3=0;\textrm{AMD})=\mathcal {E}\bigg \} \end{aligned}$$

(36)

where $\mathcal {H}_\textrm{int}$ contains $\mathcal {H}_\textrm{planar}$ as well as a part of $\mathcal {H}_\textrm{space}$. The manifolds $\mathcal {M}_\textrm{int}(\mathcal {E})$, in turn, are ellipsoidal-like closed convex surfaces. The ellipsoidal form follows from the fact that the leading order terms of the restriction of $\mathcal {H}_\textrm{int}$ to the section $Y_3=0$ yield a negative-definite quadratic form¹

$$\begin{aligned} \mathcal {H}_\textrm{int}(X_2,X_3,Y_2,Y_3=0)=-\nu _2\left( {X_2^2+Y_2^2\over 2}\right) -\nu _3{X_3^2\over 2}+c_{23}X_2X_3+\ldots \end{aligned}$$

with $\nu _2,\nu _3>0$, $|c_{2,3}|<\min (\nu _2,\nu _3)$.

Owing to their proximity to the manifolds $\mathcal {M}_\textrm{int}(\mathcal {E})$, for energies low enough, the manifolds $\mathcal {M}(\mathcal {E})$ are also ellipsoidal-like (the exact evolution of the form of $\mathcal {M}(\mathcal {E})$ as the energy increases will be discussed in the next section). As a consequence, there is a negative value of the energy $\mathcal {E}=\mathcal {E}_{\min }$ such that for $\mathcal {E}<\mathcal {E}_{\min }$ the ellipsoidal manifold $\mathcal {M}(\mathcal {E})$ surrounds the sphere $\mathcal {I}_0$. Since, however, from Eq. (29) it follows that the condition $\cos (i_\textrm{mut})\le 1$ leads² to $X_3^2+X_2^2+Y_2^2\le 2 \textrm{AMD}\,$, the condition $\mathcal {E}<\mathcal {E}_{\min }$ leads to the unphysical condition $\cos (i_\textrm{mut})>1\,$. At the energy $\mathcal {E}=\mathcal {E}_{\min }$ the ellipsoidal manifold $\mathcal {M}(\mathcal {E}_{\min })$ has two points of tangency with the sphere $\mathcal {I}_0$ (see Fig. 1). Both tangencies occur at the plane $Y_2=0$, but only one of them satisfies the condition ${\dot{Y}}_3\ge 0$. Then, we set $(X_{2,0},Y_{2,0}=0,X_{3,0}^2=2\textrm{AMD}-X_{2,0}^2\,)$ equal to the coordinates of the corresponding point of tangency, and $\mathcal {E}_{\min }=\mathcal {H}_\textrm{sec}(X_2=X_{2,0},Y_2=0,X_3=X_{3,0},Y_3=0;\textrm{AMD})$.

Fig. 1

Left: tangency between the ellipsoidal surfaces of minimum possible energy $\mathcal {M}(\mathcal {E}_{\min })$, with $\mathcal {E}_{\min }=-1.18237\cdot 10^{-4}$ (light gray) and the sphere $\mathcal {I}_0$ (gray). The only permissible initial condition on the surface of section corresponds to the point of tangency (black dot). The green slightly inclined surface represents the condition ${\dot{Y}}_3=0$. Right: the tangency between the surfaces $\mathcal {M}(\mathcal {E}_{\min })$ (thick light gray closed curves) and the sphere $\mathcal {I}_0$ (thin gray curve) as seen in the plane $(X_2,X_3)$ for $Y_2=0$. The green slightly inclined curve represents the condition ${\dot{Y}}_3=0$ and separates the plane $(X_2,X_3)$ in an upper domain, where ${\dot{Y}}_3>0$, and a lower domain (${\dot{Y}}_3<0$). The point of tangency (thick dot) marks a fixed point corresponding to a planar periodic orbit of the apsidal corotation type called ‘mode A’ (anti-aligned) in Sect.  3, yielding $(X_2 = X_{2,0}=0.101237$, $X_3 = X_{3,0}=\sqrt{2\textrm{AMD}-X_{2,0}^2}=0.148862$, $Y_2 = Y_{2,0}=0$, $Y_3=0\,)$, that correspond to $e_2=0.337415$, $e_3=0.433811$

In order to numerically specify, now, the limits of the domain $\mathcal {D}(\mathcal {E})$ on the Poincaré section for any value of the energy in the range $\mathcal {E}_{\min }<\mathcal {E}<0$ we work as follows: fixing any value of the angle $w_2=-\omega _2$ in the interval $0\le w_2\le \pi$, the line defined parametrically by the relations:

$$\begin{aligned} \mathcal {L}(w_2):\{X_2=s_2\cos (w_2),~Y_2=s_2\sin (w_2),s_2\in {\mathbb {R}}\} \end{aligned}$$

(37)

defines a plane $\mathcal{P}\mathcal{L}(w_2)=\{(X_2,Y_2)\in \mathcal {L}(w_2),\,X_3\in {\mathbb {R}}\}.$ The plane $\mathcal{P}\mathcal{L}(w_2)$ intersects the ellipsoidal manifold $\mathcal {M}(\mathcal {E})$ at a nearly elliptic closed curve $\mathcal {C}_{PLM}(\mathcal {E},w_2)$, while it intersects the sphere $\mathcal {I}_0$ at the circle $\mathcal {C}_{PLI_0}: s_2^2+X_3^2=2\textrm{AMD}$. The curve $\mathcal {C}_{PLM}(\mathcal {E},w_2)$ has central symmetry, and, owing to the fact that its quadratic-form approximation is negative definite, its overall size decreases as $\mathcal {E}$ increases from the most negative possible value $\mathcal {E}=\mathcal {E}_{\min }$ towards the value $\mathcal {E}=0$, at which $\mathcal {C}_{PLM}(\mathcal {E},w_2)$ reduces to a point at the origin of the plane $\mathcal{P}\mathcal{L}(w_2)$. Finally, the surface ${\dot{Y}}_3=0$ also intersects the plane $\mathcal{P}\mathcal{L}(w_2)$ at a curve $\mathcal {C}_{PL{\dot{Y}}_3=0}(w_2)$. As a result, fixing the value of $w_2$, there are three possibilities as regards the intersections of the curve $\mathcal {C}_{PLM}(\mathcal {E},w_2)$, which varies with the energy, and the curves $\mathcal {C}_{PL{\dot{Y}}_3=0}(w_2)$ and $\mathcal {C}_{PLI_0}$:

Regime 1 the curve $\mathcal {C}_{PLM}(\mathcal {E},w_2)$ intersects the circle $\mathcal {C}_{PLI_0}$ at four points, two of which ($P_1,P_2$) are above the curve $\mathcal {C}_{PL{\dot{Y}}_3=0}(w_2)$, hence corresponding to ${\dot{Y}}_3>0$, while it intersects the curve $\mathcal {C}_{PL{\dot{Y}}_3=0}(w_2)$ itself at two points which are exterior to the circle $\mathcal {C}_{PLI_0}$ (Fig. 2). In this case, the permissible initial conditions in the Poincaré surface of section are given by $X_2=s_2\cos (w_2)$, $Y_2=s_2\sin (w_2)$ with $s_2$ in the interval $s_{P_1}\le s_2\le s_{P_2}$, where $s_{P_1}$, $s_{P_2}$ are the values of $s_2$ to which project the points of intersection $P_1,\,P_2$. At both these points we have ${\dot{Y}}_3>0$.

Fig. 2

Left: intersections between the surfaces $\mathcal {M}(\mathcal {E})$ at the energy $\mathcal {E}=-9.16\cdot 10^{-5}$ (light gray), the sphere $\mathcal {I}_0$ (thin gray), and the surface ${\dot{Y}}_3=0$ (green slightly inclined). The thick black curve projected on the plane $(X_2,Y_2)$ defines the permissible domain of initial conditions (see text). Right: the intersections between the surface $\mathcal {M}(\mathcal {E})$ (thick light gray closed curve), the sphere $\mathcal {I}_0$ (thin gray curve) and the condition ${\dot{Y}}_3=0$ (green slightly inclined curve) as viewed in the plane $(X_2,X_3)$ for $Y_2=0$. The points of intersection mark the limits of possible initial conditions along $X_2$ for $X_3=Y_2=0$

Regime 2 for larger energies, the curve $\mathcal {C}_{PLM}(\mathcal {E},w_2)$ still intersects the circle $\mathcal {C}_{PLI_0}$ at four points, two of which ($P_1,P_2$) are above the curve $\mathcal {C}_{PL{\dot{Y}}_3=0}(w_2)$, while now intersecting the curve $\mathcal {C}_{PL{\dot{Y}}_3=0}(w_2)$ itself at two points $(P_3,P_4)$, which are interior to the circle $\mathcal {C}_{PLI_0}$ (Fig. 3). In this case, the permissible initial conditions in the Poincaré surface of section are given by $X_2=s_2\cos (w_2)$, $Y_2=s_2\sin (w_2)$ with $s_2$ in one of the intervals $s_{P_3}\le s_2\le s_{P_1}$, or $s_{P_2}\le s_2\le s_{P_4}$, where $s_{P_1}$, $s_{P_2}$, $s_{P_3}$, $s_{P_4}$ are the values of $s_2$ to which project the points of intersection $P_1$, $P_2$, $P_3$, $P_4$. In this case we have ${\dot{Y}}_3>0$ at the inner limits $P_1$, $P_2$, while we have ${\dot{Y}}_3=0$ (i.e., the orbit arrives tangently to the surface of section) at the outer limits $P_3$, $P_4$.

Fig. 3

Same as in Fig. 2 but for the surface $\mathcal {M}(\mathcal {E})$ calculated at $\mathcal {E}=-7.09\cdot 10^{-5}$. The limits of possible initial conditions along $X_2$ for $X_3=Y_2=0$ are marked by the intersections $\mathcal {M}(\mathcal {E})$ both with $\mathcal {I}_0$ and the curve $\mathcal {C}_{PL{\dot{Y}}_3=0}(w_2)$, for $w_2=0$ (see text)

Regime 3 for still larger energies, the curve $\mathcal {C}_{PLM}(\mathcal {E},w_2)$ decreases in size in such a way that it no longer intersects the circle $\mathcal {C}_{PLI_0}$. Then the only limits are posed by its intersections with the curve $\mathcal {C}_{PL{\dot{Y}}_3=0}(w_2)$ at the points $(P_3,P_4)$, which remain interior to the circle $\mathcal {C}_{PLI_0}$ (Fig. 4). Then, the permissible initial conditions in the Poincaré surface of section are given by $X_2=s_2\cos (w_2)$, $Y_2=s_2\sin (w_2)$ with $s_2$ in the interval $s_{P_3}\le s_2\le s_{P_4}$, and both limits correspond to ${\dot{Y}}_3=0$, i.e., to orbits tangent to the surface of section.

Fig. 4

Inner tangency between the surface $\mathcal {M}(\mathcal {E}_{2,3})$ at the energy $\mathcal {E}_{2,3}=-6.77\cdot 10^{-5}$ and the sphere $\mathcal {I}_0$. The point of tangency marks a fixed point which corresponds to a planar periodic orbit of the type ‘mode B’ (aligned apsidal corotation)

Having fixed the value of the energy $\mathcal {E}$, and repeating the computation of the above intersection points for various lines $\mathcal {L}(w_2)$ (i.e., various choices of the argument of the perihelion $\omega _2$), we can explicitly compute the limits of the whole domain $\mathcal {D}(\mathcal {E})$ and proceed in the computation of the phase portraits, obtaining several initial conditions within the domain $\mathcal {D}(\mathcal {E})$. In practice, by symmetry, it is sufficient to consider only initial conditions along the line $\mathcal {L}(w_2=0)$, i.e., $X_2=s_2$, $Y_2=0$. Note also that the regimes 1 and 2, and then 2 and 3, are separated at the energies $\mathcal {E}_{1,2}$ and $\mathcal {E}_{2,3}$, respectively. We discuss in the next section the topological differences in the phase portraits between the various regimes. We only note here that the generic regime is regime 3, which emerges beyond the energy $\mathcal {E}_{2,3}$, at which the manifold of constant energy $\mathcal {M}(\mathcal {E}_{2,3})$ has an inner tangency with the sphere $\mathcal {I}_0$. Finally, we note that both critical tangencies occurring at the energies $\mathcal {E}_{\min }$ and $\mathcal {E}_{2,3}$ correspond to basic periodic orbits of the system (the apsidal corotation resonances), while the final limit $\mathcal {E}=0$ corresponds to the Kozai–Lidov fixed point of the system, at which $i_\textrm{mut}=i_{\max }$ and $e_2=e_3=0$.

2.4.2 Poincaré surface of section: precision tests

Figure 5 shows a summary of our basic example of computation of phase portraits, in the form of Poincaré surfaces of section computed as explained in the previous subsection.

Fig. 5

Poincaré surfaces of section in the plane $(e_2 \cos (\omega _2 ), e_2 \sin (\omega _2 ))$ with $L_z$ fixed and different values of energy. The surfaces of section are computed by a numerical integration of trajectories in the Hamiltonian truncated at: Top multipolar degree $N_{P}=5$, order $N_\textrm{bk}=10$ in the eccentricities, and energies (from left to right) $\mathcal {E}= -6.67\cdot 10^{-5},-2.53\cdot 10^{-5},-1.9\cdot 10^{-5},-1.17\cdot 10^{-5},-2.61\cdot 10^{-6}$. (Middle) $N_{P}=6$, $N_\textrm{bk}=10$, and energies $\mathcal {E}= -6.75\cdot 10^{-5}, -2.69\cdot 10^{-5}, -1.9\cdot 10^{-5},-1.16\cdot 10^{-5}, -2.61\cdot 10^{-6}$. (Bottom) $N_{P}=6$, $N_\textrm{bk}=12$, and energies $\mathcal {E}= -6.75\cdot 10^{-5},-2.69\cdot 10^{-5},-1.9\cdot 10^{-5},-1.16\cdot 10^{-5},-2.59\cdot 10^{-6}$

Fig. 6

Poincaré surfaces of section in the plane $(e_2 \cos (\omega _2 ), e_2 \sin (\omega _2 ))$ with $L_z$ fixed and different values of energy. The surfaces of section are computed by a numerical integration of trajectories in the Hamiltonian truncated at: Top multipolar degree $N_{P}=5$, order $N_\textrm{bk}=10$ in the eccentricities, and energies (from left to right) $\mathcal {E}= -6.67\cdot 10^{-5},-2.53\cdot 10^{-5},-2.61\cdot 10^{-6}$. (Bottom) $N_{P}=4$, $N_\textrm{bk}=10$, and energies top $\mathcal {E}= -6.53\cdot 10^{-5}, -2.16\cdot 10^{-5}, -2.62\cdot 10^{-6}$

In Fig. 5 we already observe some of the significant changes in the phase portraits when the energy is varied from a value close to $\mathcal {E}_{\min }\simeq -1.2\cdot 10^{-4}$ (slightly different in each of the models considered in the figure), to another close to $\mathcal {E}_{\max }=0$. In the next section, we make a detailed discussion of the changes observed in the phase portraits as the energy moves from a value, in which the system is closer to the planar regime, to another, in which the system is highly inclined. Here, instead, we only discuss how these figures control the robustness of the phase portraits with respect to the model chosen, which can differ in the maximum multipole degree $N_{P}$ up to which the original Hamiltonian $\mathcal {H}$ is expanded, before the averaging, as well as in the maximum order in book-keeping $N_\textrm{bk}$ of the Hamiltonian $\mathcal {H}_\textrm{sec}$, which is also equal to the maximum order at which the eccentricities appear in $\mathcal {H}_\textrm{sec}$. Checking with increasing values of $N_{P}$ and $N_\textrm{bk}$, we find that the phase portraits stabilize at $N_{P}=5$, $N_\textrm{bk}=10$. Beyond these values, the sequence of bifurcations of new fixed points and the corresponding changes in the topological features of the phase portraits become marginal, with only changes in the second significant figure observed in both the value of the energy $\mathcal {E}$ where a bifurcation occurs, and the position of the corresponding fixed points, separatrices, etc.

Since we are interested only in a qualitative description of the structure and evolution of the phase portraits, for reducing computational time we choose as our basic model the one with $N_{P}=5$, $N_\textrm{bk}=10$.

As shown in Fig. 6, fixing $N_\textrm{bk}=10\,$, the multipolar order of truncation $N_P=4\,$ is not sufficient to reproduce correctly the evolution of the phase portraits. In particular, while the cases $N_P=4$ and $N_P=5$ exhibit a similar sequence of bifurcations of new fixed points, we observe an important difference in the curve $\mathcal {C}\,$, which divides two regions dominated by the continuation of the apsidal corotation orbit. Namely, $\mathcal {C}$ surrounds only one fixed point (called, in the following, A) in the case $N_P=5\,$, while it surrounds also the new fixed points (called $C_1$ and $C_2\,$) when $N_P=4$.

As an independent test, we perform a comparison between the phase portraits obtained with this model and those obtained by a completely independent Laplace–Lagrange expansion of the Hamiltonian $\mathcal {H}$ averaged over the fast angles. In the latter case, the book-keeping process described in Sect. 2.3 must be altered for the decomposition $\mathcal {H}_{\text {sec}}=\mathcal {H}_\textrm{planar}+\mathcal {H}_\textrm{space}$ to naturally emerge while the Jacobi reduction is performed. Namely, we first introduce the book-keeping $e_j\rightarrow \varepsilon e_j$, $j=2,3$ and expand the direct term $-\mathcal {G}m_2m_3/|{\textbf{r}}_2-{\textbf{r}}_3|$ in powers of the orbital eccentricities up to the book-keeping order $N_\textrm{bk}=10$. The so-computed expression has the form:

$$\begin{aligned} \begin{aligned}&-{\mathcal {G}m_2m_3\over |{\textbf{r}}_2-{\textbf{r}}_3|}\\&\quad {=}{\sum _{s=0}^{N_\textrm{bk}} } \sum _{{\varvec{\alpha }},{\varvec{\beta }},{\varvec{\gamma }}} \!\!\frac{D_{{\varvec{\alpha }},{\varvec{\beta }},{\varvec{\gamma }}}(a_j,e_j,i_j;\varepsilon )}{(\Delta _{2,3})^{\frac{2\,s{+}1}{2}}} \cos (\alpha _2\lambda _2 {+} \alpha _3\lambda _3 {+} \beta _2\Omega _2 {+} \beta _3\Omega _3 {+} \gamma _2\varpi _2 {+} \gamma _3\varpi _3), \end{aligned} \end{aligned}$$

(38)

$j=2,3\,$, where the denominator $\Delta _{2,3}$, after the substitution $\Omega _3=\Omega _2+\pi \,$, takes the form:

$$\begin{aligned} \begin{aligned} \Delta _{2,3}&=a_2^2 + a_3^2 - a_2 a_3 \cos (\lambda _2 - \lambda _3) - a_2 a_3 \cos (\lambda _2 - \lambda _3)\cos (i_2+i_3)\\&\quad {+} a_2 a_3 \cos (\lambda _2 {+} \lambda _3 {-} 2 \Omega _2)\cos (i_2{+}i_3) {-} a_2 a_3 \cos (\lambda _2 {+} \lambda _3 {-} 2 \Omega _2), \end{aligned}\, \end{aligned}$$

(39)

with $\varpi _j=\omega _j\,+\,\Omega _j$ and $\lambda _j=M_j\,+\,\varpi _j$ denoting the longitudes of the pericenter and the mean longitudes of the bodies $j=2,3$, respectively. The integer vectors ${\varvec{\alpha }}\equiv (\alpha _2,\alpha _3)$, ${\varvec{\beta }}\equiv (\beta _2,\beta _3)$, ${\varvec{\gamma }}\equiv (\gamma _2,\gamma _3)$ satisfy the D’Alembert rule $\alpha _{2}+\alpha _{3}+\beta _2+\beta _3+\gamma _2+\gamma _3=0\,$. The coefficients $D_{{\varvec{\alpha }},{\varvec{\beta }},{\varvec{\gamma }}}$ are polynomial in the book-keeping parameter $\varepsilon$. At this point, we introduce the book-keeping identities:

$$\begin{aligned} \cos (i_2+i_3)=\eta \varepsilon ^2(\cos (i_2+i_3)-1)+1, \end{aligned}$$

i.e., (see Eq (18))

$$\begin{aligned}&\cos (i_2)\cos (i_3)\!=\!\varepsilon ^2\eta \left( \cos (i_2)\cos (i_3)-1\right) +1\, , \!\!\!{} & {} \sin (i_2)\sin (i_3)\!=\!\varepsilon ^2\eta \sin (i_2)\sin (i_3)\, , \end{aligned}$$

and expand again the expression (38) with respect to the book-keeping parameter $\varepsilon$ up to the truncation order $N_\textrm{bk}=10$. This leads to an expression of the form

$$\begin{aligned} \begin{aligned} -{\mathcal {G}m_2m_3\over |{\textbf{r}}_2-{\textbf{r}}_3|}&= {\sum _{s=0}^{N_\textrm{bk}} \sum _{{\varvec{\alpha }},{\varvec{\beta }},{\varvec{\gamma }}}} \frac{C_{{{\varvec{\alpha }},{\varvec{\beta }},{\varvec{\gamma }}}}(a_2,a_3,e_2,e_3,i_2,i_3;\varepsilon ,\eta )}{\left( a_2^2 + a_3^2 - 2 \,a_2\,a_3\, \cos (\lambda _2 - \lambda _3)\right) ^{\frac{2\,s+1}{2}}}\\&\quad \cos (\alpha _2\lambda _2 + \alpha _3\lambda _3 + (\beta _2+\beta _3)\Omega _2 + \gamma _2\varpi _2 + \gamma _3\varpi _3), \end{aligned} \end{aligned}$$

(40)

where the coefficients $C(a_2,a_3,e_2,e_3,i_2,i_3;\varepsilon ,\eta )$ are polynomial in the book-keeping parameters $\varepsilon ,\eta$. Finally, we perform the classical Laplace–Lagrange averaging ‘by scissors’: the denominator of (40) is Fourier-expanded

$$\begin{aligned} \frac{1}{\left( a_2^2 + a_3^2 - 2 \,a_2\,a_3\, \cos (\lambda _2 - \lambda _3)\right) ^{\frac{2\,s+1}{2}}}\!=\!a_3^{-(2\,s+1)}\!\sum _{j\ge 0} b_{s+\frac{1}{2}}^{(j)}\!\left( \frac{a_2}{a_3}\right) \cos (j(\lambda _2-\lambda _3))~, \nonumber \\ \end{aligned}$$

(41)

where $b_{s+\frac{1}{2}}^{(j)}$ are Laplace coefficients, which can be computed numerically, as

$$\begin{aligned} \begin{aligned}&b_{s+\frac{1}{2}}^{(0)}\left( \alpha \right) =\frac{1}{2\pi }\int _{0}^{2\pi } \left( 1+\alpha ^2 - 2 \,\alpha \, \cos (\vartheta )\right) ^{-(s+\frac{1}{2})}\,d\vartheta ,\\&b_{s+\frac{1}{2}}^{(j)}\left( \alpha \right) =\frac{1}{\pi }\int _{0}^{2\pi } \left( 1+\alpha ^2 - 2 \,\alpha \, \cos (\vartheta )\right) ^{-(s+\frac{1}{2})}\cos (j\vartheta )\,d\vartheta \qquad j\ge 1, \end{aligned} \end{aligned}$$

(42)

where $\alpha =a_2/a_3\,$, or via a multipolar expansion (see the Appendix A). After performing the above expansions, the secular Hamiltonian is obtained by dropping from the original Hamiltonian all terms depending on the ‘fast’ angles $\lambda _i$. This leads to a model in which the inclinations appear only through the combination $\cos (i_2+i_3)$. Switching, now, back to the angles $\omega _j,\Omega _j$, performing the Jacobi reduction as in Sect. 2.3 (i.e., using Eq. (19), collecting together all terms independent or depending on $\eta$, and restoring the numerical values of the book-keeping coefficients $\varepsilon =\eta =1\,$), we arrive at a ‘Laplace–Lagrange’ model $\mathcal {H}_{LL}=\mathcal {H}_{LL,\textrm{planar}}+\mathcal {H}_{LL,\textrm{space}}$ which has the same form as the model of Eq. (24). This is further processed with the introduction of Poincaré variables as in step 4 of Sect. 2.3.

Fig. 7

Comparison of the Poincaré surfaces of section in the plane $(e_2 \cos (\omega _2 ), e_2 \sin (\omega _2 ))$ with $L_z$ fixed and different values of the energy, in two models: Top: our basic model $N_{P}=5$, $N_\textrm{bk}=10$ (same as in the top row of Fig. 5). Bottom: the Laplace–Lagrange secular Hamiltonian model $\mathcal {H}_{LL}$ (see text) truncated at order 10 in the eccentricities, with energies (from left to right) $\mathcal {E}= -6.62\cdot 10^{-5}, -2.94\cdot 10^{-5}, -1.92\cdot 10^{-5},-1.18\cdot 10^{-5}, -2.73\cdot 10^{-6}$

From Fig. 7 we conclude that similar remarks as those of Fig. 5 can be made as regards the comparison of the secular model $\mathcal {H}_\textrm{sec}$ adopted in the present work, based on a multipolar expansion, and the model $\mathcal {H}_{LL}$ obtained by the classical Laplace–Lagrange expansion in the eccentricities. We note, however, that the latter is much harder to compute, while it is obtained by using a series reversion of Kepler’s equation which has a limited convergence. On the other hand, the model based on closed-form averaging of the multipole Hamiltonian expansion requires a special treatment as regards the computation of the underlying canonical transformation which eliminates (instead of ‘scissor cutting’) the fast angles (see Cavallari and Efthymiopoulos 2022). Since in the present work we do not consider this transformation, we will hereafter deal with results derived only by use of the simple multipole and closed-form averaged model $\mathcal {H}_\textrm{sec}$, with the truncation orders $N_{P}=5$, $N_\textrm{bk}=10$.

3 Dynamics

3.1 General

In this section we are interested in analyzing the most important phenomena observed in the phase portraits of the Hamiltonian $\mathcal {H}_\textrm{sec}(X_2, X_3, Y_2, Y_3; \textrm{AMD})$ computed with the basic reference model corresponding to the truncation orders $N_{P}=5$, $N_\textrm{bk}=10$. In the following subsections we will first present the general picture of the transitions taking place in the structure of the phase space as the energy increases in the range $\mathcal {E}_{\min }\le \mathcal {E}\le 0$. Such transitions are caused, for example, by changes in the nature (e.g., stability) of the main equilibria of the system, giving birth to new families of periodic orbits which connect the families dominant in the planar-like regime (high eccentricities, low mutual inclination) with those of the highly inclined regime (low eccentricities, high mutual inclination).

In the analysis of phase portraits as in the sequel, we parametrize all transitions due to bifurcations of new periodic orbits using the fixed (in the Poincaré section) energy $\mathcal {E}$ as the parameter. All energies referred to below are given in units of $M_\odot AU^2/yr^2$. It is easy to see that a certain value of the energy $\mathcal {E}$ establishes a range of allowed mutual inclinations

$$\begin{aligned} i_\textrm{mut}^{\min }(\mathcal {E})\le i_\textrm{mut}\le i_\textrm{mut}^{\max }(\mathcal {E}) \end{aligned}$$

(43)

where both $i_\textrm{mut}^{\min }(\mathcal {E})$ and $i_\textrm{mut}^{\max }(\mathcal {E})$ are increasing functions of $\mathcal {E}$, as shown in Fig. 8.

Fig. 8

The minimum and maximum possible values of the mutual inclination $i_\textrm{mut}^{\min }$, $i_\textrm{mut}^{\max }$, as a function of the energy $\mathcal {E}$. The vertical lines correspond to the energies $\mathcal {E}_{\min }=-1.18237\cdot 10^{-4}$ (birth of the mode A - anti-aligned apsidal corotation), $\mathcal {E}_{1,2}=-8\cdot 10^{-5}$ (completion of the libration domain around the mode A), $\mathcal {E}_{2,3}=-6.77\cdot 10^{-5}$ (birth of the mode B - aligned apsidal corotation), $\mathcal {E}_C=-2.7\cdot 10^{-5}$ (bifurcation of the inclined Kozai–Lidov periodic orbits $C_1,C_2\,$), and $\mathcal {E}_{C,2}=-5.0\cdot 10^{-6}$ (inclined orbit $C_2$ becomes unstable, see text)

The minimum and maximum possible values of the mutual inclination $i_\textrm{mut}^{\min }$, $i_\textrm{mut}^{\max }$, as a function of the energy $\mathcal {E}$ are computed as follows: consider the ellipsoidal surface $\mathcal {I}_{i_\textrm{mut}}$ defined by setting $Y_3=0$ in Eq. (29) for a certain value of $i_\textrm{mut}$. Consider the critical energy $\mathcal {E}=\mathcal {E}_{2,3}$, equal to $\mathcal {E}_{2,3}=-6.77\cdot 10^{-5}$ in our example. For energies $\mathcal {E}\ge \mathcal {E}_{2,3}$, we compute the values $i_\textrm{mut}=i_\textrm{mut}^{\min }(\mathcal {E})$ and $i_\textrm{mut}=i_\textrm{mut}^{\max }(\mathcal {E})$ for which the ellipsoid $\mathcal {I}_{i_\textrm{mut}}$ comes tangent to the ellipsoidal manifold of constant energy $\mathcal {M}(\mathcal {E})$, with $\mathcal {I}_{i_\textrm{mut}}$ being at the interior of $\mathcal {M}(\mathcal {E})$ (for $i_\textrm{mut}=i_\textrm{mut}^{\max }$) or at the exterior of $\mathcal {M}(\mathcal {E})$ (for $i_\textrm{mut}=i_\textrm{mut}^{\min }$). For energies $\mathcal {E}_{\min }\le \mathcal {E}<\mathcal {E}_{2,3}$ (with $\mathcal {E}_{\min }=-1.182\cdot 10^{-4}$ in our example), the condition of tangency of $\mathcal {I}_{i_\textrm{mut}}$ at the exterior of $\mathcal {M}(\mathcal {E})$ leads to unphysical values $\cos (i_\textrm{mut})>1$. Thus, in this interval of energies we have $i_\textrm{mut}^{\min }=0$, while $i_\textrm{mut}^{\max }$ keeps being represented by an increasing function of $\mathcal {E}$ (Fig. 8).

The vertical lines in Fig. 8 indicate values of the energy where important changes take place in the structure of the phase portraits due to the birth, or change of stability character, of some of the most important families of periodic orbits of the system. The most important transitions taking place in the structure of the phase portraits are shown in Fig. 9, whose details will be presented in subsequent subsections. Increasing the energy $\mathcal {E}$, these transitions appear in summary by the following sequence:

Fig. 9

Phase portraits (Poincaré surfaces of section in the plane $(e_2 \cos (\omega _2 ), e_2 \sin (\omega _2 ))$) in the basic model $N_{P}=5$, $N_\textrm{bk}=10$, and for the energies (from top left to bottom right) $\mathcal {E}= -1.145\cdot 10^{-4}$, $-9.16\cdot 10^{-5}$, $-8.23\cdot 10^{-5}$, $-7.09\cdot 10^{-5}$, $-6.77\cdot 10^{-5}, -5.72\cdot 10^{-5}, -4.93\cdot 10^{-5},-3.81\cdot 10^{-5},-3.19\cdot 10^{-5},-2.76\cdot 10^{-5},-2.53\cdot 10^{-5},-2.16\cdot 10^{-5}, -2.08\cdot 10^{-5}, -1.9\cdot 10^{-5}, -1.58\cdot 10^{-5}, -1.53\cdot 10^{-5}, -1.17\cdot 10^{-5}, -7.69\cdot 10^{-6}, -2.61\cdot 10^{-6}, -7.39\cdot 10^{-7}$. The positions of the fixed points corresponding to the periodic orbits of the modes A, B, $C_1$, $C_2$, $D_1$, $D_2$ (see text) are marked by arrows

• (i) At the energy $\mathcal {E}=\mathcal {E}_{\min }$, the available domain $\mathcal {D}(\mathcal {E})$ reduces to a point, corresponding to the point of tangency of Fig. 1. This is a fixed point of the Poincaré map, whose associated orbit yields two coplanar ellipses with anti-aligned pericenters precessing by the same frequency, known as the apsidal corotation orbit (see Laughlin et al. 2002; Lee and Peale 2003; Beaugé et al. 2003). For energies $\mathcal {E}>\mathcal {E}_{\min }$, the above fixed point is continued by a family of periodic orbits, called below the mode A. These correspond physically to inclined planetary orbits whose eccentricities undergo small periodic oscillations around some non-zero constant values $e_{2,A}$, $e_{3,A}$ (functions of the energy), while the arguments of perihelia undergo small periodic oscillations around the fixed relation $\omega _2-\omega _3=0$ (see Sects. 3.2.3 and 3.3). For energies $\mathcal {E}_{\min }<\mathcal {E}<\mathcal {E}_{2,3}$ (equal to $-6.77\cdot 10^{-5}$ in our numerical example), the mode A, which generalizes the anti-aligned apsidal corotation family to the non-planar case, is the unique important stable family in the surface of section (see first three panels of top row of Fig. 9). The corresponding fixed point is surrounded by closed invariant curves, which represent orbits performing quasi-periodic oscillations around the configuration of anti-aligned perihelia. Up to the energy $\mathcal {E}_{1,2}$ (equal to about $-8\cdot 10^{-5}$ in our example), which marks the transition from two to four limits of permissible motion as in Figs. 2 and 3, only quasi-periodic orbits around the A mode exist. On the other hand for energies $\mathcal {E}_{1,2}<\mathcal {E}<\mathcal {E}_{2,3}$, we can also have trajectories with argument $\omega _3-\omega _2$ either circulating or librating around the value $\omega _3-\omega _2=\pi$ (alignment). As explained in detail in Sect. 3.2, the separation between the various librating or circulating regimes is not due to the presence of a dynamical separatrix, but can be explained by an integrable Hamiltonian model approximating $\mathcal {H}_\textrm{sec}$ in the corresponding energy regime, whose phase space has the topology of a 3-sphere rather than the plane ${\mathbb {R}}^2$.

• (ii) For energies $\mathcal {E}_{1,2}<\mathcal {E}<\mathcal {E}_{2,3}$ there is a prohibited domain surrounding the center of the librating motions around $\omega _3-\omega _2=\pi$ (see fourth panel, top row of Fig. 9). At the energy $\mathcal {E}=\mathcal {E}_{2,3}$ this domain shrinks to zero, and at the center of the librations appears a second fixed point of the Poincaré map, which is a periodic orbit physically corresponding to the planar aligned apsidal corotation orbit (see the first panel in the second row of Fig. 9). This also marks the inner point of tangency of the sphere $\mathcal {I}_0$ with the energy manifold $\mathcal {M}(\mathcal {E}_{2,3})$ (Fig. 4). As indicated in Fig. 8, for energies $\mathcal {E}>\mathcal {E}_{2,3}$ there can no longer be any planar orbit intersecting the surface of section. However, similarly as for the mode A, the fixed point corresponding to the aligned apsidal corotation is continued as a family of off-plane periodic orbits, hereafter called the mode B. Physically, such orbits undergo small periodic oscillations around some non-zero constant values $e_{2,B}$, $e_{3,B}$ (also being both functions of the energy), while the arguments of perihelia undergo small periodic oscillations around the fixed relation $\omega _2-\omega _3=\pi$ (see also Sects. 3.2.3 and 3.3). The mode B also is surrounded by quasi-periodic orbits with arguments of the perihelia librating around the relation $\omega _3-\omega _2=\pi$.

• (iii) The topology of the phase space induced by the alternation between the domains of libration and circulation around the modes A and B dominates the picture obtained for the phase portraits in a large subinterval within the permissible range of values of the energy (up to about $\mathcal {E}=-3\cdot 10^{-5}$ in our example). As seen in Fig. 8, this covers most cases of highly inclined orbits, with mutual inclinations being in our example as high as $\sim 40^\circ$. However, at a critical energy $\mathcal {E}_{C}$ (equal to about $-2.7\cdot 10^{-5}$ in our example), a saddle-node bifurcation takes place, giving rise to two new fixed points of the Poincaré map, corresponding to periodic orbits hereafter called the Kozai–Lidov orbits $C_1$ and $C_2$. Physically, these are highly inclined orbits with planetary eccentricities $e_2$ and $e_3$ smaller than those of the modes A and B, tending actually to zero as $\mathcal {E}\rightarrow 0$. The detailed sequence of bifurcations related to these orbits is discussed in detail in Sect. 3.4 (see panels 11–20 of Fig. 9). The most important transitions regard the orbit $C_1$, which becomes stable nearly immediately after its birth, while the orbit $C_2$ undergoes the classical Lidov–Kozai transition from stability to instability, accompanied by the appearance of chaotic motions around it. Such phenomena appear in a small range of energies near the limit $\mathcal {E}=0$, where the whole phase space shrinks again to a unique point at the origin of the surface of section, corresponding to two circular orbits having the maximum possible mutual inclination ($i_\textrm{mut}^{\max }(\mathcal {E}=0)\simeq 46^\circ \,$ in our example). In fact, Fig. 8 indicates that in this Lidov–Kozai regime, the range in possible inclinations for all orbits becomes narrow, being limited to values around the critical $\mathcal {E}_{C,2}$ where the periodic orbit $C_2$ undergoes the Lidov–Kozai instability.

In the following subsections we examine the above phenomena in more detail, by the order of their appearance as the value of the energy $\mathcal {E}$ increases.

3.2 Planar-like regime

In the present section we discuss the structure of the phase portraits in the range of energies $\mathcal {E}_{2,3}\le \mathcal {E}<\mathcal {E}_C$, were the dominant periodic orbits are the Modes A and B, which generalize the apsidal corotations (anti-aligned and aligned, respectively) of the planar case. It was already mentioned that the phase portrait in this case contains two domains where the argument $\omega _2-\omega _3$ librates (around the values 0 and $\pi$, respectively), separated by a domain where $\omega _2-\omega _3$ circulates, as in panels 5–10 of Fig. 9. Owing to its similarity with the phase portrait of the planar problem, this will be called the planar-like regime. It was emphasized, however, that the maximum mutual inclination can be quite high in this regime (see Fig. 8), thus the analogy with the planar case stems from the dynamics, and not necessarily from the degree of coplanarity of the orbits in this regime. Excluding the energetically prohibited domains (as specified in Sect. 2.4.1), this can be extended to cover the cases where the librational domains are only partly covered by quasi-periodic orbits, as in panels 1–4 of Fig. 9.

3.2.1 Integrable approximation of the Hamiltonian

The main qualitative features of the planar-like regime, as well as precise computations regarding its periodic and surrounding quasi-periodic orbits, can be obtained in the context of an integrable approximation for the Hamiltonian $\mathcal {H}_\textrm{sec}$, stemming from the splitting $\mathcal {H}_\textrm{sec}=\mathcal {H}_\textrm{planar}+\mathcal {H}_\textrm{space}$ as in Eq. (21). Starting from

$$\begin{aligned} \begin{aligned}&\mathcal {H}_\textrm{sec}( w_2, w_3,{W_2,\,W_3}; L_z)\\&\quad =\mathcal {H}_\textrm{planar}(w_2-w_3,W_2,W_3) +\mathcal {H}_\textrm{space}(w_2,w_3,W_2,W_3; L_z) \end{aligned} \end{aligned}$$

(44)

and splitting $\mathcal {H}_\textrm{space}$ into those terms which depend only on the difference $w_2-w_3=\omega _3-\omega _2$, denoted by $\mathcal {H}_{0,\textrm{space}}$, and those which do not, denoted by $\mathcal {H}_{1,\textrm{space}}$, we arrive at the following decomposition of the Hamiltonian

$$\begin{aligned} \begin{aligned} \mathcal {H}_\textrm{sec}&=\underbrace{\mathcal {H}_\textrm{planar}( w_2-w_3,W_2,W_3) + \mathcal {H}_{0,\textrm{space}}(w_2-w_3,W_2,W_3; L_z)}_{\mathrm{{Integrable\,\, part}}:=\mathcal {H}_\textrm{int}}\\&\quad + \mathcal {H}_{1,\textrm{space}}(w_2,w_3,W_2,W_3; L_z). \end{aligned} \end{aligned}$$

(45)

The first two terms in the above expression give rise to a 2 degrees of freedom integrable Hamiltonian

$$\begin{aligned} \begin{aligned}&\mathcal {H}_\textrm{int}( w_2-w_3,W_2,W_3;\textrm{AMD}) \\&\quad = \mathcal {H}_\textrm{planar}( w_2-w_3,W_2,W_3)+ \mathcal {H}_{0,\textrm{space}}( w_2-w_3,W_2,W_3;\textrm{AMD}) \end{aligned} \end{aligned}$$

(46)

whose second integral is $W_2+W_3$.

A fact hidden in the process of Jacobi reduction is that a decomposition of the Hamiltonian as in Eq. (45) yields a relative importance of the terms $\mathcal {H}_\textrm{int}$ and $\mathcal {H}_{1,\textrm{space}}$ varying with the energy level $\mathcal {E}$ at which the orbits are computed. This is due to the fact that, after throwing apart constants, all the terms in $\mathcal {H}_\textrm{space}$ stem from substitutions of the inclinations $i_2$ and $i_3$ depending only on the small quantity $1-\cos (i_2+i_3)$ and being of order second or higher in the eccentricities, according to Eq. (18) and (19). Thus, both $\mathcal {H}_{0,\textrm{space}}$ and $\mathcal {H}_{1,\textrm{space}}$ contain terms weighted by factors $(1-\cos (i_2+i_3))^s(e_j^2+\ldots )$ (with $s\ge 1$, $j=2,3$). On the other hand, the terms $\mathcal {H}_\textrm{planar}$ contain no factors $(1-\cos (i_2+i_3))^s$ and are of degree quadratic or higher in the eccentricities. Thus, due to the bound between increasing energy $\mathcal {E}$ and increasing mutual inclination (Fig. 8), the relative importance of the terms $\mathcal {H}_{1,\textrm{space}}$ with respect to the terms $\mathcal {H}_\textrm{int}$ in the Hamiltonian rises as the energy (and hence the level of mutual inclination) increases. This is demonstrated graphically in Fig. 10, which shows a comparison between the shape of the manifolds of constant energy $\mathcal {M}(\mathcal {E})$ and $\mathcal {M}_\textrm{int}(\mathcal {E})$ (Eqs. (35) and (36)) computed at four different energy levels chosen as $\mathcal {E}_{\min }<\mathcal {E}_1<\mathcal {E}_{2,3}$, $\mathcal {E}_{2,3}<\mathcal {E}_2<\mathcal {E}_{C,2}$, $\mathcal {E}_3=\mathcal {E}_{C,2}$, $\mathcal {E}_{C,2}<\mathcal {E}_4<0$. While the manifolds of $\mathcal {H}_\textrm{int}$ remain always ellipsoidal-like, we note the progressive change of the form of the manifolds of constant energy in the complete model from an ellipsoidal to a peanut-shaped form, as the energy increases. This is caused by the growing importance of some terms in $\mathcal {H}_{1,\textrm{space}}$, in particular the terms $\cos (2\omega _2)$ quadratic in $e_2$, which are the same terms causing the transition to the Lidov–Kozai regime (see Sect. 3.4). In fact, near the energy $\mathcal {E}_{C,2}$, where the orbit $C_2$ turns from stable to unstable, the manifold $\mathcal {M}(\mathcal {E})$ under the complete model becomes nearly cylindrical, marking the change of its section with the plane $(X_2,Y_2)$ from elliptic-like to hyperbolic-like, as implied by the Lidov–Kozai mechanism.

Fig. 10

The top row shows the manifolds of constant energy $\mathcal {M}(\mathcal {E})$ compared to the manifolds of constant energy $\mathcal {M}_\textrm{int}(\mathcal {E})$ of the integrable model $\mathcal {H}_\textrm{int}$ (bottom row), for the energies (from left to right) $\mathcal {E}_1=-8\cdot 10^{-5}$ ($\mathcal {E}_1<\mathcal {E}_{2,3}$), $\mathcal {E}_2=-4\cdot 10^{-5}$ ($\mathcal {E}_{2,3}<\mathcal {E}_2<\mathcal {E}_{C,2}$), $\mathcal {E}_3=-5\cdot 10^{-6}$ ($\mathcal {E}_3=\mathcal {E}_{C,2}$), $\mathcal {E}_4=-5\cdot 10^{-7}$ ($\mathcal {E}_4>\mathcal {E}_{C,2}$)

Given the above, we will now focus on a description of the phase portraits in the energy regime (roughly identified as $\mathcal {E}<\mathcal {E}_C$) where the dynamics induced by $\mathcal {H}_\textrm{sec}$ can be well approximated by the dynamics of $\mathcal {H}_\textrm{int}$. In this regime, the following canonical transformation proves useful in semi-analytical (normal form) calculations related to the periodic orbits A and B and their surrounding quasi-periodic orbits:

$$\begin{aligned}&\psi =w_2-w_3\, ,{} & {} \Gamma =\frac{W_2-W_3}{2}\, , \nonumber \\&\varphi =w_2+w_3\, ,{} & {} J=\frac{W_2+W_3}{2}~~. \end{aligned}$$

(47)

The Hamiltonian in the new variables reads (apart from a constant)

$$\begin{aligned} \mathcal {H}_\textrm{sec}(\psi , \varphi ,\Gamma , J)=\mathcal {H}_\textrm{int}( \psi , \Gamma ; J)+\mathcal {H}_{1,\textrm{space}}( \psi , \varphi , \Gamma , J). \end{aligned}$$

(48)

Fig. 11 shows the phase portrait (in the representative planes $(e_2 \cos (\omega _2), e_2 \sin (\omega _2)$ and $(\omega _2, e_2)\,$) corresponding to the Hamiltonian $\mathcal {H}_\textrm{int}$ at the energy $\mathcal {E}=-6.6\cdot 10^{-5}$. The phase portrait, computed as a Poincaré surface of section $\mathcal {P}(\mathcal {E}=-6.6\cdot 10^{-5}, \, \textrm{AMD}=0.0162044)$ (see Eq. (28)), yields invariant curves equivalent to those obtained by the continuous flow after treating $\mathcal {H}_\textrm{int}$ as a one degree of freedom Hamiltonian in the variables $(\psi ,\Gamma )$, with J serving as parameter.

We observe that with the integrable model $\mathcal {H}_\textrm{int}$ we obtain a phase portrait with features qualitatively very similar to those of the phase portraits in the ‘planar-like’ regime under the complete Hamiltonian (e.g., the panels 5–10 in Fig. 9). In particular, the modes A and B of the integrable model are found as fixed points of $\mathcal {H}_\textrm{int}$, given by the solutions of the equations

$$\begin{aligned} {\left\{ \begin{array}{ll} \displaystyle {{\dot{\psi }}=\frac{\partial \mathcal {H}_\textrm{int}}{\partial \Gamma }=0}\\ \displaystyle {{\dot{\Gamma }}=-\frac{\partial \mathcal {H}_\textrm{int}}{\partial \psi }=0} \end{array}\right. }. \end{aligned}$$

(49)

Since J is an integral of motion of $\mathcal {H}_\textrm{int}$, setting $J=K=$ constant implies that any solution $( \psi _{*},\,\Gamma _{*})$ satisfying (49) is a periodic orbit with period given by $T_{\varphi }=2\pi /\omega _{\varphi }\,$, where $\displaystyle {{\dot{\varphi }}=\omega _{\varphi }=\frac{\partial \mathcal {H}_\textrm{int}}{\partial J}\vert _{(\psi =\psi _{*},\,\Gamma =\Gamma _{*}\,, J=K})}\,$. In particular, the modes A and B are given, respectively, by $(\psi ^{(A)}=0,\,\Gamma ^{(A)},K^{(A)})$ and $( \psi ^{(B)}=\pi ,\,\Gamma ^{(B)},\, K^{(B)})\,$, where $(\psi ^{(A)},\,\Gamma ^{(A)})\,$, $(\psi ^{(B)},\,\Gamma ^{(B)})\,$ are solutions of (49). In fact, since the Hamiltonian $\mathcal {H}_\textrm{int}$ depends only on the harmonics $\cos (k\psi )$, with $k\in {\mathbb {N}}\,$, the partial derivative in the second of Eqs. (49) yields only $\sin (k\psi )\,$ terms, hence, it admits the solution $\psi =\psi ^{(A)}=0$ and $\psi =\psi ^{(B)}=\pi$. Given now that $\Omega _3=\Omega _2+\pi$, the condition $\omega _3=\omega _2\,$, i.e., $\psi =0=\psi ^{(A)}\,$, implies $\varpi _3=\varpi _2+\pi$ (perihelia anti-aligned), while the condition $\omega _3=\omega _2+\pi \,$, i.e., $\psi =\pi =\psi ^{(B)}\,$, implies (modulo $2\pi$) $\varpi _3=\varpi _2$ (perihelia aligned). On the other hand, substituting one of the angles, $\psi ^{(A)}$ or $\psi ^{(B)}$, in the first of Eqs. (49), we obtain an algebraic equation of the form ${\dot{\psi }}(\psi =\psi ^{(A)},\Gamma ,J)=0$ or ${\dot{\psi }}(\psi =\psi ^{(B)},\Gamma ,J)=0$. This, together with the constant energy condition $\mathcal {H}_\textrm{int}(\psi =\psi ^{(A)},\Gamma ,J)=\mathcal {E}$, or $\mathcal {H}_\textrm{int}(\psi =\psi ^{(B)},\Gamma ,J)=\mathcal {E}$, can be solved to yield the pairs of values $\Gamma ^{(A)},J=K^{(A)}$, or $\Gamma ^{(B)},J=K^{(B)}$. Finally, the frequency of the apsidal precession for anyone of the periodic orbits A, B is given by $\nu _{apsidal}=-\omega _\varphi /2=-{\dot{\varphi }}/2=-(1/2)(\partial \mathcal {H}_\textrm{int}(\psi ,J,\Gamma )/\partial J)$, with $(\psi ,J,\Gamma )$ substituted with one of the solutions A or B.

3.2.2 The phase space of $\mathcal {H}_\textrm{int}\,$: Hopf variables

An alternative method to compute the equilibria A and B stems from the use of a particular set of variables, called the Hopf variables (Cushman and Bates 1997), which, besides the computation of the equilibria, provides a global mapping of the phase space of the integrable Hamiltonian $\mathcal {H}_\textrm{int}$ to the 3-sphere, thus allowing for a clear identification of all possible orbital dynamical regimes. We introduce the variables $(\sigma _1,\, \sigma _2,\, \sigma _3\,)$ defined by:

$$\begin{aligned} \sigma _1=X_2 X_3+Y_2 Y_3,\quad \sigma _2=Y_2 X_3-Y_3 X_2,\quad \sigma _3=\frac{1}{2}\left( X_2^2+Y_2^2-X_3^2-Y_3^2\right) ~, \end{aligned}$$

(50)

satisfying the Poisson algebra $\lbrace \sigma _i,\sigma _j \rbrace =-2\,\epsilon _{i j k}\sigma _k\,$, where $\epsilon _{i j k}$ is the Levi-Civita symbol and $i,j,k=1,2,3$. Furthermore, we introduce the variable

$$\begin{aligned} \sigma _0=\frac{1}{2}\left( X_2^2+Y_2^2+X_3^2+Y_3^2\right) \end{aligned}$$

(51)

which is a Casimir invariant of the previous algebra, since all Poisson brackets $\{\sigma _i,\sigma _0\}$, $i=1,2,3\,$, vanish. From the definition (50) it follows that

$$\begin{aligned} \begin{aligned}&\sigma _1=2\sqrt{J+\Gamma }\sqrt{J-\Gamma }\cos (\psi ),\,{} & {} \sigma _2=-2\sqrt{J+\Gamma }\sqrt{J-\Gamma }\sin (\psi ),\,\\&\sigma _3=W_2-W_3=2\Gamma .{} & {} \end{aligned} \end{aligned}$$

(52)

We also have the relation $\sigma _0=W_2+W_3=2J$, as well as

$$\begin{aligned} \sigma _1^2+\sigma _2^2+\sigma _3^2=\sigma _0^2=4J^2. \end{aligned}$$

(53)

Then, given the values of $(\sigma _1,\sigma _2,\sigma _3)$, the values of $\Gamma$, J and $\psi$ can be computed unequivocally using the relations (52) and (53). Furthermore, since $J=\sigma _0/2$, and since the only trigonometric terms in the Hamiltonian $\mathcal {H}_\textrm{int}$ are terms $\cos (k(w_3-w_2))=\cos (k\psi )$, $k={\mathbb {N}}^{*}$, it follows that $\mathcal {H}_\textrm{int}=\mathcal {H}_\textrm{int}(\sigma _1,\sigma _3;\sigma _0)$, i.e., the Hamiltonian $\mathcal {H}_\textrm{int}$ does not depend on $\sigma _2$. This implies that, fixing a value of $\sigma _0$ (i.e., of the integral J), the continuous in time phase flow obtained by solving the equations

$$\begin{aligned} \begin{aligned}&{\dot{\sigma }}_1=\{\sigma _1,\sigma _3\}{\partial \mathcal {H}_\textrm{int}\over \partial \sigma _3},~~~{} & {} {\dot{\sigma }}_2= \{\sigma _2,\sigma _1\}{\partial \mathcal {H}_\textrm{int}\over \partial \sigma _1}+ \{\sigma _2,\sigma _3\}{\partial \mathcal {H}_\textrm{int}\over \partial \sigma _3},~~~ \\&{\dot{\sigma }}_3=-\{\sigma _1,\sigma _3\}{\partial \mathcal {H}_\textrm{int}\over \partial \sigma _1},~~~{} & {} \end{aligned} \end{aligned}$$

(54)

yields a flow equivalent to the one obtained under the Hamiltonian $\mathcal {H}_\textrm{int}(\psi ,\Gamma ;J)$, i.e., treating J as a parameter. Due to the constrain on the sphere (Eq. (53)), the curves of the flow (54) are given by the intersection of the constant energy surface $\mathcal {H}_\textrm{int}(\sigma _1,\sigma _3;\sigma _0)=\mathcal {E}$ with the sphere, i.e., they are closed curves which can be mapped to invariant curves in the plane $(\psi ,\Gamma )$. These are geometrically equivalent to the invariant curves of the Poincaré surface of section of $\mathcal {H}_\textrm{int}$ treated as a 2DOF system, mapping $(X_2,Y_2)$ as $X_2=-\sqrt{2(\Gamma +J)}\cos (\psi -\pi )$, $Y_2=\sqrt{2(\Gamma +J)}\sin (\psi -\pi )$. It should be stressed, however, that the above Hamiltonian reduction only yields a geometric equivalence of the two curves, since in the 1DOF reduced system the flow is continuous, while in the 2DOF full system the curves are traced stroboscopically, and they correspond to the intersection of the 2D invariant tori involving both the angles $(\psi ,\varphi )$ with the selected surface of section.

Fig. 11

Left column: The phase portrait of the integrable Hamiltonian $\mathcal {H}_\textrm{int}$ at the energy $\mathcal {E}=-6.6\cdot 10^{-5}$, projected in the variables $(e_2\cos \omega _2,e_2\sin \omega _2)$ (top), or simply $(\omega _2,e_2)$ (bottom). Right column, top: At the values $\sigma _0=\sigma _0^{(A)}$ and $\sigma _0=\sigma _0^{(B)}$, the corresponding spheres $\mathcal {S}_{\sigma _0^{(A)}}$, $\mathcal {S}_{\sigma _0^{(B)}}$ become tangent to the energy surfaces $\mathcal {C}_{\sigma _0^{(A)},\,\mathcal {E}}$, $\mathcal {C}_{\sigma _0^{(B)},\,\mathcal {E}}$. The points of tangency yield the position of the fixed points A and B in the surface of section (see text). Right column, bottom: the intersection of the spheres $\mathcal {S}_{\sigma _0}$ and of the energy surfaces $\mathcal {C}_{\sigma _0,\,\mathcal {E}}$ with the plane $(\sigma _1,\sigma _3)$ for $\sigma _2=0$, for various values of $\sigma _0$. The intersection of one sphere with one energy surface yields a curve on the sphere which is projected to a curve in the above plane. For a particular value of $\sigma _0=\sigma _0^{(S)}$, the curve (thick purple) passes through the south pole S of the corresponding sphere $\mathcal {S}_{\sigma _0}$. This corresponds to a trajectory forming a closed curve in the Poincaré section, which surrounds mode A and passes through the origin. This curve delimits the domain of orbits whose angle $\psi =\omega _3-\omega _2$ librates around the value $\psi =0$. At a different value of $\sigma _0=\sigma _0^{(N)}$ the curve of constant energy (thick black in the right column, bottom) passes through the corresponding sphere’s north pole N. This corresponds to a curve in the surface of section which surrounds the previous curve as well as the origin. In particular, the whole black curve, except for the point $N\,$, yields the part of the corresponding curve in the Poincaré section (left, top figure) contained in the positive semi-plane, while the point N itself inflates to the part of the corresponding closed curve contained in the negative semi-plane. The domain in the surface of section between the thick purple and the thick black curves corresponds to orbits whose argument $\psi =\omega _3-\omega _2$ circulates. All trajectories beyond the outer delimiting curve exhibit librations of the argument $\omega _3-\omega _2$ around the value $\pi$, characteristic of the B-mode

The above information can now be used to the purpose of analyzing the structure of the phase portraits using the mapping of each invariant curve in the sphere (53) to the corresponding curve in the usual Poincaré surface of section. To this end, let

$$\begin{aligned} \mathcal {S}_{\sigma _0}=\lbrace (\sigma _1, \sigma _2, \sigma _3)\in {\mathbb {R}}^3: \, \sigma _1^2+\sigma _2^2+\sigma _3^2=\sigma _0^2 \rbrace \end{aligned}$$

(55)

denote the sphere corresponding to the value $\sigma _0=2J$ of the integral J, and

$$\begin{aligned} \mathcal {C}_{\sigma _0,\, \mathcal {E}} =\lbrace (\sigma _1, \sigma _2, \sigma _3)\in {\mathbb {R}}^3: \, \mathcal {H}_\textrm{int}(\sigma _0, \sigma _1, \sigma _3)=\mathcal {E}\rbrace , \end{aligned}$$

(56)

denote the energy surface in the space $(\sigma _1,\sigma _2,\sigma _3)\in {\mathbb {R}}^3$ corresponding to a fixed energy value $\mathcal {E}$. We can have a physical trajectory for all values of $\sigma _0$ (i.e., of J) for which the surfaces $\mathcal {S}_{\sigma _0}$ and $\mathcal {C}_{\sigma _0,\, \mathcal {E}}$ intersect, limited by two values of $\sigma _0\,$, that are $\sigma _0^{(A)}$ and $\sigma _0^{(B)}$ (corresponding to $J^{(A)}=\sigma _0^{(A)}/2$ and $J^{(B)}=\sigma _0^{(B)}/2\,$), where the two surfaces become tangent (Fig. 11). By the non-dependence of $\mathcal {H}_\textrm{int}$ on $\sigma _2$, the constant energy surface $\mathcal {C}_{\sigma _0,\, \mathcal {E}}$ is normal to any plane $(\sigma _1,\sigma _3)$ with $\sigma _2=const$. Hence, at a tangency point of $\mathcal {S}_{\sigma _0}$ with $\mathcal {C}_{\sigma _0,\, \mathcal {E}}\,$ we necessarily have that $\sigma _2=0$, as well as the tangency condition

$$\begin{aligned} \textrm{rank}\begin{pmatrix} \displaystyle {2\sigma _1} &{} \displaystyle {2\sigma _2} &{} \displaystyle {2\sigma _3} \\ \displaystyle {\frac{\partial \mathcal {H}_\textrm{int}}{\partial \sigma _1}} &{} \displaystyle {0} &{} \displaystyle {\frac{\partial \mathcal {H}_\textrm{int}}{\partial \sigma _3}} \end{pmatrix}=1~~. \end{aligned}$$

The latter condition implies that

$$\begin{aligned} {\dot{\sigma }}_2=\sigma _3\frac{\partial \mathcal {H}_\textrm{int}}{\partial \sigma _1}-\sigma _1\frac{\partial \mathcal {H}_\textrm{int}}{\partial \sigma _3}=0 \end{aligned}$$

that is, the point of tangency is a fixed point of the flow. Up to terms of second order in the variables $\sigma _i$ (i.e., of fourth order in the eccentricities), apart from constant, we find

$$\begin{aligned} \mathcal {H}_\textrm{int}=A\sigma _1^2+B\sigma _3^2+C\sigma _1\sigma _3+D(\sigma _0)\sigma _1+E(\sigma _0)\sigma _3+F(\sigma _0)+\ldots \end{aligned}$$

where A, B, C are constants, while the functions $D(\sigma _0)$ and $E(\sigma _0)$ are linear in $\sigma _0$ and $F(\sigma _0)$ contains terms linear and quadratic in $\sigma _0$. In our numerical example (i.e., semi-major axes, masses and $\textrm{AMD}$ parameter as at the beginning of Sect. 2.4) the coefficients are given by:

$$\begin{aligned}&A\!=0.00249352,&\!\!\!\!&B\!= 0.00144184, \\&C\!=-0.00317818,&\!\!\!\!&D(\sigma _0)\!=0.000151982 - 0.00327448\, \sigma _0, \\&E(\sigma _0)\!=0.0000208226 - 0.00552308\, \sigma _0,&\!\!\!\!&F(\sigma _0)\!=-0.00214621 \,\sigma _0 - 0.104387\, \sigma _0^2. \end{aligned}$$

The quadratic form $A\sigma _1^2+B\sigma _3^2+C\sigma _1\sigma _3$ yields hyperbolas, being A, B, C such that $C^2>AB\,$. Thus, for any permissible value $\sigma _0$ (or, equivalently, of the integral $J\,$), the surface $\mathcal {C}_{\sigma _0,\, \mathcal {E}}$ intersects the plane $\sigma _2=0$ along hyperbola-like curves (Fig. 11). The two points of tangency occur at the values $\sigma _0^{(A)}=2J^{(A)}$ and $\sigma _0^{(B)}=2J^{(B)}$. We find that $\sigma _0^{(A)}<\sigma _0^{(B)}$, while, checking the sign of $\cos (w_3-w_2)$ for the corresponding fixed points, we identify the left tangency (see Fig. 11) as the B-mode and the right tangency as the A-mode at the given level of energy.

With the help of the bottom-right panel of Fig. 11 it is possible, now, to interpret the form of the phase portraits as in the left column of the same figure. To this end, we specify the correspondence between the various curves of the phase flow on the sphere, obtained by the intersections between the surfaces $\mathcal {S}_{\sigma _0}$ and $\mathcal {C}_{\sigma _0,\mathcal {E}}$ as $\sigma _0$ is altered in the interval $\sigma _0^{(A)}\le \sigma _0\le \sigma _0^{(B)}$, and the mapping of these curves to the surface of section $(e_2\cos \omega _2,e_2\sin \omega _2)$. Since $\mathcal {H}_\textrm{int}$ does not depend on $\sigma _2$, all the curves produced by intersections of the surfaces $\mathcal {S}_{\sigma _0}$ and $\mathcal {C}_{\sigma _0,\,\mathcal {E}}$ contain points which lie in the meridian circle produced by the intersection of $\mathcal {S}_{\sigma _0}$ with the plane $\sigma _2=0$. In particular, the points of tangency A and B belong to this meridian. Besides these points, there are two critical curves which separate domains of libration of the angle $\omega _2-\omega _3$ around the value 0 (mode A), or $\pi$ (mode B), from domains where the angle $\omega _2-\omega _3$ circulates.

By varying the value of $\sigma _0$ in the interval $\sigma _0^{(A)}\le \sigma _0\le \sigma _0^{(B)}$ we progressively obtain curves on the sphere which pass from a librating domain around the fixed point A to a circulating domain, and then to a librating domain around the fixed point B. The first such transition occurs at a value $\sigma _0^{(S)}$ where the curve corresponding to the intersection between $\mathcal {S}_{\sigma _0^{(S)}}$ and $\mathcal {C}_{\sigma _0^{(S)},\,\mathcal {E}}$ passes from the south pole S of the sphere $\mathcal {S}_{\sigma _0^{(S)}}$. The coordinates of the south pole are $\sigma _1^{(S)}=\sigma _2^{(S)}=0$, $\sigma _3^{(S)}=-\sigma _0^{(S)}$, implying $J=-\Gamma$ or $W_2+W_3=-(W_2-W_3)$, hence $W_2=0$. This means a curve in the Poincaré section (thick purple) which crosses the origin $e_2=0$. As shown in Fig. 11, we stress the well-known fact that this curve means no real separatrix in the surface of section, generated by any kind of unstable periodic orbit, but it merely reflects the singularity induced by projecting an (all continuous) transition taking place in the phase space of the integrable Hamiltonian $\mathcal {H}_\textrm{int}$, which is the 3-sphere, to the usual Poincaré section applicable to the full problem, i.e., the plane $(e_2\cos \omega _2,e_2\sin \omega _2)$.

Passing the value $\sigma _0=\sigma _0^{(S)}$, we have curves of the sphere which are projected to invariant curves still surrounding the fixed point A, but for which the argument $\omega _2-\omega _3$ (or $\omega _2-\pi$, in the surface of section) circulates. A second limit of the circulation domain occurs at a value $\sigma _0^{(N)}$ where the curve corresponding to the intersection between $\mathcal {S}_{\sigma _0^{(N)}}$ and $\mathcal {C}_{\sigma _0^{(N)}\!,\,\,\mathcal {E}}$ passes from the north pole N of the sphere $\mathcal {S}_{\sigma _0^{(N)}}$. We readily find that the whole curve in the sphere $\mathcal {S}_{\sigma _0^{(N)}}$, except for the pole N itself, yields the open black curve in the surface of section corresponding to the positive semi-plane $e_2\cos (\omega _2)\ge 0$ (or $-\pi /2\le \omega _2\le \pi /2$), while the north pole itself inflates to the dotted semicircle obtained in the negative semi-plane $e_2\cos (\omega _2)< 0$ (or $\pi /2< \omega _2<3\pi /2$). Here again the singularity is not real but only due to the choice of the variables representing the surface of section (see, e.g., Pauwels 1983; Henrard and Libert 2004; Michtchenko et al. 2006). In fact, N has coordinates $\left( \sigma _0^{(N)},\, \sigma _1^{(N)}=0,\,\sigma _2^{(N)}=0,\,\sigma _3^{(N)}=\sigma _0^{(N)}\right) \,$, implying $W_3=0$, i.e., $e_3=0$. However, the equality $\sigma _3^{(N)}=\sigma _0^{(N)}$ implies also $X_2^2+Y_2^2=2\sigma _0^{(N)}$, i.e., a circle on the section $Y_3=0$. Together with the condition ${\dot{Y}}_3\ge 0$ of the Poincaré section, this implies the semi-circle $X_2\ge 0$, i.e., the dotted part of the black curve in the top-left panel of Fig. 11. It can be shown that the two parts or the curve join each other smoothly at two limiting values $X_2=0,Y_2=\pm Y_{2,\max }=\pm \sqrt{2\sigma _0^{(N)}}$. In fact, the semi-circle $Y_2=\pm \sqrt{2\sigma _0^{(N)}-X_2^2}$, $X_2>0\,$, corresponding to the inflation of the north pole, yields $\lim _{X_2\rightarrow 0^+}(\textrm{d} Y_2/ \textrm{d} X_2)=0$, while the open curve $Y_2=Y_2(X_2)$, corresponding to all other points of the intersection of $\mathcal {S}_{\sigma _0^{(N)}}$ and $\mathcal {C}_{\sigma _0^{(N)}, \, \mathcal {E}}$ except for the pole, yields the limit

$$\begin{aligned} \lim _{X_2\rightarrow 0^-}\frac{\textrm{d}Y_2}{\textrm{d} X_2}= \lim _{X_2\rightarrow 0^-} \left( -\frac{\frac{\partial \mathcal {H}_\textrm{int}}{\partial X_2}}{\frac{\partial \mathcal {H}_\textrm{int}}{\partial Y_2}}\right) ~~. \end{aligned}$$

(57)

From the form of $\mathcal {H}_\textrm{int}\,$, recalling Eq. (27), we readily find $\partial \mathcal {H}_\textrm{int}/\partial X_2=0$, $\partial \mathcal {H}_\textrm{int}/\partial Y_2\ne 0$ at the north pole limit $X_2=X_3=Y_3=0$, $Y_2=\pm Y_{2,\max }$.

Finally, for $\sigma _0$ in the interval $\sigma _0^{(N)}<\sigma _0\le \sigma _0^{(B)}$ we find invariant curves in the sphere $\mathcal {S}_{\sigma _0}$ mapped to closed invariant curves around the tangency corresponding to the B-mode fixed point, which yield also closed curves in the Poincaré section for which the argument $\omega _2-\omega _3$ (or $\omega _2-\pi$ in the section) librates around the value $\pi$, i.e., $\omega _2$ librates around the value $\omega _2=0$ of the B-mode.

3.2.3 Semi-analytical (normal form) determination of the periodic orbits A and B

In the previous subsection we have seen how the existence of the A and B modes, which generalize the apsidal corotation periodic orbits of the model $\mathcal {H}_\textrm{planar}$ to the spatial case, can be established within the framework of the integrable model $\mathcal {H}_\textrm{int}=\mathcal {H}_\textrm{planar}+\mathcal {H}_{0,\textrm{space}}$. In the present subsection we discuss how to recover semi-analytically the periodic orbits A and B under the full Hamiltonian $\mathcal {H}_\textrm{sec}=\mathcal {H}_\textrm{int}+\mathcal {H}_{1,\textrm{space}}$. Besides its relevance in establishing the existence of these orbits in the full model (up to an exponentially small remainder), a computation of the orbits A and B using normal forms allows to obtain a semi-analytical representation of the long term time series of the orbital elements for these planetary trajectories. Since the modes A and B are among the most probable expected end states of the formation process for exoplanetary systems, such a representation can be of use also in the interpretation of the observational data regarding the planetary orbital configurations in such systems.

Let $(\psi _*,\Gamma _*, J_*)$ be a fixed point of the integrable Hamiltonian $\mathcal {H}_\textrm{int}(\psi ,\Gamma ;J)$, corresponding to one of the modes A or B. Introduce the translation

$$\begin{aligned} \psi =\psi _{*}+\delta \psi ,~~\Gamma =\Gamma _{*}+\delta \Gamma ,~~J=J_{*}+\delta J. \end{aligned}$$

(58)

The transformation $(\psi ,\varphi ,\Gamma ,J)\rightarrow (\delta \psi ,\varphi ,\delta \Gamma ,\delta J)$ is canonical. The Hamiltonian $\mathcal {H}_\textrm{sec}$ in the new variables reads:

$$\begin{aligned} \mathcal {H}_\textrm{sec}(\delta \psi ,\varphi ,\delta \Gamma ,\delta J)=\mathcal {H}_\textrm{int}(\delta \psi ,\delta \Gamma ,\delta J) + \mathcal {H}_{1,\textrm{space}}(\delta \psi ,\varphi ,\delta \Gamma ,\delta J). \end{aligned}$$

(59)

We then have the following

Proposition

There is a near-to-identity canonical transformation

$$\begin{aligned} (\delta \psi ,\varphi ,\delta \Gamma ,\delta J)=\Phi ^{(r)}(\delta \tilde{\psi },\tilde{\varphi },\delta \tilde{\Gamma },\delta \tilde{J})~~, \end{aligned}$$

(60)

real-analytic in a suitably defined domain, and obtained by a composition of r Lie canonical transformations, with r a sufficiently high integer, such that the Hamiltonian in the variables $(\delta \tilde{\psi },\tilde{\varphi },\delta \tilde{\Gamma },\delta \tilde{J})$ takes the form:

$$\begin{aligned} \begin{aligned} \mathcal {H}_\textrm{sec}^{(r)}(\delta \tilde{\psi },\tilde{\varphi },\delta \tilde{\Gamma },\delta \tilde{J})= Z^{(r)}(\delta \tilde{\psi },\delta \tilde{\Gamma },\delta \tilde{J}) +R^{(r)}(\delta \tilde{\psi },\tilde{\varphi },\delta \tilde{\Gamma },\delta \tilde{J})~ \end{aligned} \end{aligned}$$

(61)

with $||R^{(r)}||<<||Z^{(r)}||$ for a suitably defined norm in the domain of the transformation. The superscript (r) in the above expressions means the expression derived after substituting the original variables $(\delta \psi ,\varphi ,\delta \Gamma ,\delta J)$ with the new canonical variables obtained by the transformation inverse to the r-step transformation (62):

$$\begin{aligned} (\delta \tilde{\psi },\tilde{\varphi },\delta \tilde{\Gamma },\delta \tilde{J})= \left( \Phi ^{(r)}\right) ^{-1}(\delta \psi ,\varphi ,\delta \Gamma ,\delta J)~~. \end{aligned}$$

(62)

The functions $Z^{(r)}$ and $R^{(r)}$ are hereafter called the ‘normal form’ and ‘remainder’, respectively.

The procedure by which to obtain the transformation $\Phi ^{(r)}$ as well as its inverse is summarized in Appendix B, while a detailed demonstration of the above proposition is given elsewhere (Mastroianni and Efthymiopoulos, in preparation). In summary, this is a classical Birkhoff-like normalization procedure by which the angle $\varphi$ is eliminated from the Hamiltonian via a sequence of Lie canonical transformations (see Efthymiopoulos 2012).

The physical meaning of the above normalization procedure is the following: in the planar-like regime, we have $||\mathcal {H}_{1,\textrm{space}}||<<\mathcal {H}_\textrm{int}$. Then, through the normalization we find a new set of variables in which, except for a very small remainder, the dynamics locally (around the equilibrium $(\psi _*,\Gamma _*,J_*)$) is given by the normal form $Z^{(r)}(\tilde{\psi },\tilde{\Gamma },\tilde{J})$, with

$$\begin{aligned} \tilde{\psi }=\psi _*+\delta \tilde{\psi },~~\tilde{\Gamma }=\Gamma _*+\delta \tilde{\Gamma },~~\tilde{J}=J_*+\delta \tilde{J}~~. \end{aligned}$$

(63)

The phase flow induced by the integrable Hamiltonian $Z^{(r)}$ has the same structure as the one of the Hamiltonian $\mathcal {H}_\textrm{int}$ analyzed in the previous subsection, differing by it just in the fact that $Z^{(r)}$ contains terms arising from the normalization of $\mathcal {H}_{1,\textrm{space}}$ up to a very small remainder. In particular, Hamilton’s flow under the normal form in the transformed variables admits a periodic orbit given by

$$\begin{aligned} \tilde{\psi }_*= & {} \psi _*+\delta \tilde{\psi }_*~~, \nonumber \\ \tilde{\Gamma }_*= & {} \Gamma _*+\delta \tilde{\Gamma }_*~~, \nonumber \\ \tilde{J}_*= & {} J_*+\delta \tilde{J}_*~~, \nonumber \\ \tilde{\varphi }(t)= & {} \tilde{\varphi }(0)+\nu _*t~~, \end{aligned}$$

(64)

where $(\tilde{\psi }_*$, $\tilde{\Gamma }_*,\tilde{J}_*)$ are computed by the system of algebraic equations

$$\begin{aligned} {\left\{ \begin{array}{ll} \displaystyle {\dot{\tilde{\psi }}=\frac{\partial Z^{(r)}}{\partial \tilde{\Gamma }}=0}\\ \displaystyle {\dot{\tilde{\Gamma }}=-\frac{\partial Z^{(r)}}{\partial \tilde{\psi }}=0}\\ Z^{(r)}(\tilde{\psi },\,\tilde{\Gamma },\,\tilde{J}\,)=\mathcal {E}\end{array}\right. } \end{aligned}$$

(65)

and the frequency $\nu _*$ is given by

$$\begin{aligned} \nu _*=\left( {\partial Z^{(r)} \over \partial \tilde{J}}\right) _{\tilde{\psi }=\tilde{\psi }_*,\tilde{\Gamma }=\tilde{\Gamma }_*,\tilde{J}=\tilde{J}_*}~~. \end{aligned}$$

(66)

Fig. 12

First row: Poincaré surfaces of section for the values of the energy (from left to right), $\mathcal {E}=-5.72\cdot 10^{-5},\, -3.19\cdot 10^{-5},\, -2.53\cdot 10^{-5}\,$. Second and third rows: time series of the evolution of the eccentricities for both planets along the mode A (anti-apsidal, second row) and B (apsidal, third row). The curves in red and in magenta show the time series for the eccentricities $e_2$ and $e_3$, respectively, as computed semi-analytically. The black and blue curves, instead, show the time series $e_2$ and $e_3$ as computed through a numerical evaluation of the periodic orbits. The numbers of performed normalization steps $N_t$ are: (second row, from left to right) $N_t=12,\,10,\,12\,$, (third row) $N_t=12$. They are chosen in the base of optimizing the comparison between the semi-analytical and the numerical solutions

Figure 12 shows an example of the comparison between the semi-analytical computation of the periodic orbits A, B on the basis of the normal form flow of $Z^{(r)}$, and a full numerical computation of the same orbits. The semi-analytical computation of the periodic orbits proceeds by the following steps:

• (i) We use the tangency method of Sect. 3.2.2 to compute the fixed points of the integrable Hamiltonian $\mathcal {H}_\textrm{int}$, first in the Hopf variables and then in the variables $(\psi$, $\Gamma$, $J)\,$, obtaining $\psi _*$, $\Gamma _*$, $J_*$.

• (ii) Using an appropriate expansion of the Hamiltonian, as well as the method of composition of Lie series truncated at order $N_t$ (see Appendix B for details), we then obtain the transformation $\Phi ^{(r)}$ and its inverse $\left( \Phi ^{(r)}\right) ^{-1}$, as well as the normal form $Z^{(r)}$ representation of the full Hamiltonian.

• (iii) Implementing a Newton method, we then compute the root $(\tilde{\psi }_*$, $\tilde{\Gamma }_*,\tilde{J}_*)$ of the system of algebraic equations (65), as well as the frequency (66). This yields the time evolution of all four quantities $(\delta \tilde{\psi }=\tilde{\psi }_*-\psi _*,\delta \tilde{\Gamma }=\tilde{\Gamma }_*-\Gamma _*,\delta \tilde{J}=\tilde{J}_*-J_*)$ (fixed) and $\tilde{\varphi }(t)$ as in Eq. (64).

• (iv) We finally obtain the semi-analytical approximation to the time flow of all four variables $\psi (t)=\psi _*+\delta \psi (t)$, $\varphi (t)$, $\Gamma (t)=\Gamma _*+\delta \Gamma (t)$, $J(t)=J_*+\delta J(t)$, through the normalizing transformation

$$\begin{aligned} (\delta \psi (t),\varphi (t),\delta \Gamma (t),\delta J(t)) =\left( \Phi ^{(r)}(\delta \tilde{\psi },\tilde{\varphi },\delta \tilde{\Gamma },\delta \tilde{J})\right) _{ \delta \tilde{\psi }=\delta \tilde{\psi }_*,\tilde{\varphi }=\tilde{\varphi }(t), \delta \tilde{\Gamma }=\delta \tilde{\Gamma }_*,\delta \tilde{J}=\delta \tilde{J}_*}~~ \end{aligned}$$

(67)

This can be further transformed into a time series of the evolution of the orbital elements along the periodic orbit through the equations

$$\begin{aligned}&\omega _2=\frac{-\psi -\varphi }{2}\, ,{} & {} W_2=J+\Gamma \, , \nonumber \\&\omega _3=\frac{\psi -\varphi }{2}\, ,{} & {} W_3=J-\Gamma \,, \end{aligned}$$

(68)

which allow to recover the evolution of the arguments of perihelia and eccentricities

$$\begin{aligned} e_j=\sqrt{1-(1-W_j/\Lambda _j)^2},~~j=2,3 \end{aligned}$$

as well as (equivalently to Eq. (17))

$$\begin{aligned} \Theta _2&=\frac{(\textrm{AMD}-W_2-W_3)(2 \Lambda _3+W_2-W_3-\textrm{AMD})}{2 (\Lambda _2+\Lambda _3-\textrm{AMD})}\nonumber \\ \Theta _3&=\frac{(\textrm{AMD}-W_2-W_3)(2 \Lambda _2-W_2+W_3-\textrm{AMD})}{2 (\Lambda _2+\Lambda _3-\textrm{AMD})}\, \end{aligned}$$

(69)

which allow to compute the time series for the inclinations

$$\begin{aligned} i_j=\cos ^{-1}\left( 1-\frac{\Theta _j}{\Lambda _j-W_j}\right) ,~~j=2,3 \end{aligned}$$

of the planetary periodic orbits.

As shown in Fig. 12, the semi-analytical representation of the periodic orbits fits with precision the numerical ones for both the modes A and B. As regards the physical interpretation, we note that the main effect of the perturbation $\mathcal {H}_{1,\textrm{space}}$ is to induce a periodic oscillation in the eccentricities (and, hence, also the inclinations) of both planets, which are no longer constant, contrary to what holds for the classical apsidal corotation orbits in the planar case, or for the orbits of the integrable approximation $\mathcal {H}_\textrm{int}$. From Fig. 12 it is evident that the amplitude of the oscillation of the eccentricities of both planets increases with the energy $\mathcal {E}$, and, hence, with the mutual inclination. Most, notably, however, we point out the ability of the semi-analytical theory to well represent the orbits of both modes A and B in the regime after the onset of the saddle-node bifurcation giving rise to the periodic orbits $C_1$ and $C_2$, as in the third panel of Fig. 12 (see Sect. 3.1). This implies that, while at the corresponding level of energies the structure of the phase portraits is altered considerably with respect to the one of $\mathcal {H}_\textrm{int}$, the representation of the motions by a local normal form computed in the neighborhood of each of the modes A or B yields the correct picture of the dynamics, both qualitatively and quantitatively.

3.3 Transition regime between planar-like and Lidov–Kozai: sequences of bifurcations

The planar-like regime discussed in the previous subsection characterizes the structure of the phase portraits of the full system up to the energy $\mathcal {E}_C$, where the orbits $C_1,C_2$ are generated by a saddle node bifurcation in the taking place in the neighborhood of the mode B, as in the transition seen in the third row of Fig. 9. We call transition regime the one holding at energies in the interval $\mathcal {E}_{C}{\le \mathcal {E}}\le \mathcal {E}_{C,2}$.

Fig. 13

Poincaré surfaces of section for some values of the energy illustrating the sequence of bifurcations taking place in the ‘transition regime’ (see text): $\mathcal {E}=-2.53\cdot 10^{-5}$ (top left), $\mathcal {E}=-2.08 \cdot 10^{-5}$ (top right), $\mathcal {E}=-1.9\cdot 10^{-5}$ (bottom left), $\mathcal {E}=-1.58\cdot 10^{-5}$ (bottom right), corresponding, respectively, to panels $11\,$, $13\,$, 14 and 15 of Fig. 9

Fig. 14

Top row: the evolution of the orbital eccentricities $e_2,\,e_3$ for the periodic orbits corresponding to the modes A (top left panel: top curve $e_2$, bottom curve $e_3$), and B (top middle panel: top curve $e_2$, bottom curve $e_3$). The top right panel shows the librations of the argument $\omega _2-\omega _3$ for the B-mode (top curve) or the A-mode (bottom curve). Bottom row: the evolution of the orbital eccentricities $e_2,\,e_3$ for the periodic orbits corresponding to the fixed points $C_1$ (bottom left panel: top curve $e_3$, bottom curve $e_2$), and $C_2$ (bottom middle panel: top curve $e_2$, bottom curve $e_3$).The bottom right panel shows the librations of the argument $\omega _2-\omega _3$ for the orbit $C_2$ (top curve) or $C_1$ (bottom curve). The above time series are computed at the value of the energy $\mathcal {E}=-2.08\cdot 10^{-5}\,$, corresponding to the 13-th panel of Fig. 9

Figure 13 presents the sequence of bifurcations taking place across the transition regime, which eventually lead to turning unstable the periodic orbit of mode B. At the beginning of the transition, for energies slightly larger than $\mathcal {E}_C$, the orbit $C_1$ turns from unstable to stable by a pitchfork bifurcation, which gives rise to an unstable periodic orbit yielding two fixed points, $D_1,D_2$, in the Poincaré surface of section. As the energy increases, these fixed points move initially away from the fixed point of the orbit $C_1$, while later (for still larger energy) they approach the fixed point of the mode B. Finally, at a second critical value of the energy the fixed points $D_1,D_2$ collide with the B-mode. This terminates the D-family of periodic orbits, by an inverse pitchfork bifurcation which renders the B-mode unstable.

Figure 14 illustrates the evolution of the eccentricity vectors for all four periodic orbits A, B, $C_1,\,C_2$. In all four cases the eccentricities of both planets oscillate periodically, while the argument $\omega _2-\omega _3$ undergoes small librations around one of the values 0 or $\pi$.

3.4 Lidov–Kozai regime

We finally discuss the transition observed in the last row of Fig. 9, in which the periodic orbit $C_2$ turns from stable to unstable via the Lidov–Kozai mechanism. This is accompanied by a large volume of trajectories around $C_2$ becoming chaotic. In the case of a test inner particle ($m_2=0$) at circular orbit $e_2=0$, the mutual inclination is a preserved quantity, equal to the inclination of the test particle $i_\textrm{mut}=i_2$ (since $m_2=0$ the Laplace plane coincides with the constant orbital plane of the outer particle). Furthermore, the stability character of the so-called Lidov–Kozai state (Lidov 1962; Kozai 1962; Lidov and Ziglin 1976; Ito and Ohtsuka 2019) depends only on the value of the inclination. In particular, in the quadrupolar approximation, the transition from stability to instability occurs at a critical inclination equal to $\cos ^{-1}\sqrt{\frac{3}{5}}\sim 39^{\circ }.2\,$ (see Naoz 2016 for a review).

Here, instead, we use a criterion analogous to the one of the classical Lidov–Kozai mechanism in order to obtain an estimate of the critical energy $\mathcal {E}_{C,2}$ at which the orbit $C_2$ turns from stable to unstable in the framework of the quadrupolar approximation to the secular Hamiltonian $\mathcal {H}_\textrm{sec}$ in the full three-body problem, i.e., for $m_2\ne 0$. The quadrupolar Hamiltonian, apart from constants, reads

$$\begin{aligned} \mathcal {H}_\textrm{quad}=-\frac{3\,\mathcal {G}\,m_2\,m_3}{8\,a_3}\left( \frac{a_2}{a_3}\right) ^2\frac{1}{\left( 1-e_3^2\right) ^{3/2}}\mathcal {F}_\textrm{quad}, \end{aligned}$$

(70)

where

$$\begin{aligned} \mathcal {F}_\textrm{quad}=-\frac{1}{3}-\frac{e_2^2}{2}+\frac{3}{2}\,e_2^2\,\theta ^2+\theta ^2+\frac{5}{2}\,e_2^2\left( 1-\theta ^2\right) \cos (2\,\omega _2), \end{aligned}$$

(71)

with $\theta =\cos (i_2+i_3)\,$. Following the process of Jacobi reduction, without any book-keeping control, amounts to expanding the cosine of the mutual inclinations as $\cos (i_2+i_3)=\cos (i_2)\cos (i_3)-\sin (i_2)\sin (i_3)\,$, we can use expression (19), that automatically cancel the dependence of the Hamiltonian on the mutual inclination. The Hamiltonian takes now the form (apart from constants):

$$\begin{aligned}{} & {} \mathcal {H}_\textrm{quad}=\nonumber \\{} & {} \quad -\frac{3 \,a_2 \mathcal {G} \left( 3 \,e_2^2+2\right) \,m_3^3}{64 \,a_3^2 \left( 1-\,e_2^2\right) \sqrt{1-\,e_3^2} \,m_2}-\frac{\,a_2^2 \mathcal {G} \left( 3 \,e_2^2+2\right) \,m_2 \,m_3}{32 \,a_3^3 \left( 1-\,e_3^2\right) ^{3/2}}+\frac{3 \,a_2^3 \mathcal {G} \left( 3 \,e_2^4-\,e_2^2-2\right) \,m_2^3}{64 \,a_3^4 \left( 1-\,e_3^2\right) ^{5/2} \,m_3}\nonumber \\{} & {} \quad +L_z^2\left( \frac{3 \,a_2 \left( 3 \,e_2^2+2\right) \,m_3}{32 \,a_3^3 \left( 1-\,e_2^2\right) \left( 1-\,e_3^2\right) ^{3/2} \,m_0 \,m_2}+\frac{3 \,a_2^2 \left( 3 \,e_2^2+2\right) \,m_2}{32 \,a_3^4 \left( 1-\,e_3^2\right) ^{5/2} \,m_0 \,m_3}\right) \nonumber \\{} & {} \quad -\frac{3 \,a_2 \left( 3 \,e_2^2+2\right) \,L_z^4}{64 \,a_3^4 \mathcal {G} \left( 1-\,e_2^2\right) \left( 1-\,e_3^2\right) ^{5/2} \,m_0^2 \,m_2 \,m_3}\nonumber \\{} & {} \quad {+}\cos (2\,\omega _2)\Bigg [\frac{15 \,a_2 \mathcal {G} \,e_2^2 \,m_3^3}{64 \,a_3^2 \left( 1{-}\,e_2^2\right) \sqrt{1{-}\,e_3^2} \,m_2}{-}\frac{15 \,a_2^2 \mathcal {G} \,e_2^2 \,m_2 \,m_3}{32 \,a_3^3 \left( 1{-}\,e_3^2\right) ^{3/2}}{+}\frac{15 \,a_2^3 \mathcal {G} \,e_2^2 \left( 1{-}e_2^2\right) \,m_2^3}{64 \,a_3^4 \left( 1{-}\,e_3^2\right) ^{5/2} \,m_3}\nonumber \\{} & {} \quad +L_z^2\left( -\frac{15 \,a_2 \,e_2^2 \,m_3}{32 \,a_3^3 \left( 1-\,e_2^2\right) \left( 1-\,e_3^2\right) ^{3/2} \,m_0 \,m_2}-\frac{15 \,a_2^2 \,e_2^2 \,m_2}{32 \,a_3^4 \left( 1-\,e_3^2\right) ^{5/2} \,m_0 \,m_3}\right) \nonumber \\{} & {} \quad +\frac{15 \,a_2 \,e_2^2\,L_z^4}{64 \,a_3^4 \mathcal {G} \left( 1-\,e_2^2\right) \left( 1-\,e_3^2\right) ^{5/2} \,m_0^2 \,m_2 \,m_3}\Bigg ]. \end{aligned}$$

(72)

Note that the secular Hamiltonian at the quadrupolar level does not depend on the argument of the pericenter of the outer planet $\omega _3\,$, therefore the system is integrable (a fact known as the “happy coincidence”, see Lidov and Ziglin (1976) or Naoz (2016) for a review). In particular, the non-dependence of the Hamiltonian on $\omega _3$ implies that the eccentricity of the outer planet $e_3$ is a conserved quantity.

Using $e_3$ as a parameter, the Hamiltonian $\mathcal {H}_\textrm{quad}$ can now be regarded as a one degree of freedom system. This can obtain a polynomial form by passing to the Poincaré variables $(X_2, Y_2)$ described by (25). After such a substitution, the quadratic part of the above Hamiltonian is given by

$$\begin{aligned} \mathcal {H}_{2,\textrm{quad}}(X_2, Y_2; G_3, L_z)=\frac{{\mathfrak {a}}(G_3, L_z)}{2}\, Y_2^2-\frac{{\mathfrak {b}}(G_3, L_z)}{2}\, X_2^2, \end{aligned}$$

(73)

where the coefficients ${\mathfrak {b}}$ and ${\mathfrak {a}}$ are functions of $G_3=L_3\sqrt{1-e_3^2}=$const and of $L_z\,$ (see Eq. (11)); in particular

$$\begin{aligned} \begin{aligned}&{\mathfrak {b}}= \frac{3\, \mathcal {G}^2\, L_2^3 \,\left( 3 \,G_3^2 - L_2^2 + L_z^2\right) \,m_0 \,m_3^7}{8 \,G_3^5\, L_3^3 \,m_2^3}\\&{\mathfrak {a}}=-\frac{3\, \mathcal {G}^2\, L_2 \left( 5\, G_3^4 - 4\, G_3^2\, L_2^2 + 3\, L_2^4 - 10 \,G_3^2\, L_z^2 - 8\, L_2^2\, L_z^2 + 5\, L_z^4\right) m_0\, m_3^7}{16\, G_3^5\, L_3^3\, m_2^3}. \end{aligned} \end{aligned}$$

(74)

In order to find the critical value of the energy $\mathcal {E}_{C,2}$, for fixed $L_z$, at which the periodic orbit $e_3=$const, $e_2=0$ becomes unstable, it is sufficient to compute the eigenvalues of the Jacobian matrix of the Hamiltonian vector field:

$$\begin{aligned} M=\begin{pmatrix} \displaystyle {\frac{\partial \dot{X_2}}{\partial X_2}} &{} \displaystyle {\frac{\partial \dot{X_2}}{\partial Y_2}}\\ \displaystyle {\frac{\partial \dot{Y_2}}{\partial X_2}}&{} \displaystyle {\frac{\partial \dot{Y_2}}{\partial Y_2}} \end{pmatrix}= \begin{pmatrix} \displaystyle {\frac{ \partial ^2 \mathcal {H}_\textrm{quad}}{\partial X_2\,\partial Y_2}} &{} \displaystyle {\frac{\partial ^2 \mathcal {H}_\textrm{quad}}{\partial ^2 Y_2}}\\ \displaystyle {-\frac{\partial ^2 \mathcal {H}_\textrm{quad}}{\partial ^2 X_2}}&{} \displaystyle {-\frac{\partial ^2 \mathcal {H}_\textrm{quad}}{\partial Y_2\,\partial X_2}} \end{pmatrix}= \begin{pmatrix} \displaystyle {0} &{} \displaystyle {{\mathfrak {a}}}\\ \displaystyle {{\mathfrak {b}}}&{} \displaystyle {0} \end{pmatrix}. \end{aligned}$$

(75)

The transition occurs at a critical value of $e_3=e_{3,C_2}$ at which the eigenvalues of M pass from imaginary to real. The critical energy is then given by $\mathcal {E}_{C,2}=\mathcal {H}_\textrm{quad}(e_2=0,e_{3,C_2};L_z)$. The critical value $e_{3,C_2}$ can be computed by the following

Proposition 3.1

Consider the secular Hamiltonian developed up to a quadrupolar expansion. Define the quantities

$$\begin{aligned} \begin{aligned}&A=\frac{1}{5}\left( 4L_2^2+5 L_3^2(1-e_3^2)-L_2\sqrt{L_2^2+60 L_3^2(1-e_3^2)}\right) ,\\&B=\frac{1}{5}\left( 4L_2^2+5 L_3^2(1-e_3^2)+L_2\sqrt{L_2^2+60 L_3^2(1-e_3^2)}\right) ,\\&C=L_2^2-3 L_3^2 (1-e_3^2), \end{aligned} \end{aligned}$$

(76)

with $L_2=m_2\sqrt{\mathcal {G} m_0 a_2}>0\,$, $L_3=m_3\sqrt{\mathcal {G} m_0 a_3}>0\,$. Then, the following cases hold:

• Case i) : $\Big (0< L_2<\sqrt{3}L_3\Big )$ $\wedge$ $\bigg (0\le e_3\le \sqrt{1-\frac{L_2^2}{3 L_3^2}}\bigg )\,$, the periodic orbit $C_2$ is Floquet-stable if $\big (0<L_z^2<A\big )\,$ or $\big (L_z^2 > B\big )\,$ and Floquet-unstable if $\big (A<L_z^2<B\big )$.

• Case ii) : $\Big (0< L_2<\sqrt{3}L_3\Big )$ $\wedge$ $\bigg ( \sqrt{1-\frac{L_2^2}{3 L_3^2}}< e_3 < \sqrt{1-\frac{L_2^2}{4 L_3^2}}\bigg )\,$ or $\Big (\sqrt{3} L_3 \le L_2< 2 L_3\Big )$ $\wedge$ $\bigg (0\le e_3 < \sqrt{1-\frac{L_2^2}{4 L_3^2}}\bigg )\,$, the periodic orbit $C_2$ is Floquet-stable if $\big (C<L_z^2<A\big )$ or $\big (L_z^2>B\big )$ and Floquet-unstable if $\big (0<L_z^2<C\big )$ or $\big (A<L_z^2<B\big )$.

• Case iii) : $\Big (0< L_2\le 2 L_3\Big )$ $\wedge$ $\bigg ( e_3 = \sqrt{1-\frac{L_2^2}{4 L_3^2}}\bigg )\,$, the periodic orbit $C_2$ is Floquet-stable if $\big (L_z^2>B\big )$ and Floquet-unstable if $\big (0<L_z^2<B\big )$.

• Case iv) : $\Big (0< L_2\le 2 L_3\Big )$ $\wedge$ $\bigg ( e_3 > \sqrt{1-\frac{L_2^2}{4 L_3^2}}\bigg )\,$ or if $\big (L_2> 2 L_3\big )$, the periodic orbit $C_2$ is Floquet-stable if $\big (A<L_z^2<C\big )$ or $\big (L_z^2>B\big )$ and Floquet-unstable if $\big (0<L_z^2<A\big )$ or $\big (C<L_z^2<B\big )$.

Moreover, having the critical points of $L_z^2\,$, it is easy to compute the critical values for the mutual inclination, being³ (by Eq. (29))

$$\begin{aligned} \begin{aligned}&\cos (i_2+i_3)=\frac{L_z^2-L_2^2-L_3^2+L_2^2\,e_2^2+L_3^2\,e_3^2}{2 L_2 L_3\sqrt{1-e_2^2}\sqrt{1-e_3^2}}, \\&\max i_\textrm{mut}=\arccos \left( \frac{L_z^2-L_2^2-L_3^2}{2 \,L_2\, L_3}\right) . \end{aligned} \end{aligned}$$

(77)

The proof of the previous Proposition is reported in the Appendix C. We readily find that in the limit $m_2\rightarrow 0$, $e_3\rightarrow 0$, the Kozai angles $i=39.2^\circ$ and $i=140.77^\circ$ are recovered (see Naoz 2016). In fact, for $e_3=0\,$, the critical values $A\,$, B and C become

$$\begin{aligned} \begin{aligned}&A=\frac{1}{5}\left( 4L_2^2+5 L_3^2-L_2\sqrt{L_2^2+60 L_3^2}\right) , \\&B=\frac{1}{5}\left( 4L_2^2+5 L_3^2+L_2\sqrt{L_2^2+60 L_3^2}\right) , \\&C=L_2^2-3\,L_3^2. \end{aligned} \end{aligned}$$

(78)

Then, at the limit $m_2\rightarrow 0\,$ we readily obtain that

$$\begin{aligned}&\left( \cos i_\textrm{mut}\right) _{L_z^2=A}=\frac{A-L_2^2-L_3^2}{2 L_2 L_3}=-\frac{\sqrt{L_2^2+60\,L_3^2}+L_2}{10\,L_3}\;\; \xrightarrow {L_2\rightarrow 0}\;\; -\sqrt{\frac{3}{5}}\\&\left( \cos i_\textrm{mut}\right) _{L_z^2=B}=\frac{B-L_2^2-L_3^2}{2 L_2 L_3}= \frac{\sqrt{L_2^2+60\,L_3^2}-L_2}{10\,L_3} \;\; \xrightarrow {L_2\rightarrow 0}\;\; \sqrt{\frac{3}{5}}\, . \end{aligned}$$

Note that, while the above proposition strictly establishes the limit of $\mathcal {E}_{C,2}$ only in the quadrupolar approximation, in practice we find that the estimate is quite precise when the full Hamiltonian with higher order multipoles is considered. For example, applying the above criterion in the numerical example of Fig. 9 yields the estimate $\mathcal {E}_{C,2}\approx -2.2\cdot 10^{-6}$, which is in agreement with the value of the energy where the transition is observed in the numerical phase portraits. On the other hand, since the quadrupolar approximation yields an integrable Hamiltonian, for any value of the energy $\mathcal {E}>\mathcal {E}_{C,2}$ $\mathcal {H}_\textrm{quad}$ yields a figure-8 phase portraits with invariant curves surrounding the ‘frozen’ (stable) orbits on both sides of the unstable orbit ${e}_2=0$. However, this picture changes by adding just the octupolar terms to the model, given by

$$\begin{aligned} \mathcal {H}_\textrm{oct}=\mathcal {H}_\textrm{quad}+\tilde{\mathcal {H}}_\textrm{oct}, \end{aligned}$$

(79)

where

$$\begin{aligned} \tilde{\mathcal {H}}_\textrm{oct}=\frac{75\,\mathcal {G}\,m_2\,m_3}{64\,a_3}\left( \frac{a_2}{a_3}\right) ^3\frac{1}{\left( 1-e_3^2\right) ^{5/2}}\,e_2\,e_3\,\mathcal {F}_\textrm{oct} \end{aligned}$$

(80)

with

$$\begin{aligned} \begin{aligned} \mathcal {F}_\textrm{oct}\,&=\,\frac{1}{40}\left( 3\,e_2^2+4\right) \left( 1+11\theta -5\theta ^2-15\,\theta ^3\right) \cos (\omega _2-\omega _3)\\&\quad +\frac{1}{40}\left( 3\,e_2^2+4\right) \left( 1-11\theta -5\theta ^2+15\,\theta ^3\right) \cos (\omega _2+\omega _3)\\&\quad + \frac{7}{8}\,e_2^2(\theta ^2-1)(1+\theta )\cos (3\omega _2-\omega _3)+\frac{7}{8}\,e_2^2(\theta ^2-1)(1-\theta )\cos (3\omega _2+\omega _3), \end{aligned} \end{aligned}$$

(81)

with $\theta =\cos (i_2+i_3)\,$. Note that the above formula is equivalent, after some algebraic operations, to the formulas given in Migaszewski and Goździewski (2011) and Naoz (2016), see also Lithwick and Naoz (2011), Naoz et al. (2013), Ford et al. (2000). As depicted in Fig. 15, the addition of more harmonics besides $\cos (2\omega _2)$ via the octupole approximation implies the creation of a large domain of resonance overlap, mostly between the islands around the frozen orbits and the island of the $C_1$ orbit, which appears already in the octupole approximation. This implies, in turn, the disappearance of most quasi-periodic trajectories around the frozen orbits, and the appearance, instead, of a large domain of chaotic orbits, in accordance to what is observed in the last two panels of Fig. 9 for the full model.

Fig. 15

Comparison between the phase portraits (Poincaré surfaces of section) obtained with the quadrupolar approximation $\mathcal {H}_\textrm{quad}$ (left column) and the octupolar approximation $\mathcal {H}_\textrm{oct}$ (right column), at the energy levels $\mathcal {E}=-4.9\cdot 10^{-6}$ (top), and $\mathcal {E}=-6.9\cdot 10^{-7}$ (bottom). The quadrupolar approximation yields the phase portrait of an integrable system, which contains a figure-8 separatrix for energies beyond $\mathcal {E}_{C,2}$. The octupolar approximation is necessary to obtain both periodic orbits $C_1$ and $C_2$, as well as the chaotic zone around $C_2$ caused by the overlapping of resonances in the neighborhood of this orbit

We note that the use of the above proposition for the determination of critical values for the transition to instability has to be done in conjunction with the test that $\cos (\max i_\textrm{mut})$ does not produce unphysical values $|\cos (i_\textrm{mut})|>1$. Such an unphysical value can occur in case ii), for the critical value $L_z^2=C\,$ and in case iv), for the critical value $L_z^2=A\,$. For example, in the second case of ii), setting $\sqrt{3}L_3\le L_2< 2\, L_3$ and $e_3=0\,$, we find $C=L_2^2-3\,L_3^2$ which, together with $L_z^2=C\,$, leads to $\cos (\max i_\textrm{mut})=-2\,L_3/L_2<-1$ iff $L_2< 2\,L_3$. Similarly, in the second case of iv), where $L_2>2\,L_3$, setting $e_3=0\,$, by the relation $L_z^2=A=\frac{1}{5}\left( 4L_2^2+5 L_3^2-L_2\sqrt{L_2^2+60 L_3^2}\right) \,$, we obtain $\cos (\max i_\textrm{mut})=-(\sqrt{L_2+60\,L_3^2}+L_2)/(10\,L_3)$, which is physical only iff $L_2\le 2\,L_3\,$.

3.5 Where does the $\upsilon$-Andromedae system lie in phase space?

Fig. 16

Poincaré surface of section for the energy $\mathcal {E}=-2.081\cdot 10^{-5}$. The red curve (1) shows the orbit obtained by adopting our basic assumption as regards the initial conditions of the $\upsilon$-Andromedae system (see Sect. 2.4), i.e., (after reduction to the Laplace plane) $e_2=0.2445$, $e_3=0.316$, $\omega _2=289.049^\circ$, $\omega _3=235.464^\circ$. The black curve (2) shows the trajectory obtained by changing by only about 10% the eccentricity $e_2$ and argument of the periastron $\omega _2$, while maintaining the energy and argument of periastron of the outer planet invariant ($e_2=0.269$, $\omega _2=296.1^\circ$, $\omega _3=235.464^\circ$, $e_3=0.299\,$)

We finally comment on the implications of the above analysis of the phase space structure on the interpretation of the data given by astronomical observations. Figure 16 shows the phase portrait (surface of section) for the value of the energy $\mathcal {E}=-2.081\cdot 10^{-5}$, corresponding to the one found after reducing to the Laplace plane the data for the observed $\upsilon$-Andromedae system (see Sect. 2.4). The invariant curve (1) (marked in red) corresponds to the actual initial conditions of the system. According to the figure, this is a quasi-periodic orbit surrounding the family $C_2$. We note, however, that the orbit is very close to the separatrix between the $C_2$ librational domain and the domain of stability surrounding the fixed point of the mode B orbit. As a consequence, a change of the initial conditions $(e_2,\omega _2)$ by less than 10% results in a trajectory ((2), black curve) which undergoes quasi-periodic oscillations around the (aligned) apsidal corotation orbit of the system. The proximity of the real trajectory to one of the separatrices generated by the families $C_1,\,C_2$ is noteworthy, as it is in contrast with the basic scenario of evolution of planetary orbits, according to which the dissipative phase of planetary migration should lead to an end state close to one of the stable fixed points of the system (see, for example, Beaugé et al. 2012).

4 Parametric study

The purpose of the present section is to provide an overview of how the form of the phase portraits (Poincaré sections) is varied with the energy $\mathcal {E}$, and hence with an increase in the level of mutual inclination, in various types of planetary systems differing from our so far main example as regards the choice of mass ratio $m_2/m_3$ as well as of semi-major axis ratio $a_2/a_3$ between the two planets. A thorough parametric study of the latter question is beyond our present scope. Instead, here we focus on only one central aspect of this study, namely the question of how generic is the description, as in Sect. 3, of the transition from the planar-like to the Lidov–Kozai regime, when either the mass ratio $m_2/m_3$ or the semi-major axis ratio $a_2/a_3$ are altered, including the so-called hierarchical limits, which correspond to mass ratio limits $m_2/m_3\rightarrow 0$ (restricted three-body problem with the inner planet being a test particle), or $m_2/m_3\rightarrow \infty$ (the outer planet is a test particle), and semi-major axis ratio $a_2/a_3<<1$ (outer planet way further from the star than inner planet). To produce a suite of numerical experiments covering most cases of practical interest, we consider in the sequel five different values of the mass ratio $m_2/m_3$ representative of all possible mass hierarchies (or lack thereof) namely $m_2/m_3=1/10,\,1/3,\,1,\,3,\,10$, produced by the corresponding combination of the masses $m_0=1M_\odot \,$ and $1\,M_{J}\,$, $3\,M_{J}$ or $10\,M_{J}$ for $m_2$ and $m_3\,$, while we consider the semimajor axes ratio $a_2/a_3=1/7$ ($a_2=1\,AU$, $a_3=7\,AU$) as a case representing distance hierarchy, or $a_2/a_3=1/3$ ($a_2=1\,AU$, $a_3=3\,AU$) as a case of no distance hierarchy. In each one of the previous cases, a comparison with the analysis of Sect. 3 requires keeping fixed the value of the $\textrm{AMD}$ (or $L_z$) while computing phase portraits with a varying value of the energy $\mathcal {E}$. To find a relevant value for $L_z$ in each experiment, we consider a uniform value of the maximum possible mutual inclination $i_{\max }$ (Eq. (30)) in all the experiments, set as $i_\textrm{mut}=45^\circ$. This typically proves to be slightly above the limit of the Lidov–Kozai instability (which is equal to $39.2^\circ \,$ in the quadrupolar approximation of the restricted three-body problem with $m_2=0$ and turns to be about $\sim 42.5^\circ \,$ in our main numerical example of Sect. 3). By Eq. (30), this implies setting

$$\begin{aligned} L_z=\sqrt{\Lambda _2^2+\Lambda _3^2+\sqrt{2}\Lambda _2\Lambda _3} \end{aligned}$$

with $\Lambda _2,\Lambda _3$ specified by the choice of masses and semi-major axes in each experiment. Finally, in order to get some safety from any accuracy issues, we make computations with a Hamiltonian expanded at slightly larger orders with respect to those of the main numerical example of the previous section, i.e., multipole order 6, and order 12 in the eccentricities. To arrive at the final secular model, all Hamiltonians are processed, and their phase portraits computed, in the same way as in Sect. 2.

Fig. 17

Poincaré surfaces of section in the plane $(e_2 \cos (\omega _2 ), e_2 \sin (\omega _2 ))$ with $L_z$ fixed such that max$\,i_\textrm{mut}$ is $45^\circ \,$ and different values of energy. We consider the models $a_2/a_3=1/3$ and (from top to bottom) $m_2/m_3=1/10,\,1/3,\,1,\,3,\,10\,$. The values of the energy (from top to bottom) are (from left to right), top $\mathcal {E}=-2.71\cdot 10^{-6}, -1.01\cdot 10^{-6}, -9.24\cdot 10^{-7},-5.73\cdot 10^{-7}, -8.23\cdot 10^{-8}$, $\mathcal {E}=-9.03\cdot 10^{-7},-3.37\cdot 10^{-7},-2.87\cdot 10^{-7}, -1.76\cdot 10^{-7}, -2.35\cdot 10^{-8}$, $\mathcal {E}=-2.68\cdot 10^{-7},-1.39\cdot 10^{-7},-1.14\cdot 10^{-7},-6.37\cdot 10^{-8}, -6.98\cdot 10^{-9}$, $\mathcal {E}=-1.12\cdot 10^{-6},-9.26\cdot 10^{-7},-3.37\cdot 10^{-7}, -2.08\cdot 10^{-7},-1.67\cdot 10^{-8}$, bottom $\mathcal {E}= -2.75\cdot 10^{-6}, -1.91\cdot 10^{-6},-4.72\cdot 10^{-7},-3.54\cdot 10^{-7}, -2.91\cdot 10^{-8}\,$. The surfaces of section have been computed by a numerical integration of trajectories in a Hamiltonian averaged in closed form with a multipolar expansion truncated at degree 6 and expanded up to order 12 in the eccentricities

Figure 17 shows the phase portraits obtained for increasing energy $\mathcal {E}$ in the case $a_2/a_3=1/3$, and in various mass hierarchy scenarios, namely, for the values $m_2/m_3$ (from top to bottom row) 1/10, 1/3, 1/1, 3/1, 10/1. The values of the energies whose corresponding phase portraits are shown are selected in each panel of Fig. 17 so as to visualize the most important changes observed in the phase-space structure as the energy increases.

Our main numerical example (Fig. 9) exhibits a similar structural change in the phase portraits, with increasing energy, as in the fully non-hierarchical case $m_2/m_3=1$ illustrated in the third row of Fig. 17. However, in Fig. 17 it is clear that the transition from the planar-like regime, where the modes A and B dominate the phase space, through the saddle-node bifurcation giving rise to the orbits $C_1$ and $C_2$ (transition regime), and, finally, the Lidov–Kozai regime (orbit $C_2$ becomes unstable and surrounded by a chaotic figure-8 separatrix-like layer) is generic in all cases $m_2/m_3\le 1$ (top three rows of Fig. 17) and follows by the same sequence of bifurcations. As the ratio $m_2/m_3$ tends to small values, the main differences observed with respect to the non-hierarchical case are: i) in the planar like regime (i.e., top row, first panel), the domain of quasi-periodic orbits around the aligned apsidal corotation orbit (B-mode) occupies most of the available phase space, and ii) at energies beyond the one of the Lidov–Kozai instability, there is a significant domain of regular orbits obtained around two stable periodic orbits of non-zero eccentricity $e_2$ with fixed points along the axes $\omega _2=\pm \pi /2$ in the surface of section (the so-called frozen orbits). Both properties (i) and (ii) can be interpreted by analogy with the restricted three-body problem $(m_2=0)$, in which, for low inclinations, the only fixed point of the quadratic (Laplace–Lagrange) secular Hamiltonian is associated with an eccentric orbit of eccentricity $e_2$ equal to the forced value induced by the outer perturber and pericenter aligned with the one of the outer perturber (see Ferraz-Mello 2007). On the other hand, in the cases $m_2/m_3>1$ we do not observe at all the transition to the Lidov–Kozai regime. Again, this can be interpreted in analogy with the RTBP, in which the outer Lidov–Kozai instability ($m_3\rightarrow 0$) occurs at an inclination ($\approx 63^\circ$; see von Zeipel 1910) much higher than the maximum mutual inclination $i_{\max }=45^\circ$ considered in our examples. The case of a very high mutual inclination has been considered in the literature (see for example Hansen and Naoz 2020 and references therein), but it appears of rather theoretical interest compared to available observations on the orbital inclinations in exoplanetary systems.

Fig. 18

Poincaré surfaces of section in the plane $(e_2 \cos (\omega _2 ), e_2 \sin (\omega _2 ))$ with $L_z$ fixed such that max$\,i_\textrm{mut}$ is $45^\circ \,$ and different values of energy. We consider the models $a_2/a_3=1/7$ and (from top to bottom) $m_2/m_3=1/10,\,1/3,\,1,\,3,\,10\,$. The values of the energy (from top to bottom) are (from left to right), top $\mathcal {E}=-1.54\cdot 10^{-7},-6.71\cdot 10^{-8},-5.78\cdot 10^{-8},-2.46\cdot 10^{-8}, -6.15\cdot 10^{-9}$, $\mathcal {E}=-4.5\cdot 10^{-8},-2.05\cdot 10^{-8}, -1.7\cdot 10^{-8},-7.16\cdot 10^{-9}, -1.79\cdot 10^{-9}$, $\mathcal {E}=-2.75\cdot 10^{-8},-8.25\cdot 10^{-9},-7.01\cdot 10^{-9},-4.97\cdot 10^{-9}, -5.51\cdot 10^{-10}$, $\mathcal {E}=-7.5\cdot 10^{-8},-5.94\cdot 10^{-8},-3.69\cdot 10^{-8}, -2.5\cdot 10^{-8}, -1.38\cdot 10^{-9}$, bottom $\mathcal {E}=-2.22\cdot 10^{-7},-1.21\cdot 10^{-7},-8.96\cdot 10^{-8},-4.91\cdot 10^{-8}, -2.42\cdot 10^{-9}\,$. The surfaces of section have been computed by a numerical integration of trajectories in a Hamiltonian averaged in closed form with a multipolar expansion truncated at degree 6 and expanded up to order 12 in the eccentricities

Figure 18 shows how the phase portraits evolve with the energy when, in addition to altering the mass ratio $m_2/m_3$, the limit of distance hierarchy is also approached ($a_2/a_3=1/7$). As an overall observation, we note again that the full sequence of bifurcations leading from the planar-like to the Lidov–Kozai regime is realized only in the cases $m_2/m_3\le 1$. However, the main difference, with respect to the case $a_2/a_3=1/3$, is that in the hierarchical in distance case the phase portraits contain many more regular orbits, with a considerably large domain of stability around the frozen orbits surviving the system’s perturbations even at the mass ratio $m_2/m_3=1$ (third row of Fig. 18). This can be interpreted by the fact that the distance hierarchy brings the system closer to the dynamics of the integrable Hamiltonian $\mathcal {H}_\textrm{quad}$ (Eq. (72)), reducing the relative importance of perturbations including and beyond the octupolar one.

5 Concluding remarks

From what was exposed in Sects. 2, 3 and 4 above, we arrive at the following summary of conclusions, serving also as a navigation guide to the main methodological steps proposed in the preceding sections of the paper.

• (i) Sect. 2 exposed the methodology for obtaining a suitable secular Hamiltonian model, as well as our choice of variables and representation for phase portraits, aiming to present in a unified way both the planar-like regime, where apsidal corotation orbits dominate, and the Lidov–Kozai regime, where nearly circular highly inclined orbits dominate. The use of a particular book-keeping method was explained in Sect. 2.3, leading, after Jacobi reduction, to a Hamiltonian decomposition of the form $\mathcal {H}_\textrm{sec}=\mathcal {H}_\textrm{planar}+\mathcal {H}_\textrm{space}$, where the term $\mathcal {H}_\textrm{planar}$ is integrable and the term $\mathcal {H}_\textrm{space}$ depends on the system’s $\textrm{AMD}$ (or, equivalently, the value of the system’s angular momentum $L_z$). We argue on the merits stemming from a choice of simple definition for the Poincaré surface section as in 2.4.1, allowing for a straightforward mapping of phase portraits to the evolution of the corresponding orbits in terms of Keplerian elements. This is accompanied by a detailed analysis of the permissible domains of initial conditions on the surface of section as the energy $\mathcal {E}$ is altered. It is demonstrated that these limits are defined by the intersections between any two of three important manifolds, namely the manifold of constant energy $\mathcal {M}(\mathcal {E})$, the manifold of zero mutual inclination ${{\mathcal {I}}}_{0}$, and the condition of tangency of an orbit with the surface of section (${\dot{Y}}_3=0$ in the Poincaré variables (25)). In particular, we explain how the tangencies between $\mathcal {M}(\mathcal {E})$ and ${{\mathcal {I}}}_0$ at two particular values of the energy $\mathcal {E}_A\,(=\mathcal {E}_{\min })$, $\mathcal {E}_B\,(=\mathcal {E}_{2,3})$ mark the birth of the planar anti-aligned and aligned apsidal corotation orbits, respectively. Finally, we test the precision of various models defined either by a closed-form scissors-averaged model obtained from a multipole expansion of the Hamiltonian truncated at degree $N_P$, or by the usual (Jacobi-reduced) Laplace–Lagrange expansion. In both the previous cases, we introduce an additional ‘book-keeping’ order of truncation $N_\textrm{bk}$, with a book-keeping collecting at the same time powers of the eccentricities, of the mutual inclination and of the system’s $\textrm{AMD}$. As discussed in Sect. 2.4.2, we find that, in general, a multipole expansion with $N_P=5$ or 6 is sufficient to represent even non-hierarchical systems with semi-major axis ratio between the two planets $a_2/a_3=1/3$ or smaller. The multipole model has the salient features of being easy to scissors-average in closed form, while its lowest order truncations (quadrupolar and octupolar) yield formulas sufficiently small to offer analytical insight into the results. Comparison with the Laplace–Lagrange model (harder to obtain) shows the practical equivalence of the phase portraits at $N_P=5$ and with order of expansion as high as $N_\textrm{bk}=10$. On the other hand, obtaining the canonical transformation corresponding to the averaging, or going to second order in the masses, seems quite harder to achieve with the closed-form approach.

• (ii) Sect. 3 gives a qualitative overview of the main mechanism behind the transition from the ‘planar-like’ to the ‘Lidov–Kozai’ regime, i.e., the birth of the families $C_1,C_2$ via a saddle-node bifurcation around the aligned AC (mode B), accompanied by the transition of the family $C_1$ from unstable to stable via a pitchfork bifurcation, and, finally, the de-stabilization of the B-mode via an inverse pitchfork transformation. These phenomena are described qualitatively via the corresponding phase portraits, i.e., surfaces of section obtained numerically in a basic example by increasing the value of the energy $\mathcal {E}$. We demonstrate how this latter increase is connected to the overall rise of the level of mutual inclination of the orbits (Fig. 8). Finally, we numerically observe the Lidov–Kozai instability in the same phase portraits which turns the family $C_2$ from stable to unstable at a critical energy $\mathcal {E}_{C,2}$.

• (iii) Based on the form of the phase portraits, we identify three basic regimes in which a two-planets system with inclined orbits can be found: - The ‘planar-like’ regime is characterized by a phase-portrait similar to the one of an integrable model $\mathcal {H}_\textrm{int}$ (Sect. 3.2), which is the 3D analogue of the integrable model $\mathcal {H}_\textrm{planar}$. As pointed out in past literature (see Cushman and Bates 1997; Pauwels 1983; Henrard and Libert 2004; Michtchenko et al. 2006) this is a model whose natural phase space is the two-sphere, represented by the Hopf variables of Eq.(52). We first implement a tangency method (Cushman and Bates 1997; Palacián et al. 2006; Marchesiello and Pucacco 2016) to locate the periodic orbits of the integrable secular model. Then, by a normal form approach, we demonstrate how the perturbations to the integrable model can be normalized and absorbed in the definition of the periodic orbits corresponding to the apsidal corotations in the non-planar case. The main conclusion is that both the A (anti-aligned) and B (aligned) modes of the integrable model turn into orbits whose eccentricities are not exactly constant, but undergo periodic oscillations with a secular period easy to compute semi-analytically. At the same time, the pericenters of the planets in either mode do not remain locked exactly to the alignment or anti-alignment condition. Instead, the difference $\omega _2-\omega _3$ also undergoes oscillations around the central value (0 for anti-alignment, $\pi$ for alignment). The amplitude of these oscillations grows with the energy, and hence the level of mutual inclination, where each of these periodic orbits is computed. - The ‘transition regime’, starting with the birth of the families $C_1$, $C_2$ and ending with the termination of the intermediate ($D_{1,2}$) family at a collision with the B-mode rendering the latter unstable. - The ‘Lidov–Kozai’ regime, in which the structure of the phase space is similar to the one provided by the octupolar model. As well known in literature, the turning of the orbit $C_2$ from stable to unstable can be predicted analytically in the framework of the (integrable) quadrupolar approximation. We adapt this prediction to a proposition (Sect. 3.4) allowing to compute approximately the value of the transition energy $\mathcal {E}_{C,2}$ and of the corresponding critical inclination for a system with fixed $\textrm{AMD}$. Actually, a careful investigation of the same proposition shows that it can be applied also independently of the choice of parameter, i.e., using the outer planet’s eccentricity (which is constant in the context of the quadrupolar model) as well as altering the value of the systems $\textrm{AMD}$ (or angular momentum). At any rate, the result obtained by such an analysis is only approximate, at least for non-hierarchical systems, since the octupolar terms introduce a significant perturbation to the quadrupolar model. It is noteworthy that the effect of the octupolar terms can be locally absorbed by a perturbation theory eliminating all harmonics except for $\cos (2\omega _2)$ (attempts to provide such a theory in the framework of the restricted three-body problem are nearly as old as the subject itself, see Ito and Ohtsuka (2019) for a review). We are aware of no work attempting to provide the thresholds for the onset of the Lidov–Kozai instability in the framework of the general three-body problem at truncations $N_P\ge 3$ by analytical means (normal forms), thus, this is proposed as a subject for further study.

• (iv) We give an overview of how the picture of the phase-space structure and dynamics depicted as above applies to systems in which we have either a mass hierarchy, or a distance hierarchy. As regards models with mass hierarchy, we give numerical evidence that a necessary condition to obtain the full chain of transitions from the ‘planar-like’ to the ‘Lidov–Kozai’ regime is that $m_2$ be smaller than $m_3$. This is justified by the fact that the critical mutual inclination for obtaining the Lidov–Kozai instability is $\approx 39^\circ$ in the limit $m_2$ $\ll$  $m_3$, but it raises, instead, to $\approx 63^\circ$ in the limit $m_3<<m_2$. Such a high value of the mutual inclination is rather improbable in exoplanetary systems. On the other hand, a quick comparison of Figs. 8 and 9 shows that the so-called planar-like regime corresponds to a dynamical characterization (or, simply, the similarity of phase portraits with the planar case) rather than a really small value of the mutual inclination. For example, in our basic model the planar-like regime persists up to the energy $\mathcal {E}_C$ of birth of the $C_1,C_2$ families, which corresponds to trajectories whose mutual inclination can be as high as $35^\circ$. On the other hand, as can be deduced from Fig. 18, in models with distance hierarchy we arrive rather quickly to forms of the phase portraits reminiscent of those obtained in the Lidov–Kozai regime, i.e., with the $C_1$ and $C_2$ orbits rather, than the mode A and B orbits, dominating the dynamics. Insight to this phenomenon is provided by the fact that the octupolar model gives a far better approximation to the full Hamiltonian in distance-hierarchical cases. In that model, however, the harmonics $\cos (\omega _2-\omega _3)$, which is the basis for the appearance of apsidal corotation orbits, has smaller amplitude than the harmonics $\cos (2\omega _2)$ (already present at quadrupolar level). This implies that the dominant periodic orbits are $C_1$ and $C_2$, as well as the frozen orbits arising after the Lidov–Kozai instability. Note that the orbits $C_1$ and $C_2$ have aligned and anti-aligned apsides, while, by the position of the corresponding fixed points in the phase portraits, we readily deduce that they have smaller eccentricities than the one of the A mode (which remains stable at all values of the energy). Thus, once again, the difference between these periodic orbits is dynamical rather than morphological, i.e., they play distinct roles in shaping the phase space locally around them.

Appendices

Appendix A Analytic calculation of Laplace coefficients

A computation of the Laplace coefficients (Eq. (42), Sect. 2.4.2) via a multipolar expansion, is possible on the basis of the following.

Lemma 1

The Laplace coefficients $b_{s+\frac{1}{2}}^{(j)}$ in the expression

$$\begin{aligned} \frac{1}{\left( a_2^2 + a_3^2 - 2 \,a_2\,a_3\, \cos (\lambda _2 - \lambda _3)\right) ^{\frac{2\,s+1}{2}}}=a_3^{-(2\,s+1)}\sum _{j\ge 0} b_{s+\frac{1}{2}}^{(j)}\left( \alpha \right) \cos (j(\lambda _2-\lambda _3)), \end{aligned}$$

can be computed as

$$\begin{aligned} \begin{aligned} b_{s+\frac{1}{2}}^{(0)}\left( \alpha \right)&= 1+\sum _{k=1}^{+\infty }(-1)^{k}\, \frac{\frac{l}{2}(\frac{l}{2}+1)\ldots (\frac{l}{2}+k-1)}{k!}\,\alpha ^{2k}\\&\quad +\sum _{h\in \mathcal {A}_2}\sum _{k=h}^{\infty }(-1)^{k-h} \frac{(s+\frac{1}{2})(s+\frac{3}{2})\ldots (s-\frac{1}{2}+k)}{\left( \frac{h}{2}\right) !\left( \frac{h}{2}\right) !(k-h)!}\alpha ^{2k-h},\\ b_{s+\frac{1}{2}}^{(j)}\left( \alpha \right)&=\sum _{h\in \mathcal {A}_j}\sum _{k=h}^{\infty }2(-1)^{k-h} \frac{(s+\frac{1}{2})(s+\frac{3}{2})\ldots (s-\frac{1}{2}+k)}{\left( \frac{h-j}{2}\right) !\left( \frac{h+j}{2}\right) !(k-h)!}\alpha ^{2k-h},\quad j\ge 1, \end{aligned} \end{aligned}$$

(A1)

where $\alpha =a_2/a_3\,$ and $\mathcal {A}_j=\big \lbrace h\in {\mathbb {N}}\,:\,h={\left\{ \begin{array}{ll} 2i \quad \textit{if j is even}\\ 2i+1\quad \textit{if j is odd} \end{array}\right. },\,i\in {\mathbb {N}}\,,\, i\ge \lfloor \frac{j}{2}\rfloor \bigg \rbrace \,$, $j\ge 1\,$.

Proof

We will expand

$$\begin{aligned} \frac{1}{\left( a_2^2 + a_3^2 - 2 \,a_2\,a_3\, \cos (\lambda _2 - \lambda _3)\right) ^{\frac{2\,s+1}{2}}}=a_3^{-(2s+1)}\frac{1}{\left( 1+\alpha ^2-2\alpha \cos (\lambda _2 - \lambda _3)\right) ^{\frac{2\,s+1}{2}}},\nonumber \\ \end{aligned}$$

(A2)

in powers of $\alpha =a_2/a_3\,$. Defining $l=2s+1$ and $\sigma =\lambda _2-\lambda _3\,$ Taylor-expanding the r.h.s. in (A2), we find

$$\begin{aligned} \frac{1}{\left( 1+\alpha ^2-2\alpha \cos (\sigma )\right) ^{\frac{l}{2}}}=1+\sum _{k=1}^{+\infty }\frac{\frac{l}{2}(\frac{l}{2}+1)\ldots (\frac{l}{2}+k-1)}{k!}\,\left( 2\alpha \cos (\sigma )-\alpha ^2\right) ^k. \nonumber \\ \end{aligned}$$

(A3)

Moreover, we can expand the quantity $\left( 2\alpha \cos (\sigma )-\alpha ^2\right) ^k\,$, obtaining:

$$\begin{aligned} \begin{aligned}&\frac{1}{\left( 1+\alpha ^2-2\alpha \cos (\sigma )\right) ^{\frac{l}{2}}}=1+\sum _{k=1}^{+\infty }\frac{\frac{l}{2}(\frac{l}{2}+1)\ldots (\frac{l}{2}+k-1)}{k!}\,\left( 2\alpha \cos (\sigma )-\alpha ^2\right) ^k\, \\&\quad = 1+\sum _{k=1}^{+\infty }\frac{\frac{l}{2}(\frac{l}{2}+1)\ldots (\frac{l}{2}+k-1)}{k!}\,\left[ \sum _{h=0}^{k}\frac{k!}{h!(k-h)!}(-1)^{k-h}\alpha ^{2(k-h)}(2\alpha \cos (\sigma ))^h\right] \,\\&\quad = 1+\sum _{k=1}^{+\infty }\sum _{h=0}^{k}(-1)^{k-h}\, \frac{\frac{l}{2}(\frac{l}{2}+1)\ldots (\frac{l}{2}+k-1)}{h!(k-h)!}\alpha ^{2k-h}\,2^h\,\cos (\sigma )^h. \end{aligned}\nonumber \\ \end{aligned}$$

(A4)

After having explicitly written the sum corresponding to $h=0\,$, it is possible to reverse the order of the sums (over k and h), arriving to

$$\begin{aligned} \begin{aligned} \frac{1}{\left( 1+\alpha ^2-2\alpha \cos (\sigma )\right) ^{\frac{l}{2}}}&=1+\sum _{k=1}^{+\infty }(-1)^{k}\, \frac{\frac{l}{2}(\frac{l}{2}+1)\ldots (\frac{l}{2}+k-1)}{k!}\alpha ^{2k} \\&\quad +\sum _{h=1}^{+\infty }\sum _{k=h}^{+\infty }(-1)^{k-h}\,2^h\, \frac{\frac{l}{2}(\frac{l}{2}+1)\ldots (\frac{l}{2}+k-1)}{h!(k-h)!}\alpha ^{2k-h}\,\cos (\sigma )^h \\&:= \sum _{h=0}^{+\infty }D_{h}\cos (\sigma )^h, \end{aligned}\nonumber \\ \end{aligned}$$

(A5)

where $D_h\,$ (with $h\ge 0$) is defined as:

$$\begin{aligned} D_h= {\left\{ \begin{array}{ll} \displaystyle {1+\sum _{k=1}^{+\infty }(-1)^{k}\, \frac{\frac{l}{2}(\frac{l}{2}+1)\ldots (\frac{l}{2}+k-1)}{k!}\,\alpha ^{2k}}\quad \quad \,\,\,\textit{for}\quad h=0\\ \displaystyle {\sum _{k=h}^{+\infty }(-1)^{k-h}\,2^h\, \frac{\frac{l}{2}(\frac{l}{2}+1)\ldots (\frac{l}{2}+k-1)}{h!(k-h)!}\,\alpha ^{2k-h}}\quad \textit{for}\quad h\ge 1 \end{array}\right. }. \end{aligned}$$

(A6)

In order to compute the Laplace coefficients, we need to expand also the $\cos (\sigma )^h\,$ in (A5). We set (Eq. (41))

$$\begin{aligned} \frac{1}{\left( a_2^2 + a_3^2 - 2 \,a_2\,a_3\, \cos (\sigma )\right) ^\frac{l}{2}}=a_3^{-l}\sum _{h\ge 0} D_h\cos (\sigma )^h =a_3^{-l}\sum _{j\ge 0} b^{(j)}_\frac{l}{2}(\alpha )\cos (j\sigma ). \end{aligned}$$

Using again the binomial formula, we have

$$\begin{aligned} \cos (\sigma )^h\!=\!\left( \frac{e^{i\sigma }+e^{-i\sigma }}{2}\right) ^h\!\!\!\!=\frac{1}{2^h}\sum _{k=0}^{h} \left( {\begin{array}{c}h\\ k\end{array}}\right) e^{ik\sigma } e^{-i(h-k)\sigma }=\frac{1}{2^h}\sum _{k=0}^{h} \left( {\begin{array}{c}h\\ k\end{array}}\right) \cos ((2k-h)\sigma ).\nonumber \\ \end{aligned}$$

(A7)

Inserting th expression (A7) in Eq. (A5), we obtain

$$\begin{aligned} \begin{aligned} \frac{1}{\left( 1+\alpha ^2-2\alpha \cos (\sigma )\right) ^{\frac{l}{2}}}=&D_0+\sum _{h=1}^{+\infty }\sum _{k=0}^{h} \frac{D_{h}}{2^h}\frac{h!}{k!(h-k)!}\cos ((h-2k)\sigma )\\ =&D_0+\sum _{h=1}^{+\infty }\,\sum _{j\in \mathcal {B}_h } \frac{D_{h}}{2^h}\frac{h!}{\left( \frac{h-j}{2}\right) !\left( \frac{h+j}{2}\right) !}\cos (j\sigma ), \end{aligned} \end{aligned}$$

(A8)

where we define $j=h-2k$ and the set

$$\begin{aligned} \mathcal {B}_h=\left\{ j\in {\mathbb {Z}}: \, j= {\left\{ \begin{array}{ll} 2n \quad \textrm{if}\, h \,\mathrm{{is\, even}}, \, n\in {\mathbb {Z}},\,-\frac{h}{2}\le n \le \frac{h}{2}\\ 2n+1 \quad \textrm{if}\, h \,\mathrm{{is\, odd}}, \, n\in {\mathbb {Z}},\,\lfloor -\frac{h}{2}\rfloor \le n \le \lfloor \frac{h}{2}\rfloor \end{array}\right. } \right\} , \qquad h\ge 1. \end{aligned}$$

On the other hand, the previous set of indexes can be thought also as

$$\begin{aligned} \mathcal {B}_h= {\left\{ \begin{array}{ll} \mathcal {B}_{h_-} \cup \mathcal {B}_{h_+} \cup \lbrace j=0 \rbrace \quad \textrm{if}\, h \,\mathrm{{is\, even}}\\ \mathcal {B}_{h_-} \cup \mathcal {B}_{h_+} \quad \textrm{if}\, h \,\mathrm{{is \,odd}} \end{array}\right. }, \end{aligned}$$

where $\mathcal {B}_{h_-}=\mathcal {B}_{h}\cap {\mathbb {Z}}_{<0}$ and $\mathcal {B}_{h_+}=\mathcal {B}_{h}\cap {\mathbb {Z}}_{>0}\,$. With the above notation, the sum appearing in Eq. (A8) can be decomposed as follows:

$$\begin{aligned}&\sum _{h=1}^{+\infty }\,\sum _{j\in \mathcal {B}_h }=\sum _{\begin{array}{c} h=1\\ h\,\, \textrm{odd} \end{array}}^{+\infty }\,\left( \sum _{j\in \mathcal {B}_{h_-}} + \sum _{j\in \mathcal {B}_{h_+}}\right) +\sum _{\begin{array}{c} h=2\\ h\,\, \textrm{even} \end{array}}^{+\infty }\,\left( \sum _{j\in \mathcal {B}_{h_-}} + \sum _{j\in \mathcal {B}_{h_+}}+\sum _{j=0}\right) \, . \end{aligned}$$

Thus, observing that

$$\begin{aligned} \sum _{j\in \mathcal {B}_{h_-}} \frac{D_{h}}{2^h}\frac{h!}{\left( \frac{h-j}{2}\right) !\left( \frac{h+j}{2}\right) !}\cos (j\sigma )=\sum _{j\in \mathcal {B}_{h_+}} \frac{D_{h}}{2^h}\frac{h!}{\left( \frac{h-j}{2}\right) !\left( \frac{h+j}{2}\right) !}\cos (j\sigma ), \end{aligned}$$

we can write Eq. (A8) as:

$$\begin{aligned}&\frac{1}{\left( 1+\alpha ^2-2\alpha \cos (\sigma )\right) ^{\frac{l}{2}}}\\&\quad = \sum _{\begin{array}{c} h=0\\ h\,\,\text {even} \end{array}}^{+\infty }\,\frac{D_{h}}{2^h}\frac{h!}{\left( \frac{h}{2}\right) !\left( \frac{h}{2}\right) !}+\sum _{\begin{array}{c} h=2\\ h\, \textrm{even} \end{array}}^{+\infty }\,\sum _{j\in \mathcal {B}_{h_+} } \frac{D_{h}}{2^{h-1}}\frac{h!}{\left( \frac{h-j}{2}\right) !\left( \frac{h+j}{2}\right) !}\cos (j\sigma )\\&\qquad +\sum _{\begin{array}{c} h=1\\ h\,\, \text {odd} \end{array}}^{+\infty }\,\sum _{j\in \mathcal {B}_{h_+} }\frac{D_h}{2^{h-1}}\frac{h!}{\left( \frac{h-j}{2}\right) !\left( \frac{h+j}{2}\right) !}\cos (j\sigma )\, . \end{aligned}$$

Finally, reversing the order of the sums (over j and $h\,$), we arrive at

$$\begin{aligned} \begin{aligned}&\frac{1}{\left( 1+\alpha ^2-2\alpha \cos (\sigma )\right) ^{\frac{l}{2}}}\\&\quad =\sum _{\begin{array}{c} h=0\\ h\,\, \textrm{even} \end{array}}^{+\infty }\,\frac{D_{h}}{2^h}\frac{h!}{\left( \frac{h}{2}\right) !\left( \frac{h}{2}\right) !}+\sum _{\begin{array}{c} j=2\\ j\,\, \textrm{even} \end{array}}^{+\infty }\,\sum _{h\in \mathcal {A}_{j} } \frac{D_{h}}{2^{h-1}}\frac{h!}{\left( \frac{h-j}{2}\right) !\left( \frac{h+j}{2}\right) !}\cos (j\sigma )\\&\qquad +\sum _{\begin{array}{c} j=1\\ j\, \textit{odd} \end{array}}^{+\infty }\,\sum _{h\in \mathcal {A}_{j} }\frac{D_h}{2^{h-1}}\frac{h!}{\left( \frac{h-j}{2}\right) !\left( \frac{h+j}{2}\right) !}\cos (j\sigma )\\&\quad =\sum _{\begin{array}{c} h=0\\ h\,\, \textrm{even} \end{array}}^{+\infty }\,\frac{D_{h}}{2^h}\frac{h!}{\left( \frac{h}{2}\right) !\left( \frac{h}{2}\right) !}+\sum _{j=1}^{+\infty }\,\sum _{h\in \mathcal {A}_{j} } \frac{D_{h}}{2^{h-1}}\frac{h!}{\left( \frac{h-j}{2}\right) !\left( \frac{h+j}{2}\right) !}\cos (j\sigma ), \end{aligned} \end{aligned}$$

(A9)

where

$$\begin{aligned} \mathcal {A}_j=\left\{ h\in {\mathbb {N}}: \, h= {\left\{ \begin{array}{ll} 2i \quad \textrm{if}\, j \,\mathrm{{is\, even}}, \, i\in {\mathbb {N}},\,i\ge \frac{j}{2}\\ 2i+1 \quad \textrm{if} j \,\mathrm{{is odd}}, \, i\in {\mathbb {N}},\,i\ge \lfloor \frac{j}{2}\rfloor \end{array}\right. } \right\} , \qquad j\ge 1. \end{aligned}$$

Now we can finally put Eq. (A9) in the expression (A2) and compare the obtained quantity with (41), yielding

$$\begin{aligned}&b_{s+\frac{1}{2}}^{(0)}(\alpha ){=}D_0{+}\sum _{h\in \mathcal {A}_2}\frac{D_h}{2^{h}}\frac{h!}{\left( \frac{h}{2}\right) !\left( \frac{h}{2}\right) !}\, ,{} & {} b_{s+\frac{1}{2}}^{(j)}(\alpha ){=}\sum _{h\in \mathcal {A}_j}\frac{D_h}{2^{h-1}}\frac{h!}{\left( \frac{h-j}{2}\right) !\left( \frac{h+j}{2}\right) !}\, , \end{aligned}$$

(A10)

with $\mathcal {A}_2=\left\{ h\in {\mathbb {N}}\,: \, h=2 i\,, \, i\in {\mathbb {N}}\,,\,i\ge 1 \right\} \,$. Finally, substituting in the previous expression (A10) the definition of $D_h$ (given by (A6)) and remembering that $l=2s+1\,$ and $\sigma =\lambda _2-\lambda _3\,$, we obtain (A1). This concludes the proof. $\square$

Appendix B Normal form around the periodic orbits A or B

The normal form function $Z^{(r)}$ and the normalizing transformation $\Phi ^{(r)}$ of Eqs. (61) and (62) are computed by the following steps:

• (i) Development of the Hamiltonian around the apsidal solution of the integrable part $\mathcal {H}_\textrm{int}$: Let $(\psi _{*},\,\Gamma _{*},\,J_{*})$ be one of the two fixed points (apsidal corotation solutions) of the integrable Hamiltonian $\mathcal {H}_\textrm{int}(\psi ,\,\Gamma ;J)\,$. We consider the translations

  $$\begin{aligned} \psi =\psi _{*}+\varepsilon _*\delta \psi , \qquad \Gamma =\Gamma _{*}+\varepsilon _*\,\delta \Gamma ,\qquad J=J_{*}+\varepsilon _{J_*}\,\delta J, \end{aligned}$$

  (B1)

  where $\varepsilon _*$, $\varepsilon _{J_*}$ are both book-keeping symbols with numerical value equal to one. We then perform the following algebraic operations:

  • Substitute (B1) into the Hamiltonian $\mathcal {H}_\textrm{sec}(\psi ,\varphi ,\Gamma ,J)$ and expand the Hamiltonian in powers of the symbols $\varepsilon _*$, $\varepsilon _{J_*}$;

  • in the resulting expression, replace $\cos (\varphi )$ and $\sin (\varphi )$ with $\varepsilon _*\cos (\varphi )$ and $\varepsilon _*\sin (\varphi )$, respectively;

  • assign a unique book-keeping symbol to the expanded Hamiltonian $\mathcal {H}_\textrm{sec}(\delta \psi ,\,\varphi ,\,\delta \Gamma ,\,$$\delta J\,)$, by setting $\varepsilon _{J_*}^{a}\rightsquigarrow \,\lambda ^{a-1}\,\, a\ge 1\,$, $\varepsilon _*\rightsquigarrow \,\lambda \,$;

  • truncate the resulting expression at a maximum order $N_t$ in the book-keeping symbol $\lambda$.

  After performing the above algebraic steps, the Hamiltonian resumes the form

  $$\begin{aligned} \begin{aligned} \mathcal {H}^{(0)}=\mathcal {H}_\textrm{sec}&= c_1\,+Z_0 +\sum _{s=1}^{N_t}\lambda ^s h_{s}^{(0)}(\delta \psi ,\,\varphi ,\,\delta \Gamma ,\,\delta J\,) \\&= c_1\,+\nu _*^{(0)}\delta J +\sum _{s=1}^{N_t}\lambda ^s \left( \sum _{m,\,k,\,n,\,l}\theta ^{(0)}_{m,k,n,l}\,\delta \psi ^{m}\delta \Gamma ^{n} \delta J^{l}\,e^{i\,k\,\varphi } \right) \end{aligned} \end{aligned}$$

  (B2)

  where $c_1$ is a constant term, $\theta ^{(0)}_{m,k,n,l}$ are constant coefficients, and $\nu _*^{(0)}<0$ is the (also constant) unperturbed frequency of the apsidal periodic orbit in the integrable approximation.

• (ii) Normal form to eliminate $\varphi$: The Hamiltonian (B2) is normalized by iterative steps using the method of composition of Lie series. The normalization algorithm is defined recursively, for $r=1,2,\ldots$ by the relations:

  $$\begin{aligned} \mathcal {H}^{(r-1)}= & {} c_1\,+Z_0 +\sum _{s=1}^{r-1}\lambda ^s Z_s(\delta \psi ,\delta \Gamma ,\delta J)\nonumber \\{} & {} +\sum _{s=r}^{N_t}\lambda ^s \left( \sum _{m,\,k,\,n,\,l} \theta _{m,k,n,l}^{(r-1)}\,\delta \psi ^{m}\delta \Gamma ^{n} \delta J^{l}\,e^{i\,k\,\varphi } \right) , \end{aligned}$$

  (B3)
  $$\begin{aligned} \chi ^{(r)}= & {} \lambda ^{r}\!\!\!\sum _{\begin{array}{c} m,\,k,\,n,\,l\\ k\ne 0 \end{array}}\frac{\theta ^{(r-1)}_{m,k,n,l}}{i\,\nu _*^{(0)}\,k}\, \delta \psi ^{m}\delta \Gamma ^{n} \delta J^{l}\,e^{i\,k\,\varphi }, \end{aligned}$$

  (B4)
  $$\begin{aligned} \mathcal {H}^{(r)}= & {} \Bigg [\exp (L_{\chi ^{(r)}})\mathcal {H}^{(r-1)}\Bigg ]^{\le N_t}, \end{aligned}$$

  (B5)

  where $L_{\chi ^{(r)}}$ denotes the Poisson bracket operator $L_{\chi ^{(r)}}\cdot =\{\cdot ,\chi ^{(r)}\}$, and $[\cdot ]^{\le N_t}$ means truncation at the order $N_t$ in the book-keeping parameter $\lambda$.

• (iii) Final Hamiltonian and normalizing transformation: The normalizing transformation $\Phi ^{(r)}$ is defined by:

  $$\begin{aligned} \begin{aligned}&(\delta \psi ,\varphi ,\delta \Gamma ,\delta J)= \Phi ^{(r)}(\delta \tilde{\psi },\tilde{\varphi },\delta \tilde{\Gamma },\delta \tilde{J}) \\&\quad = \Bigg [ \left( \exp \left( L_{\chi ^{(r)}}\right) \exp \left( L_{\chi ^{(r-1)}}\right) \ldots \exp \left( L_{\chi ^{(1)}}\right) (\delta \psi ,\varphi ,\delta \Gamma ,\delta J)\right) \Bigg |_{\begin{array}{c} \delta \psi ={\delta \tilde{\psi }}\\ \varphi =\tilde{\varphi }\\ \delta \Gamma ={\delta \tilde{\Gamma }}\\ \delta J={\delta \tilde{J}} \end{array}} \Bigg ]^{\le N_t} \end{aligned} \end{aligned}$$

  (B6)

  and the final Hamiltonian

  $$\begin{aligned} \begin{aligned} \mathcal {H}^{(r)}&= \Bigg [ \left( \exp \left( L_{\chi ^{(r)}}\right) \exp \left( L_{\chi ^{(r-1)}}\right) \ldots \exp \left( L_{\chi ^{(1)}}\right) \mathcal {H}^{(0)}\right) \Bigg |_{\begin{array}{c} \delta \psi ={\delta \tilde{\psi }}\\ \varphi =\tilde{\varphi }\\ \delta \Gamma ={\delta \tilde{\Gamma }}\\ \delta J={\delta \tilde{J}} \end{array}} \Bigg ]^{\le N_t}~ \\&= Z^{(r)}(\delta \tilde{\psi },\,\delta \tilde{\Gamma },\,\delta \tilde{J}\,) +R^{(r)}(\delta \tilde{\psi },\,\tilde{\varphi },\,\delta \tilde{\Gamma },\,\delta \tilde{J}\,), \end{aligned} \end{aligned}$$

  (B7)

  where $1\le r\le N_t$ is chosen so that the normal form term $Z^{(r)}(\delta \tilde{\psi },\,\delta \tilde{\Gamma },\,\delta \tilde{J}\,)$ yields the best possible approximation to the dynamics.

  Note that the constant term $c_1$ produced at step 1 above has to be carried along all the successive Hamiltonians $\mathcal {H}^{(r)}$ produced through the normalization process, being eventually included in the normal form $Z^{(r)}$. This is necessary, since this constant appears in the third of the algebraic equations (65) by which the fixed point corresponding to the apsidal periodic orbit is computed in the new canonical variables.

Appendix C Proof of Proposition 3.1

Proof

The eigenvalues of the matrix M (75) are $\lambda _{1,2}=\pm \sqrt{{\mathfrak {a}}{\mathfrak {b}}}\,$; then if ${\mathfrak {a}}{\mathfrak {b}}>0\,$, then the eigenvalues are real and opposite, instead if ${\mathfrak {a}}{\mathfrak {b}}<0\,$, then the eigenvalues are complex and conjugate. We have to analyze all the possible cases. We can have

$$\begin{aligned}&{\left\{ \begin{array}{ll} {\mathfrak {a}}>0\\ {\mathfrak {b}}>0 \end{array}\right. }\, ,{} & {} {\left\{ \begin{array}{ll} {\mathfrak {a}}<0\\ {\mathfrak {b}}<0 \end{array}\right. } \quad \Longrightarrow \quad \lambda _{1,2}\quad \textrm{real} \end{aligned}$$

(C1a)

$$\begin{aligned}&{\left\{ \begin{array}{ll} {\mathfrak {a}}>0\\ {\mathfrak {b}}<0 \end{array}\right. }\, ,{} & {} {\left\{ \begin{array}{ll} {\mathfrak {a}}<0\\ {\mathfrak {b}}>0 \end{array}\right. } \quad \Longrightarrow \quad \lambda _{1,2}\quad \textrm{imaginary}\, . \end{aligned}$$

(C1b)

Let us start to understand the change of sign of ${\mathfrak {b}}\,$; remembering the definition (74)

$$\begin{aligned} {\mathfrak {b}}=\frac{3 \mathcal {G}^2 L_2^3\, (3 G_3^2 - L_2^2 + L_z^2)\, m_0\, m_3^7}{8 \,G_3^5\, L_3^3\, m_2^3}>0 \quad \Longleftrightarrow \quad \tilde{{\mathfrak {b}}}:= L_z^2 +3L_3^2(1-e_3^2)-L_2^2>0, \end{aligned}$$

being $(3 \mathcal {G}^2 L_2^3 m_0 m_3^7)/(8 (1-e_3^2)^{5/2} L_3^8 m_2^3)>0\,$. Now, let us observe that if

$$\begin{aligned} C:=L_2^2-3L_3^2(1-e_3^2)<0\quad \Longrightarrow \quad e_3^2<1-\frac{L_2^2}{3 L_3^2}\, , \end{aligned}$$

(C2)

then $\tilde{{\mathfrak {b}}}>0$ automatically; in order to have that, it is necessary that $1-\frac{L_2^2}{3 L_3^2}>0\,$, i.e., to have $0< L_2< \sqrt{3}L_3\,$. Thus, we can conclude that ${\mathfrak {b}}>0$ ( or equivalently $\tilde{{\mathfrak {b}}}>0\,$) if one of the two is fulfilled

• (i) $0< L_2<\sqrt{3}L_3 \,\,\wedge \,\,0\le e_3\le \sqrt{1-\frac{L_2^2}{3L_3^2}}\,$, with $L_z^2>0\,$ or

• (ii) $\left( 0< L_2<\sqrt{3}L_3 \,\,\wedge \,\,e_3>\sqrt{1-\frac{L_2^2}{3L_3^2}}\right) \,\,\vee \,\,L_2\ge \sqrt{3}L_3\,$, with $L_z^2>C\,$.

Now, let us study the sign of ${\mathfrak {a}}\,$; as before, from definition (74)

$$\begin{aligned} \begin{aligned}&{\mathfrak {a}}=-\frac{3 \mathcal {G}^2 L_2 (5 G_3^4 - 4 G_3^2 L_2^2 + 3 L_2^4 - 10 G_3^2 L_z^2 - 8 L_2^2 L_z^2 + 5 L_z^4) m_0 m_3^7}{16 G_3^5 L_3^3 m_2^3}>0\quad \Longleftrightarrow \\&\tilde{{\mathfrak {a}}}:= L_3^4(1-e_3^2)^2 - \frac{4}{5} L_3^2 L_2^2(1-e_3^2) + \frac{3}{5} L_2^4 - 2 L_3^2(1-e_3^2) L_z^2 - \frac{8}{5} L_2^2 L_z^2 + L_z^4<0, \end{aligned}\nonumber \\ \end{aligned}$$

(C3)

being $-(3 \mathcal {G}^2 L_2\, m_0 \,m_3^7)/(16\, L_3^8(1-e_3)^{5/2} m_2^3)<0\,$. Then ${\mathfrak {a}}>0$ (or equivalently $\tilde{{\mathfrak {a}}}<0\,$) iff $A<L_z^2<B\,$, where A and B are defined as in (76), i.e.,

$$\begin{aligned}&A=\frac{1}{5}\left( 4L_2^2+5 L_3^2(1-e_3^2)-L_2\sqrt{L_2^2+60 L_3^2(1-e_3^2)}\right) \, ,\\&B=\frac{1}{5}\left( 4L_2^2+5 L_3^2(1-e_3^2)+L_2\sqrt{L_2^2+60 L_3^2(1-e_3^2)}\right) \, . \end{aligned}$$

Let us observe that A and B are both greater than zero. In fact, being $0\le e_3 < 1\,$, it is obvious that $B>0\,$; on the other hand $A>0$ iff

$$\begin{aligned}&\sqrt{L_2^2+60 L_3^2(1-e_3^2)}<4L_2+\frac{5 L_3^2}{L_2}(1-e_3^2){} & {} \Longleftrightarrow \\&L_2^2+60 L_3^2(1-e_3^2)<16 L_2^2 + 25 \frac{L_3^4}{L_2^2}(1-e_3^2)^2+40 L_3^2(1-e_3^2){} & {} \Longleftrightarrow \\&3L_2^4-4 L_3^2 (1-e_3^2) L_2^2 + 5 (1-e_3^2)^2 L_3^4 > 0 \end{aligned}$$

that holds $\forall \,\, L_2,L_3, 0\le e_3< 1\,$, being $\Delta =-44 L_3^4 (1-e_3^2)^2 <0\,$. Then, we can conclude that if we are in the case i), (i.e., $\displaystyle {0< L_2<\sqrt{3}L_3 \,\,\wedge \,\,0\le e_3\le \sqrt{1-\frac{L_2^2}{3L_3^2}}}\,$) and $A< L_z^2< B\,$, then ${\mathfrak {b}}>0\,$ and ${\mathfrak {a}}>0\,$, then we have real eigenvalues; instead if, in the same case i), $0<L_z^2< A\,$ or $L_z^2> B\,$, then ${\mathfrak {b}}>0\,$, ${\mathfrak {a}}<0\,$, having complex imaginary eigenvalues. This proves the Case i) of the Proposition.

In the case ii), i.e., $\left( 0< L_2<\sqrt{3}L_3 \,\,\wedge \,\,e_3>\sqrt{1-\frac{L_2^2}{3L_3^2}}\right) \,\,\vee \,\,L_2\ge \sqrt{3}L_3\,$, we have to solve (C1a) and (C1b), that become, respectively

$$\begin{aligned}&{\left\{ \begin{array}{ll} {\mathfrak {a}}>0\\ {\mathfrak {b}}>0 \end{array}\right. } \, {=}\, {\left\{ \begin{array}{ll} A<L_z^2< B\\ L_z^2>C \end{array}\right. }\, ,{} & {} {\left\{ \begin{array}{ll} {\mathfrak {a}}<0\\ {\mathfrak {b}}<0 \end{array}\right. } \, {=}\, {\left\{ \begin{array}{ll} 0<L_z^2< A \, \vee \, L_z^2>B\\ 0<L_z^2<C \end{array}\right. }\, , \end{aligned}$$

(C4a)

$$\begin{aligned}&{\left\{ \begin{array}{ll} {\mathfrak {a}}>0\\ {\mathfrak {b}}<0 \end{array}\right. }\, {=}\, {\left\{ \begin{array}{ll} A<L_z^2< B\\ 0<L_z^2<C \end{array}\right. }\, ,{} & {} {\left\{ \begin{array}{ll} {\mathfrak {a}}<0\\ {\mathfrak {b}}>0 \end{array}\right. }\, {=}\, {\left\{ \begin{array}{ll} 0<L_z^2< A \, \vee \, L_z^2>B\\ L_z^2>C \end{array}\right. }\, ; \end{aligned}$$

(C4b)

then, we have to understand the position of the value C with respect to A and $B\,$.

Case 1::

         $C< A$

$$\begin{aligned} C< A\quad \Longleftrightarrow \quad \sqrt{L_2^2+60 L_3^2(1-e_3^2)}< -L_2+ \frac{20 L_3^2}{L_2}(1-e_3^2) \end{aligned}$$

i.e.,

$$\begin{aligned} {\left\{ \begin{array}{ll} \displaystyle {-L_2+ \frac{20 L_3^2}{L_2}(1-e_3^2)\ge 0}\\ \displaystyle {L_2^2+100 L_3^2(1-e_3^2)< L_2^2+ \frac{400 L_3^4}{L_2^2}(1-e_3^2)^2}\, \end{array}\right. } \Longleftrightarrow \, {\left\{ \begin{array}{ll} \displaystyle {e_3^2\le 1-\frac{L_2^2}{20 L_3^2} }\\ \displaystyle {e_3^2< 1-\frac{L_2^2}{4 L_3^2} }\,, \end{array}\right. } \end{aligned}$$

then if $e_3^2 < 1-\frac{L_2^2}{4 L_3^2}\,$. The last inequality makes sense if $1-\frac{L_2^2}{4 L_3^2}> 0\,$, that is $0\le L_2< 2 L_3\,$. Remembering that we are in the case ii), it follows that

$$\begin{aligned} C<A \quad \Longleftrightarrow \quad&\left( 0< L_2<\sqrt{3}L_3 \,\,\wedge \,\,\sqrt{1-\frac{L_2^2}{3L_3^2}}< e_3< \sqrt{1-\frac{L_2^2}{ 4 L_3^2}} \right) \, \, \vee \, \,\\&\left( \sqrt{3}L_3\le L_2< 2 L_3 \,\,\wedge \,\,0\le e_3 < \sqrt{1-\frac{L_2^2}{ 4 L_3^2}}\right) \, . \end{aligned}$$

Therefore, studying (C4a) and (C4b), we can conclude that if $\left( 0< L_2<\sqrt{3}L_3 \,\,\wedge \,\,\right.$$\left. \sqrt{1-\frac{L_2^2}{3L_3^2}}< e_3 < \sqrt{1-\frac{L_2^2}{ 4 L_3^2}} \right)$ $\vee$ $\bigg (\sqrt{3}L_3\le L_2< 2 L_3 \,\,\wedge \,\,0\le e_3 < \sqrt{1-\frac{L_2^2}{ 4 L_3^2}}\bigg )\,$, then the eigenvalues are real and opposite if $0<L_z^2<C\,$ (see the second of (C4a)), purely imaginary and conjugates if $C<L_z^2<A\,$ (see the second of (C4b)), real and opposite if $A<L_z^2<B$ (see the first of (C4a)) and again purely imaginary and conjugates if $L_z^2>B\,$ (see the second of (C4b)), proving the Case ii) of the Proposition.

• Case 2: $C= A$

Following the same calculations done for the previous case ($C<A\,$), it easily follows that

$$\begin{aligned} C= A\quad \Longleftrightarrow \quad&\left( 0< L_2\le 2 L_3 \,\,\wedge \,\, e_3 = \sqrt{1-\frac{L_2^2}{ 4 L_3^2}} \right) \,\, ; \end{aligned}$$

in this case, studying (C4a) and (C4b), we can conclude that the eigenvalues are real and opposite if $0<L_z^2<C=A\,$ (see the second of (C4a)), again real and opposite if $C=A<L_z^2<B$ (see the first of (C4a)) (i.e., are real and opposite if $0<L_z^2<B\,$, passing from 0 if $L_z^2=A=C\,$) and purely imaginary and conjugates if $L_z^2>B\,$ (see the second of (C4b)). This proves the Case iii) of the Proposition.

• Case 3: $A< C< B$

$$\begin{aligned}&C> A\quad\Longleftrightarrow & {} \quad \sqrt{L_2^2+60 L_3^2(1-e_3^2)}> -L_2+ \frac{20 L_3^2}{L_2}(1-e_3^2) \end{aligned}$$

(C5)

$$\begin{aligned}&C< B\quad\Longleftrightarrow & {} \quad \sqrt{L_2^2+60 L_3^2(1-e_3^2)}> L_2- \frac{20 L_3^2}{L_2}(1-e_3^2)\, . \end{aligned}$$

(C6)

Let us start from (C5); we have to solve the following

$$\begin{aligned}&{\left\{ \begin{array}{ll} \displaystyle {-L_2+ \frac{20 L_3^2}{L_2}(1-e_3^2)\ge 0}\\ \displaystyle {L_2^2+100 L_3^2(1-e_3^2)> L_2^2+ \frac{400 L_3^4}{L_2^2}(1-e_3^2)^2}\, \end{array}\right. } \quad \wedge \quad {\left\{ \begin{array}{ll} \displaystyle {-L_2+ \frac{20 L_3^2}{L_2}(1-e_3^2)< 0}\\ \end{array}\right. } \end{aligned}$$

that gives

$$\begin{aligned} \left( {\left\{ \begin{array}{ll} \displaystyle {e_3^2\le 1{-}\frac{L_2^2}{20 L_3^2} }\\ \displaystyle {e_3^2> 1{-}\frac{L_2^2}{4 L_3^2} }\, \end{array}\right. } \quad \wedge \quad {\left\{ \begin{array}{ll} \displaystyle {e_3^2> 1{-}\frac{L_2^2}{20 L_3^2} }\\ \end{array}\right. }\right) \qquad \Longrightarrow \qquad e_3^2> 1{-}\frac{L_2^2}{4 L_3^2}. \end{aligned}$$

(C7)

Similarly, we can solve (C6), having the following

$$\begin{aligned}&{\left\{ \begin{array}{ll} \displaystyle {L_2- \frac{20 L_3^2}{L_2}(1-e_3^2)\ge 0}\\ \displaystyle {L_2^2+100 L_3^2(1-e_3^2)> L_2^2+ \frac{400 L_3^4}{L_2^2}(1-e_3^2)^2}\, \end{array}\right. } \quad \wedge \quad {\left\{ \begin{array}{ll} \displaystyle {L_2- \frac{20 L_3^2}{L_2}(1-e_3^2)< 0}\\ \end{array}\right. } \end{aligned}$$

that gives

$$\begin{aligned} \left( {\left\{ \begin{array}{ll} \displaystyle {e_3^2\ge 1-\frac{L_2^2}{20 L_3^2} }\\ \displaystyle {e_3^2> 1-\frac{L_2^2}{4 L_3^2} }\, \end{array}\right. } \quad \wedge \quad {\left\{ \begin{array}{ll} \displaystyle {e_3^2< 1-\frac{L_2^2}{20 L_3^2} }\\ \end{array}\right. }\right) \qquad \Longrightarrow \qquad \forall \,e_3. \end{aligned}$$

(C8)

Finally, putting together (C7) and (C8), we get $e_3^2> 1-\frac{L_2^2}{4 L_3^2}\,$. Let us observe that if $1-\frac{L_2^2}{4 L_3^2}< 0\,$, i.e., $L_2> 2 L_3\,$, then the last inequality for $e_3$ is automatically satisfy. Then, remembering that we are in the case ii) and putting together the results, it follows that

$$\begin{aligned} A<C<B \quad \Longleftrightarrow \quad&\left( 0< L_2\le 2 L_3 \,\,\wedge \,\, e_3> \sqrt{1-\frac{L_2^2}{ 4 L_3^2}} \right) \quad \vee \quad \left( L_2 > 2 L_3\right) \, . \end{aligned}$$

Again, studying (C4a) and (C4b), we can conclude that if $\displaystyle {\left( 0< L_2\le 2 L_3 \,\,\wedge \,\, e_3 > \right. }$$\displaystyle {\left. \sqrt{1-\frac{L_2^2}{ 4 L_3^2}} \right) }$ $\vee$ $L_2 > 2 L_3\,$, then the eigenvalues are real and opposite if $0<L_z^2<A\,$ (see the second of (C4a)), purely imaginary and conjugates if $A<L_z^2<C\,$ (see the first of (C4b)), real and opposite if $C<L_z^2<B$ (see the first of (C4a)) and again purely imaginary and conjugates if $L_z^2>B\,$ (see the second of (C4b)). This proves the last Case iv) of the Proposition.

Actually, there is a last case to be analyzed, that will not give any contribution, i.e.,

• Case 4: $C\ge B$

$$\begin{aligned} C\ge B\quad \Longleftrightarrow \quad \sqrt{L_2^2+60 L_3^2(1-e_3^2)}\le L_2- \frac{20 L_3^2}{L_2}(1-e_3^2) \end{aligned}$$

i.e.,

$$\begin{aligned} {\left\{ \begin{array}{ll} \displaystyle {L_2- \frac{20 L_3^2}{L_2}(1-e_3^2)\ge 0}\\ \displaystyle {L_2^2+100 L_3^2(1-e_3^2)\le L_2^2+ \frac{400 L_3^4}{L_2^2}(1-e_3^2)^2}\, \end{array}\right. } \quad \Longleftrightarrow \quad {\left\{ \begin{array}{ll} \displaystyle {e_3^2\ge 1-\frac{L_2^2}{20 L_3^2} }\\ \displaystyle {e_3^2\le 1-\frac{L_2^2}{4 L_3^2} } \end{array}\right. } \end{aligned}$$

that has no solutions. This concludes the proof of the Proposition. $\square$