Introduction

We know that the classical planar Restricted three body problem(R3BP) possesses five libration points, two triangular and three collinear. The collinear libration point L 1,L 2 and L 3, are unstable for all values of mass parameter μ, while the two triangular libration points L 4 and L 5 form an equilateral triangle with the primaries and are stable for μ<μ c =0.03852... Szebehely [20]. Deprit and Deprit [6] calculated the exceptional values for the triangular libration points and proved that the nonlinear stability of these points can be answered in the affirmative for all values of the mass ratio in the range of linear stability except at three mass ratios 0.024293..., 0.013516... and 0.010913.... Bhatnagar and Hallan [4] studied the effect of perturbations ε and \(\varepsilon ^{\prime }\) in the coriolis and the centrifugal forces respectively on the nonlinear stability of the libration points in the restricted problem and found that the collinear points remain unstable for all mass parameter μ, and for the triangular points obtained the relation \(\mu _{c} =\mu _{0} +\frac {4\left ( {36\varepsilon -19{\varepsilon }^{\prime }} \right )}{27\sqrt {69} }\)thereby establishing that the range of stability increases or decreases depending upon whether the point \(\left ( {\varepsilon ,{\varepsilon }^{\prime }} \right )\) lies in one or the other of two parts in which the \(\left ( {\varepsilon ,{\varepsilon }^{\prime }} \right )\) plane is divided by the line \(\left ( {36\varepsilon -19{\varepsilon }^{\prime }} \right )=0\). Chandra and Kumar [5] have studied the effect of oblateness on the nonlinear stability of the triangular libration points of the restricted three-body problem in the presence of resonance when the more massive primary is an oblate spheroid. They have applied KAM theorem with the help of Markeev’s theorem. Aggarwal et al. [1] have also studied the nonlinear stability of the triangular libration point L 4 of the restricted three-body problem under the presence of third and fourth order resonances when the bigger primary is an oblate body and the smaller a triaxial body and both are source of radiation. It is found through Markeev’s theorem that triangular points are always unstable in the third order resonance case and stable or unstable in the fourth order resonance case depending upon the values of the parameters A 1, A 2, P 1 and P 2. Kushvah et al. [12] have discussed the nonlinear stability in the generalized photogravitational restricted three-body problem with Poynting-Robertson drag. They have considered smaller primary an oblate spheroid and bigger primary as radiating. They have found using KAM theorem that triangular points are stable except three critical mass ratios. Hallan and Mangang [11] studied the effect of perturbations in coriolis and centrifugal forces on the nonlinear stability of equilibrium point in Robe’s restricted circular three-body problem. Singh [17] has studied the effect of small perturbations ε and \(\varepsilon ^{\prime }\) in the coriolis and the centrifugal forces on the nonlinear stability of the triangular points in the restricted three-body problem with variable mass and found that the triangular points are stable for all mass ratios in the range of stability except for three mass ratios, which depends upon ε, \(\varepsilon ^{\prime }\) and β, the constant due to the variation in mass governed by Jeans’ law. Singh [18] has also studied the combined effects of perturbations in the coriolis and the centrifugal forces, radiation and oblateness on the nonlinear stability of the triangular points in the restricted three-body problem. Bombardelli and Pelaez [3] have discussed the stability of artificial equilibrium points in the circular restricted three-body problem. They analyzed that the stability is found when the distance from the second primary exceeds a minimum value which is a simple function of the mass ratio of the two primaries and their separations. Lyapunov stability under non-resonant conditions is demonstrated using Arnold’s theorem. Jain and Sinha [9] studied the nonlinear stability of the triangular libration point L 4 in the restricted three-body problem when the smaller primary is a finite straight segment of length 2l under the presence of third and fourth order resonances. For these two resonance cases Moser’s theorem is not applicable. She has used Markeev’s theorem to check the stability. For this they have normalized the Hamiltonian up to fourth order. They found that the triangular stationary solutions are always unstable in the third and fourth order resonance case for 0≤l<0.05. Suraj et al. [19] studied the photo-gravitational R3BP when the primaries are heterogeneous spheroid with three layers. They showed the effect due to heterogeneous oblate spheroid and source of radiation on the position of equilibrium point L 4 and L 5. Further they observed that as the perturbation all or one is being taken into account the equilibrium point L 4 and L 5 moves away slightly from the line joining the primaries. Also they observed that the triangular points are stable for 0≤μ<μ c r i t and the range of stability is reduced by the perturbation. The triangular libration points are unstable for μ c μ<0.5, where μ c is the critical mass parameter influenced by the both primaries taken as heterogeneous oblate spheroid of three layers and sources of radiation pressure. However they have discussed the stability of libration points in linear sense. Singh and Narayan [16] have studied the nonlinear stability of the triangular libration points for radiating and oblate primaries in CR3BP in non-resonance condition.

For the last so many years the literature of celestial mechanics is full of a number of research papers in the restricted three-body problem, where the primaries are either point masses or spherical in shape and also the many perturbing forces such as oblateness, radiation forces, coriolis, centrifugal forces, variation of the masses of the primaries and of the infinitesimal mass etc., have been included in the study of the restricted three body problem (Ishwar et al. [8]), Hallan et al. [10], and Yan-ning et al. [22]. But generally, the celestial bodies are axis-symmetric bodies and not homogeneous in density therefore, we have taken into account the effect due to different layers with different density. A heterogeneous rotating planet with a rigid crust cannot be modeled by a rotating stratified spheroidal system. We proposed this system of physical interest for many reasons. One of them is a heterogeneous rigid bodies more realistic than homogeneous incompressible spheroidal rotating system.The replacement of mass point by rigid-body is quite important because of its wide applications in practical problems. In general, scientists have taken the primaries as homogeneous point masses or spherical in shape but in my paper we have taken the primary as heterogeneous triaxial rigid body with 3 layers.

We have got the idea of our problem from the paper ‘Rotating Stratified Heterogeneous Oblate Spheroid in Newtonian Physics’ by Esteban and Vazquez [7]. By solving the Euler hydro dynamical equations they have obtained closed form solutions for the angular velocities and pressures of a three stratified non-confocal heterogeneous oblate spheroid. Limiting and particular solutions cases, such as a spheroid with N layers, a stratified spheroid with the same eccentricities, as well as confocal layered spheroids are also explicitly written down. In this paper we study the nonlinear stability of the triangular libration point L 4, in the restricted three body problem when the smaller primary with mass m 2 is a heterogeneous spheroid with three layers, having different densities ρ i and axes \(\left ( {a_{i} ,c_{i} } \right ),\left ( {i=1,2,3} \right )\) respectively. For this we will apply Moser’s modified version of Arnold’s theorem [2] and will follow the procedure as adopted by Bhatnagar and Hallan [4].

Arnold proved that if

  1. (a)

    K 1 ω 1 + K 2 ω 2≠0, for all pairs \(\left ({K_{1} ,\mathrm {K}_{2} } \right )\) of rational integers, where ω 1,ω 2 are the basic frequencies for the linear dynamical system and

  2. (b)

    The determinant D ≠ 0, where

    $$\begin{array}{@{}rcl@{}} D&=&\det\left( {b_{ij} } \right),\left( {i,j\,=1,2,3} \right),\\ b_{ij} &=& \left( {\frac{\partial^{2}H}{\partial I_{i} \partial I_{j} }} \right)_{I_{i} =I_{j} =0} \left( {i,j=1,2} \right),\\ b_{i3} &=& b_{3i} =\left( {\frac{\partial H}{\partial I_{i} }} \right)_{I_{i} =I_{j} =0}, \\ b_{33} &=& 0, \\ H&=&\omega_{1} I_{1} +\omega_{2} I_{2} +\frac{1}{2}\left( {A{I_{1}^{2}} +2BI_{1} I_{2} +C{I_{2}^{2}} } \right)+..., \end{array} $$

    is the normalized Hamiltonian with I 1 and I 2 as the action momenta co-ordinates, then on each energy manifold H = h in the neighborhood of equilibrium, there exists invariant tori of quasi periodic motions which divide the manifold and consequently the equilibrium is stable. Applying Arnold’s theorem, Leontovich [13] proved that the triangular points in the restricted problem are stable for all permissible mass ratios but a set of measure zero. Moser has shown that Arnold’s Theorem [2] is true if the condition (a) of the theorem is replaced by K 1 ω 1 + K 2 ω 2≠0, for all pairs ( K 1,K 2) of rational integers such that \(\left | {K_{1} } \right |+\left | {K_{2} } \right |\le 4\).

Using the first condition of the theorem, we have found two critical mass ratios \(\mu _{1}^{\prime }, \mu _{2}^{\prime }\) where this condition fails. By taking the second order coefficients, we have calculated the determinant D occurring in the second condition of the theorem. From this, we have found the third critical mass ratio \(\mu _{3}^{\prime }\), where the second condition of the theorem fails.

This paper should be read in conjunction with the papers by Bhatnagar and Hallan [4], Suraj et al. [19] and Shalini and Jha [15] as, to save space, we are not mentioning the values of various variables given in those papers, although they are used in this paper.

This paper is organized as follows: In “Introduction” we have reviewed the literature related to R3BP under different perturbations. In “Equations of Motion” we have derived the potential of heterogeneous spheroid with three layers and further, formulated the equations of motion of the proposed system. “First Order Normalisation”, deals with the first order normalization. In “Second Order Normalization”, we have determined the second order normalization. In “Second Order Coefficients in the Frequencies”, we have found the second order frequencies. In “Stability”, we have checked the nonlinear stability of triangular libration points. “Conclusion”, contains the discussion and conclusion of the obtained results.

Equations of Motion

Consider three masses m 1,m 2 and m 3. The bodies with masses m 1 and m 2 revolve with the angular velocity n (say) in circular orbits without rotation about their centre of mass O and the mass \(m_{3}\left ( {m_{3} <<m_{1} ,m_{2} } \right )\) is moving in the plane of motion of m 1 and m 2 and is being influenced by their motion but not influencing them. We consider the smaller primary with mass m 2 a spheroid with three layers having different densities ρ i (Fig. 1) and axes \(\left ({a_{i} ,c_{i} } \right ) \left ({i=1,2,3} \right )\), and its equatorial plane is coincident with the plane of motion. Let us consider a synodic system of coordinates O(xyz), initially coincident with the inertial system O(XYZ),rotating with the angular velocity n about the Z−axis; (the z−axis is coincident with Z−axis). We suppose, initially, the principal axes of the body with mass m 2, are parallel to the synodic axes O(xyz) and the axis of symmetry of the body with mass m 2 is perpendicular to the plane of motion (Fig. 1).

Fig. 1
figure 1

Configuration of R3BP when smaller primary is heterogeneous spheroid

Now, the gravitational potential of the body of mass m 2, at the point P is

$$V_{2} =V_{33} +V_{23} +V_{13}, $$

where V 33,V 23 and V 13 are the potential of the spheroid of densities ρ i for the regions III, II and I respectively.

Now

$$V_{33} ={V}_{33}^{\prime} -{V}_{32}^{\prime}, $$

where \({V}_{33}^{\prime }\) and \({V}_{32}^{\prime }\) are the potentials of the spheroid of axes (a 3,c 3) and (a 2,c 2) respectively with homogeneous density ρ 3 throughout at P. Thus, we have

$$\begin{array}{@{}rcl@{}} {V}^{\prime}_{33} &=& \frac{-4\pi \rho_{3} G}{3r_{2}}\left\{a_{3}^{2} c_{3} \left( 1+\frac{1}{10r_{2}^{2}}\left( a_{3}^{2} -c_{3}^{2}\right)\right)\right\}, \\ {V}^{\prime}_{32} &=& \frac{-4\pi \rho_{3} G}{3r_{2} }\left\{a_{2}^{2} c_{2} \left( 1+\frac{1}{10r_{2}^{2}}\left( a_{2}^{2} -c_{2}^{2}\right)\right)\right\}. \\ \end{array} $$

Thus,

$$V_{33} =\frac{-4\pi \rho_{3} G}{3r_{2} }\left\{a_{3}^{2} c_{3} \left( 1 +\frac{1}{10r_{2}^{2}}\left( a_{3}^{2} -c_{3}^{2} \right) \right)-a_{2}^{2} c_{2} \left( 1 +\frac{1}{10r_{2}^{2}}\left( a_{2}^{2} -c_{2}^{2}\right)\right) \right\}, $$

similarly,

$$V_{23} =\frac{-4\pi \rho_{2} G}{3r_{2}}\left\{a_{2}^{2} c_{2} \left( 1+\frac{1}{10r_{2}^{2}}\left( a_{2}^{2} -c_{2}^{2} \right) \right)- a_{1}^{2} c_{1} \left( 1 +\frac{1}{10r_{2}^{2}}\left( a_{1}^{2} - c_{1}^{2}\right)\right)\right\}, $$

and

$$V_{13} =\frac{-4\pi \rho_{1} G}{3r_{2} }\left\{a_{1}^{2} c_{1} \left( 1 +\frac{1}{10r_{2}^{2}}\left( a_{1}^{2} - c_{1}^{2}\right)\right)\right\}. $$

Hence adding these equations we get

$$V_{2} =-\frac{4\pi G}{3r_{2}}\sum\limits_{i=1}^{3}\left( \rho_{i} -\rho_{i+1}\right) a_{i}^{2} c_{i} \left( 1+\frac{\left( a_{i}^{2} -c_{i}^{2}\right)}{10r_{2}^{2}} \right) =-\frac{m_{2} G}{r_{2}}-\frac{kG}{2r_{2}^{3}}, $$

where

$$\begin{array}{@{}rcl@{}} m_{2} &=&\frac{4\pi }{3}\sum\limits_{i=1}^{3} \left( \left( \rho_{i} -\rho_{i+1}\right) a_{i}^{2} c_{i}\right), \\ k &=& \frac{4\pi }{3}\sum\limits_{i=1}^{3} \left( \left( \rho_{i} -\rho_{i+1} \right) a_{i}^{2} c_{i} \sigma_{i} \right), \\ \sigma_{i} &=&\frac{\left( a_{i}^{2}-c_{i}^{2} \right)}{5},\,\left( i=1,2,3 \right),\,\rho_{4} =0, \end{array} $$

Hence the total potential at P due to m 1 and m 2 is given by

$$V=-\frac{Gm_{1} }{r_{1} }+\left( -\frac{m_{2} G}{r_{2}}-\frac{kG}{2r_{2}^{3}}\right). $$

Now, we choose the unit for the length and mass such that the distance between the primaries as one and sum of their masses as one i.e. m 1 + m 2=1. The unit of time is so chosen that the gravitational constant becomes unity. Let μ = m 2/(m 1 + m 2)<1/2, then m 2 = μ add m 1=1−μ. Adopting the notations and terminology of Szebehely [20] and using the dimensionless variables, the equations of motion in a synodic co-ordinates system are

$$\begin{array}{@{}rcl@{}} \ddot{{x}}-2n\dot{{y}}&=&{\Omega}_{x} ,\\ \ddot{{y}}+2n\dot{{x}}&=&{\Omega}_{y} , \end{array} $$
(1)

where

$$\begin{array}{@{}rcl@{}} {\Omega}&=&\frac{n^{2}}{2}\left( x^{2}+y^{2}\right)+\frac{1-\mu }{r_{1}}+\frac{\mu }{r_{2}} + \frac{k_{1}}{2r_{2}^{3}},\\ {r_{1}^{2}} &=&\left( x+\mu \right)^{2}+y^{2},\quad r_{2}^{2} =\left( x+\mu -1 \right)^{2}+y^{2}, \\ k_{1} &=&\frac{4\pi }{3}\sum\limits_{i=1}^{3}\left( \rho_{i}^{\prime} - \rho_{i+1}^{\prime}\right) a_{i}^{\prime 2} c_{i}^{\prime} \sigma_{i}^{\prime},\\ a_{i}^{\prime} &=& \frac{a_{i} }{R},\,\, c_{i}^{\prime} =\frac{c_{i}}{R},\sigma_{i}^{\prime} =\frac{a_{i}^{\prime 2} - c_{i}^{\prime 2} }{10R^{2}},\\ \rho_{i}^{\prime} &=& \frac{\rho_{i} }{M},\,\,\,\rho_{4}^{\prime} =0,\,\,\,\,k_{1} <<1,\\ M &=& m_{1} +m_{2} ,\\ a_{i} ,c_{i} &=& \text{the semi-axes of the spheroid},\\ R &=& \text{dimensional distance between the primaries}. \end{array} $$

The mean motion n of the primaries is given by

$$n=1+\frac{3k_{2}}{4}, $$

where

$$k_{2} =\frac{\sum\limits_{i=1}^{3}\left( \left( \rho_{i}^{\prime} -\rho_{i+1}^{\prime} \right) a_{i}^{\prime2} c_{i}^{\prime} \sigma_{i}^{\prime}\right)}{\sum\limits_{i=1}^{3}\left( \left( \rho_{i}^{\prime} -\rho_{i+1}^{\prime} \right) a_{i}^{\prime 2} c_{i}^{\prime}\right)}<<1\,,\,\,\rho_{4}^{\prime} =0. $$

Here, the location of triangular libration points are solutions of the system obtained by making all the derivatives of Ω equals to zero (i.e. Ω x =0 and Ω y =0). Solving these equations, we get three collinear and two non-collinear libration points. The co-ordinates of the noncollinear libration points L 4(x,y) and L 5(x,−y) are given by Shalini and Jha [15]

$$x=\mu -\frac{1}{2}+\frac{k_{2} }{2},\,\,\,\,\,y=\frac{\sqrt 3 }{2}\left( 1-\frac{k_{2} }{3}\right). $$

First Order Normalisation

In order to investigate the non linear stability of non-collinear libration point L 4, it is necessary to reduce the Hamiltonian to its normalized form. So, we perform the first and second order normalization. The stability can be investigated by using KAM theorem, if H 2 Eq. (4) is not a function of definite sign. The Liapunov’s [14] theorem states, if H 2 is of positive definite form, then the libration point is stable for all orders and all time. We shall follow the procedure as adopted by Bhatnagar and Hallan [4] to perform the first order normalization.

The Lagrangian of the Eq. 1 is given by

$$ {\Gamma} =\frac{1}{2}\left\{ \dot{{x}}^{2}+\dot{{y}}^{2}+n^{2}\left( x^{2}+y^{2} \right)+2\,n\left( x\,\dot{{y}}- y\,\dot{{x}}\right)\right\}+\frac{1-\mu }{r_{1} }+\frac{\mu }{r_{2} }+\frac{k_{1}}{2\,{r_{2}^{3}} }. $$
(2)

Now shifting the origin to \(L_{4} \left ( {x\,,\,y} \right )\) and expanding Γ in power series of x and y, it can be expressed as:

$${\Gamma} ={\Gamma}_{0} +{\Gamma}_{1} +{\Gamma}_{2} +{\Gamma}_{3} +{\Gamma}_{4} +..., $$

where

$$\begin{array}{@{}rcl@{}} {\Gamma}_{0} &=&\frac{11+\gamma^{2}}{8}+\frac{k_{1} }{2}+\frac{3}{16}\left( 3+\gamma^{2}\right)k_{2},\\ {\Gamma}_{1} &=&\left( -\frac{\sqrt 3 }{2}+\left( \frac{1}{3}-\frac{3\sqrt 3 }{8}\right)k_{2} \right)\dot{{x}}+ \left\{\frac{\gamma }{2}-\frac{1}{8}\left( 4-3\gamma\right)k_{2}\right\} \dot{{y}}\\ &&+\frac{3}{8}\left( 2k_{1} +\left( -1+2\gamma-2\sqrt 3 \,\gamma \right)k_{2}\right)x \\ &&+\frac{3}{4}\left( -\sqrt 3 k_{1} +\left( -1+\sqrt 3 -\sqrt 3 \,\gamma\right)k_{2}\right)\text{y,} \\ {\Gamma}_{2} &=&\frac{1}{2}\left( \dot{{x}}^{2}+\dot{{y}}^{2}\right)+n\left( x\dot{{y}}-y\dot{{x}}\right)+ \left( \frac{3}{8}+\frac{3k_{1}}{16}+\left( \frac{3}{4}{+}\frac{\sqrt 3}{16}-\frac{21\gamma }{32} \right)k_{2} \right)x^{2} \\ &&+\left( \frac{9}{8}+\frac{33k_{1} }{16}+\left( \frac{3}{4}+\frac{3\sqrt 3 }{16}+\frac{33\gamma}{32} \right)k_{2} \right)y^{2}\\ &&+\left( \frac{3\sqrt 3 \gamma }{4}-\frac{15\sqrt 3 k_{1} }{8}+\left( \frac{3\sqrt 3 }{16}+\frac{11\gamma }{8}\right)k_{2}\right)xy,\\ {\Gamma}_{3} &=&\left( \frac{7\gamma }{16}-\frac{25k_{1} }{32}+\left( \frac{37}{64}+\frac{25\gamma }{32\sqrt 3 }\right)k_{2} \right)x^{3}\\ &&+\left( -\frac{3\sqrt 3 }{16}-\frac{45\sqrt 3 k_{1} }{32}+\left( -\frac{41}{32}+\frac{75\sqrt 3 \gamma }{64}\right)k_{2}\right)x^{2}y \\ &&+\left( -\frac{33\gamma }{16}+\frac{255k_{1} }{32}+\left( -\frac{123}{64}-\frac{45\sqrt 3 \gamma }{32}\right)k_{2} \right)xy^{2}\\ &&+\left( -\frac{3\sqrt 3 }{16}-\frac{45\sqrt 3 k_{1} }{32}+\left( -\frac{1}{32}-\frac{45\sqrt 3 \gamma }{64} \right)k_{2}\right)y^{3}, \end{array} $$
$$\begin{array}{@{}rcl@{}} {\Gamma}_{4} &=& \left( -\frac{37}{128}-\frac{285k_{1} }{256}+\left( -\frac{95\sqrt 3 }{256}+\frac{115\gamma }{512} \right)k_{2}\right)x^{4}\\ &&+\left( -\frac{25\sqrt 3 \gamma }{32}+\frac{105\sqrt {3\,}k_{1} }{64}-\frac{285\sqrt 3 k_{2} }{128}-\frac{215k_{2} \gamma }{192}\right)x^{3}y \end{array} $$
$$\begin{array}{@{}rcl@{}} &&{\kern10pt}+\left( \frac{123}{64}+\frac{1395k_{1} }{128}+\left( \frac{345\sqrt 3 }{128}-\frac{645\gamma }{256}\right)k_{2} \right)x^{2}y^{2}\\ &&{\kern10pt}+\left( \frac{45\sqrt 3 \gamma}{32}-\frac{525\sqrt{3\,} k_{1} }{64}+\frac{345\sqrt 3 k_{2} }{128}+ \frac{185\,k_{2} \gamma }{64}\right)x\,y^{3}\\ &&{\kern10pt}+\left( \frac{3}{128}+\frac{255k_{1} }{256}+\left( -\frac{55\sqrt 3 \,}{256}+\frac{555\gamma }{512}\right)k_{2} \right)y^{4}. \end{array} $$

Corresponding to the Lagrangian Γ given by Eq. 2, the Hamiltonian function is given by:

$$ H=-{\Gamma} +p_{x} \dot{{x}}+p_{y} \dot{{y}}, $$
(3)

where p x and p y are the momenta coordinates and given by

$$p_{x} =\frac{\partial {\Gamma} }{\partial\dot{{x}}}=\dot{{x}}-ny,\,\,\,\,\,\,p_{y} =\frac{\partial {\Gamma} }{\partial \dot{{y}}}=\dot{{y}}+nx. $$

Finally, the Hamiltonian function (3) becomes

$$H(x,y,p_{x} ,p_{y} )=\frac{1}{2}\left( p_{x}^{2} + p_{y}^{2} \right)+n\left( yp_{x} -xp_{y}\right)-\frac{1-\mu }{r_{1} }-\frac{\mu}{r_{2} }-\frac{k_{1} }{2\,{r_{2}^{3}} }, $$

After this we apply the following translation:

$$x\to x+\frac{\gamma }{2}-\frac{\mathrm{k}_{2}}{2},\,\,\,y\to y+\frac{\sqrt 3}{2}-\frac{\mathrm{k}_{2}}{3}, $$
$$p_{x} \to p_{x} -n\left( {\frac{\sqrt 3 }{2}-\frac{\mathrm{k}_{2}}{3}} \right),\,\,\,\,\,\,p_{y} \to p_{y} +n \left( \frac{\gamma }{2}-\frac{\mathrm{k}_{2} }{2}\right), $$

which transforms the Hamiltonian H given in Eq. 3 to \(H=\sum \limits _{k=0}^{\infty } {H_{k} } \), where H k = the sum of the terms of k th degree homogenous in variables x,y,p x ,p y .

Now,

$$\begin{array}{@{}rcl@{}} H_{0} &=&-{\Gamma}_{0} ,\\ H_{1} &=&-\frac{3}{8}\left( 2k_{1} +\left( -1+2\gamma -2\sqrt 3 \,\gamma \right)k_{2}\right)x + \frac{3}{4}\left( -\sqrt 3 k_{1} +\left( -1+\sqrt 3 -\sqrt 3 \,\gamma \right)k_{2}\right)\text{y,}\\ H_{2} &=&\frac{1}{2}\left( p_{x}^{2} + p_{y}^{2}\right)+n\left( yp_{x}-xp_{y}\right)+Ex^{2}+Fy^{2}+2Gxy, \\ H_{3} &=&-{\Gamma}_{3} ,\,\,\,\,\,\,H_{4} =-{\Gamma}_{4} , \end{array} $$
(4)

where

$$\begin{array}{@{}rcl@{}} \text{E} &=&- \left( \frac{1}{8}+\frac{3k_{1} }{16}+\left( \frac{3}{4}+\frac{\sqrt 3 }{16}-\frac{21\gamma }{32}\right)k_{2}\right),\\ F &=& \frac{5}{8}+\frac{33k_{1} }{16}+\left( \frac{3}{4}+\frac{3\sqrt 3}{16}+\frac{33\gamma }{32}\right)k_{2} , \\ G &=&-\left( \frac{3\sqrt 3 \gamma }{2}-\frac{15\sqrt 3 k_{1}}{4}+\left( \frac{3\sqrt 3 }{8}+\frac{11\gamma }{4}\right)k_{2}\right). \end{array} $$

To investigate the stability of motion as in Whittaker [21], we consider the following set of linear equations in the variables x and y:

$$\begin{array}{@{}rcl@{}} -\lambda p_{x} &=&\frac{\partial H_{2}}{\partial x}=2Ex+Gy-np_{y} , {\kern4pc}\lambda x=\frac{\partial H_{2}}{\partial p_{x} }=p_{x} +ny, \\ -\lambda p_{y} &=&\frac{\partial H_{2}}{\partial y}=2Fy+Gx+np_{x} , {\kern4pc}\lambda y=\frac{\partial H_{2}}{\partial p_{y} }=p_{y} -nx, \end{array} $$
(5)

i.e. AX = 0, where \(A=\left (\begin {array}{cccc} {2E} & G & \lambda & {-n} \\ G & {2F} & n & \lambda \\ {-\lambda } & n & 1 & 0 \\ {-n} & {-\lambda } & 0 & 1 \end {array}\right ), \quad X=\left (\begin {array}{c} x \\ y \\ {p_{x} } \\ {p_{y} } \end {array}\right )\).

The Eq. 5 will have a non zero solution if and only if det (A) = 0, which implies that

$$\lambda^{4}+2\lambda^{2}\left( {E+F+n^{2}} \right)+EF-G^{2}-2n^{2}\left( {E+F}\right)+n^{4}=0. $$

So the characteristic equation corresponding to Hamiltonian H 2 is given in Eq. 4 is given by

$$\begin{array}{@{}rcl@{}} 16\lambda^{4}&+&\left( 16-72k_{1} +\left( 48-8\sqrt 3 -12\gamma\right)k_{2}\right) \lambda^{2}+27 \left( 1-\gamma^{2}\right) + \left( 63+135\gamma \right)k_{1} \\ &+&\left( 72 +9\sqrt 3 -36\gamma -33\sqrt 3 \gamma^{2}\right)k_{2} =0. \end{array} $$

Stability is assured only when the discriminant of the characteristic equation is greater than zero, implying that

$$\mu <\mu_{c} =\mu_{0} -(2.24318....)k_{1} +(0.0588081...)k_{2} , $$

where μ 0=0.0385208965....

When D>0,the roots \(\pm i{\omega }^{\prime }_{1}\) and \(\pm i \omega _{2}^{\prime }\) (\(\omega _{1}^{\prime }\) and \(\omega _{2}^{\prime }\) being long/short-periodic frequencies) are related to each other as

$$\begin{array}{@{}rcl@{}} \omega_{1}^{\prime 2} + \omega_{2}^{\prime 2} &=& 1-\frac{9k_{1}}{2}+\left( 3-\frac{\sqrt 3 }{2}-\frac{3\gamma}{4} \right)k_{2}, \end{array} $$
(6)
$$\begin{array}{@{}rcl@{}} \omega_{1}^{\prime 2}\omega_{2}^{\prime2} &\,=\,&\! \frac{27}{16}\left( \!1\,-\,\gamma^{2}\right)\,+\,\left( \frac{63}{16}+\frac{135\gamma }{16}\!\right)k_{1} \,+\,\left( \!\frac{9\sqrt 3}{16}+\frac{135\gamma }{16}-\frac{9\gamma }{4}-\frac{33\sqrt 3 \gamma^{2}}{16}\! \right)k_{2},\\ &&\left( 0<\omega_{1}^{\prime} <\omega_{2}^{\prime} <\frac{1}{\sqrt 2 }\right), \end{array} $$
(7)

It may be noted that the perturbed frequencies \((\omega _{1}^{\prime }, \omega _{2}^{\prime })\) are related to the unperturbed one (ω 1,ω 2) as

$$ \omega_{1}^{\prime} =\omega_{1} \left( 1+pk_{1} +{p}^{\prime}k_{2} \right),\omega_{2}^{\prime} =\omega_{2} \left( 1+qk_{1} +{q}^{\prime}k_{2}\right), $$
(8)

where

$$\begin{array}{@{}rcl@{}} p&=&-q=-\frac{9\left( 7+15\gamma +8 \omega_{1}^{2} \right)}{32\omega_{1}^{2} k_{2}}, \qquad k^{2}=2\omega_{1}^{2} -1=1-2 \omega_{2}^{2} , \\ {p}^{\prime}&=&-{q}^{\prime}=-\frac{-162+54\sqrt 3 +81\gamma +\left( 108-62\sqrt 3 -27\lambda \right)\omega_{1}^{2} +44\sqrt 3 \omega_{1}^{4}}{72\omega_{1}^{2} k^{2}}. \end{array} $$

Following the method given in Whittaker [21], we use a canonical transformation from the phase space \(\left ( {x,y,p_{x} ,p_{y} } \right )\) into the phase space of the angles \(\left ( {\varphi _{1} ,\varphi _{2} } \right )\) and the actions \(\left ( {I_{1} ,I_{2} } \right )\), so that the Hamiltonian H 2 be normalized.

$$X=JT, $$

where

$$\begin{array}{@{}rcl@{}} X&=&\left[\begin{array}{l} x \\ y \\ p_{x} \\ p_{y} \end{array} \right]\,,\,\,\,\,\,\,\,\,T=\left[ \begin{array}{l} Q_{1} \\ Q_{2} \\ P_{1} \\ P_{2} \end{array} \right],\,\,\,\,J=\left( {{a}^{\prime}_{ij} } \right)_{1\le i,j\le 4} ,Q_{i} =\left( {\frac{2I_{i} }{{\omega }^{\prime}_{i} }} \right)^{\frac{1}{2}}sin\,\varphi_{i} , \\ P_{i} &=&\left( {2I_{i} {\omega }^{\prime}_{i} } \right)^{\frac{1}{2}}cos\varphi_{i} ,(i=1,2). \end{array} $$

Now we have calculated all the elements \(a_{ij}^{\prime }\) of J given by

$${a}_{ij}^{\prime} =a_{ij} \left( {1+\alpha_{ij} k_{1} +{\alpha }_{ij}^{\prime} k_{2} } \right),\,\,i\,,\,j=1,\,2,\,3,\,4. $$

where

$$\begin{array}{@{}rcl@{}} a_{11} &=&a_{12} =0,\,\\ a_{13} &=&\frac{l_{1} }{2\omega_{1} k_{1} },\,\,\,\,a_{14} =\frac{l_{2} }{2\omega_{2} k_{2} }, a_{21} =\frac{-4\omega_{1} }{l_{1} k},\,\,\,\,\,a_{22} =\frac{-4\omega_{2} }{l_{2} k}, a_{23} =\frac{3\sqrt 3 \gamma }{2\omega_{1} l_{1} k},\,\,\,a_{24} =\frac{3\sqrt 3 \gamma }{2\omega_{2} l_{2} k},\\ a_{31} &=&\frac{-\omega_{1} m_{1} }{2l_{1} k},\,\,\,\,a_{32} =\frac{-\omega_{2} m_{2} }{2l_{2} k}, \quad a_{33} =\frac{3\sqrt 3 \gamma }{2\omega_{1} l_{1} k},a_{34} =\frac{3\sqrt 3 \gamma }{2\omega_{2} l_{2} k}, \quad a_{41} =\frac{3\sqrt 3 \gamma \omega_{1} }{2l_{1} k},a_{42} =\frac{3\sqrt 3 \gamma \omega_{2} }{2l_{2} k}, \\ a_{43} &=&\frac{n_{1} }{2\omega_{1} l_{1} k},\,\,a_{44} =\frac{n_{2} }{2\omega_{2} l_{2} k}, \quad \alpha_{13} =\frac{w_{1} }{2k^{2}p_{1} {l_{1}^{2}} }+\frac{w_{2} p}{k^{2}p_{1} {l_{1}^{2}} },\, {\alpha }_{13}^{\prime} =\frac{w_{3} }{4k^{2}p_{1} {l_{1}^{2}} }-\frac{w_{4} {p}^{\prime}}{k^{2}{l_{1}^{2}} },\\ \alpha_{21} &=&\frac{w_{9} }{2k^{2}p_{1} }+\frac{w_{10} p}{k^{2}p_{1} {l_{1}^{2}} }, {\alpha }_{21}^{\prime} =\frac{w_{11} }{4k^{2}p_{1} {l_{1}^{2}} }+\frac{w_{12} {p}^{\prime}}{k^{2}{l_{1}^{2}} }, \alpha_{23} =\frac{w_{17} }{2k^{2}p_{1} \gamma }+\frac{w_{18} p}{k^{2}p_{1} {l_{1}^{2}} },\\ {\alpha }_{23}^{\prime} &=&\frac{w_{19} }{36\sqrt 3 \gamma k^{2}p_{1} {l_{1}^{2}} }-\frac{w_{20} {p}^{\prime}}{k^{2}{l_{1}^{2}} }, n_{i} =9-4{\omega_{i}^{2}} ,\,\,{l_{i}^{2}} =9+4{\omega_{i}^{2}} \,,\,p_{i} =3+4{\omega_{i}^{2}} \,,\,\,\,\,\,\,\,\left( {i=1,2} \right). \end{array} $$

The values of α i j and \({\alpha }_{ij}^{\prime }\)’s for j=1,2. can be obtained from those for j=1,3 respectively by replacing ω 1 by ω 2, l 1 by l 2, m 1 by m 2 ,n 1 by n 2 whenever they occur, keeping k unchanged and all the values of w i ’s \(\left ( {i=1,2,3....,24} \right )\) are given in Appendix 1.

The transformation changes the second order part of the Hamiltonian into the normal form \(H_{2} =\omega _{1}^{\prime } I_{1} -\omega _{2}^{\prime } I_{2} \) and the general solutions of the corresponding equations of motion are

$$I_{i} =\text{ Constant }\left( {i=1,2} \right)\,,\varphi_{1} =\omega_{1}^{\prime} t+\text{Constant},\varphi_{2} =-\omega_{2}^{\prime} t+\text{Constant}. $$

Second Order Normalization

Moser’s conditions are utilised for transforming the Hamiltonian to the Birkhoff’s normal form with the help of double D’Alembert’s series. Here we wish to perform Birkhoff”s normalization for which the co-ordinates \(\left ( {x,y}\right )\) are to be expanded in double D’Alembert series:

$$x=\sum\limits_{n\ge 1} {B_{n}^{1,0} } , y=\sum\limits_{n\ge 1} {B_{n}^{0,1} } , $$

where the homogeneous components \(B_{n}^{1,0} \) and \(B_{n}^{0,1} \) of degree n in \(\sqrt {I_{1} } ,\sqrt {I_{2} }\), are of the form

$$\begin{array}{@{}rcl@{}} \sum\limits_{0\le m\le n} {I_{1}^{\frac{1}{2(n-m)}} }I_{2}^{\frac{1}{2(n-m)}} \sum\limits_{\left( {i,j} \right)} {\left( {C_{n-m,m,i,j} \,cos\left( {i\varphi_{1} +j\varphi_{2} } \right)+S_{n-m,m,i,j} \,\sin \left( {i\varphi_{1} +j\varphi_{2} } \right)} \right)} , \end{array} $$
(9)

The double summation over the indices i and j is such that (a) i runs over those integers in the interval 0≤inm that have the same parity as nm (b) j runs over those integers in the interval −mjm that have the same parity as m. I 1 and I 2 are to be regarded as constants of integration and φ 1,φ 2 are to be determined as linear functions of time such that

$$\dot{{\varphi }}_{1} =\omega_{1}^{\prime} +\sum\limits_{n\ge 1} {f_{2n} } \left( {I_{1} ,I_{2} } \right), \quad \dot{{\varphi }}_{2} =-\omega_{2}^{\prime} +\sum\limits_{n\ge 1} {g_{2n} } \left( {I_{1} ,I_{2} } \right), $$

where f 2n and g 2n are of the form

$$f_{2n} =\sum\limits_{0\le m\le n} {{f}_{2\left( {n-m} \right),2m}^{\prime} I_{1}^{n-m} {I_{2}^{m}} ,} \quad g_{2n} =\sum\limits_{0\le m\le n} {{g}_{2\left( {n-m} \right),2m}^{\prime} I_{1}^{n-m} {I_{2}^{m}} .} $$

The first order components \(B_{1}^{1,0} \) and \(B_{1}^{0,1} \)are the values of x and y given by Eq. 9. The second order components\(B_{2}^{1,0} \) and \(B_{2}^{0,1} \)are solutions of the partial differential equations

$${\Delta}_{1} {\Delta}_{2} B_{2}^{1,0} ={\Phi}_{2} \text{ and }{\Delta}_{1} {\Delta}_{2} B_{2}^{0,1} =\psi_{2}, $$

where

$$\begin{array}{@{}rcl@{}} {\Delta}_{i} &=&\left( {D^{2}+{\omega }_{i}^{\prime2} } \right),\,\,\,\,\,\left( {i=1,2} \right), \quad D=\omega_{1}^{\prime} \frac{\partial }{\partial \varphi_{1} }-\omega_{2}^{\prime} \frac{\partial }{\partial \varphi_{2} },\\ {\Phi}_{2} &=&\left( {A_{1} D^{2}+A_{2} D+A_{3} } \right)\left( {B_{1}^{1,0} } \right)^{2}+\left( {B_{1} D^{2}+B_{2} D+B_{3} } \right)\left( {B_{1}^{0,1} } \right)^{2}\\ &&+\left( {C_{1} D^{2}+C_{2} D+C_{3} } \right)\left( {B_{1}^{1,0} } \right)\left( {B_{1}^{0,1} } \right), \\ \psi_{2} &=&\left( {{A}_{1}^{\prime} D^{2}+{A}_{2}^{\prime} D+{A}_{3}^{\prime} } \right)\left( {B_{1}^{1,0} } \right)^{2}+\left( {{B}_{1}^{\prime} D^{2}+{B}_{2}^{\prime} D+{B}_{3}^{\prime} } \right)\left( {B_{1}^{0,1} } \right)^{2}\\ &&+\left( {{C}_{1}^{\prime} D^{2}+{C}_{2}^{\prime} D+{C}_{3}^{\prime} } \right)\left( {B_{1}^{1,0} } \right)\left( {B_{1}^{0,1} }\right), \end{array} $$

where the values of \(A_{i} ,B_{i} ,C_{i} ,{A}_{i}^{\prime } ,{B}^{\prime },{C}_{i}^{\prime } \left ( {i=1,2,3} \right )\), are given in Appendix 1.

Now, adopting the procedure of Bhatnagar and Hallan [4], the second order components \(B_{2}^{1,0} \) and \(B_{2}^{0,1}\) are as follows

$$\begin{array}{@{}rcl@{}} B_{2}^{1,0} &=& {r}_{1}^{\prime} I_{1} +{r}_{2}^{\prime} I_{2} +{r}_{3}^{\prime} I_{3} cos\varphi_{1} +{r}_{4}^{\prime} I_{2} cos\varphi_{2} +{r}_{5}^{\prime} I_{1} sin2\varphi_{1} +{r}_{6}^{\prime} I_{2} sin2\varphi_{2} \\ &&+\left\{{{r}_{7}^{\prime} cos\left( {\varphi_{1} +\varphi_{2} } \right)+{r}_{8}^{\prime} cos\left( {\varphi_{1} -\varphi_{2} } \right)+{r}_{7}^{\prime} cos\left( {\varphi_{1} +\varphi_{2} } \right)+{r}_{9}^{\prime} sin\left( {\varphi_{1} +\varphi_{2}}\right)}\right.\\&&\left.{+{r}_{10}^{\prime} sin\left( {\varphi_{1} -\varphi_{2} } \right)} \right\}\sqrt {I_{1} I_{2} },\\ B_{2}^{0,1} &=&{s}_{1}^{\prime} I_{1} +{s}_{2}^{\prime} I_{2} +{s}_{3}^{\prime} I_{3} cos\varphi_{1} +{s}_{4}^{\prime} I_{2} cos\varphi_{2} +{s}_{5}^{\prime} I_{1} sin2\varphi_{1} +{s}_{6}^{\prime} I_{2} sin2\varphi_{2} \\ &&+\left\{ {{s}_{7}^{\prime} cos\left( {\varphi_{1} +\varphi_{2} } \right)+{s}_{8}^{\prime} cos\left( {\varphi_{1} -\varphi_{2} } \right)+{s}_{7}^{\prime} cos\left( {\varphi_{1} +\varphi_{2} } \right)+{s}_{9}^{\prime} sin\left( {\varphi_{1} +\varphi_{2}} \right)}\right.\\&&\left.{+{s}_{10}^{\prime} sin\left( {\varphi_{1} -\varphi_{2} } \right)} \right\}\sqrt {I_{1} I_{2} } , \end{array} $$

where

$${r}_{i}^{\prime} =r_{i} \left( {1+\alpha_{i} \,k_{1} +{\alpha }_{i}^{\prime} \,k_{2} } \right), {s}_{i}^{\prime} =s_{i} \left( {1+\beta_{i} k_{1} +{\beta }_{i}^{\prime} k_{2} } \right),\,\,\,\,\left( {i=1,2,3\,,....10} \right), $$

and all the values of r i ’s, s i ’s, α i ’s and β i ’s, are given in Appendix 2.

Second Order Coefficients in the Frequencies

To make use of Moser’s modified version of Arnold’s theorem [2], it is necessary to reduce the Hamiltonian to its normalized form. So, we performed the first and second order normalization. We have found the second order coefficients in the frequencies. For this we have obtained the partial differential equations which are satisfied by the third order homogeneous components of the fourth order part of Hamiltonian H 4 and second order polynomials in the frequencies.

Following the iterative procedure of Bhatnagar and Hallan [4], we note that the third order components \(B_{3}^{0,1}\) and \(B_{3}^{1,0}\), can be obtained by solving the partial differential equations

$$\begin{array}{@{}rcl@{}} {\Delta}_{1} {\Delta}_{2} B_{3}^{1,0} ={\Phi}_{3} -2f_{2} P-2g_{2} Q,\,\,\,\,\,\,\,{\Delta}_{1} {\Delta}_{2} B_{3}^{0,1} =\psi_{3} -2f_{2} U-2g_{2} V, \end{array} $$
(10)

where

$$\begin{array}{@{}rcl@{}} {\Phi}_{3} &=&X_{3} \left( {D^{2}-\left( {\frac{9}{4}+\frac{33k_{1} }{8}+\left( {\frac{3}{2}+\frac{3\sqrt 3 }{8}+\frac{33\gamma }{16}} \right)k_{2} } \right)} \right) \\ &&+Y_{3} \left( {2n} \right.D+\left( {\frac{3\sqrt 3 \gamma }{4}-\frac{15\sqrt 3 k_{1} }{8}} +\frac{11k_{1} \gamma }{8}+ {\frac{3\sqrt 3 k_{2} }{16}} \right), \\ P&=&\frac{\partial }{\partial \varphi_{1} }\left( {\omega_{1}^{\prime} \frac{\partial B_{1}^{1,0} }{\partial \varphi_{1} }-nB_{1}^{0,1} } \right)\left( {\omega_{1}^{\prime2} \frac{\partial^{2}}{\partial \varphi_{1} }-\left( {\frac{9}{4}+\frac{33k_{1} }{8}+\left( {\frac{3}{2}+\frac{3\sqrt 3 }{8}+\frac{33\gamma }{16}} \right)k_{2} } \right)} \right) \\ &&+\left( {\omega_{1}^{\prime} \frac{\partial B_{1}^{0,1} }{\partial \varphi_{1} }-nB_{1}^{1,0} } \right)\left( {2n\omega_{1}^{\prime} \frac{\partial }{\partial \varphi_{1} }+\left( {\frac{3\sqrt 3 \gamma }{4}-\frac{15\sqrt 3 k_{1} }{8}} +\frac{11k_{1} \gamma }{8}+{\frac{3\sqrt 3 k_{2} }{16}}\right)} \right), \\ Q&=&\frac{\partial }{\partial \varphi_{2} }\left( {\omega_{2}^{\prime} \frac{\partial B_{1}^{1,0} }{\partial \varphi_{2} }-nB_{1}^{0,1} } \right)\left( {\omega_{2}^{\prime2} \frac{\partial^{2}}{\partial \varphi_{2} }-\left( {\frac{9}{4}+\frac{33k_{1} }{8}+\left( {\frac{3}{2}+\frac{3\sqrt 3 }{8}+\frac{33\gamma }{16}} \right)k_{2} } \right)} \right) \\ &&+\left( {\omega_{2}^{\prime} \frac{\partial B_{1}^{0,1} }{\partial \varphi_{2} }-nB_{1}^{1,0} } \right)\left( {2n\omega_{2}^{\prime} \frac{\partial }{\partial \varphi_{2} }+\left( {\frac{3\sqrt 3 \gamma }{4}-\frac{15\sqrt 3 k_{1} }{8}} +\frac{11k_{1} \gamma }{8}+ {\frac{3\sqrt 3 k_{2} }{16}} \right)} \right), \\ {\Psi}_{3} &=&Y_{3} \left( {D^{2}-\left( {\frac{3}{4}+\frac{3k_{1} }{8}+\left( {\frac{3}{2}+\frac{\sqrt 3 }{8}-\frac{21\gamma }{16}} \right)k_{2} } \right)} \right) \\ &&-X_{3} \left( {2nD} \right.-\left( {\frac{3\sqrt 3 \gamma }{4}-\frac{15\sqrt 3 k_{1} }{8}} \right.+\frac{11k_{1} \gamma }{8}+\left. {\frac{3\sqrt 3 k_{2} }{16}} \right), \\ U&=&\frac{\partial }{\partial \varphi_{1} }\left( {\omega_{1}^{\prime} \frac{\partial B_{1}^{0,1} }{\partial \varphi_{1} }-nB_{1}^{1,0} } \right)\left( {D^{2}-\left( {\frac{3}{4}+\frac{3k_{1} }{8}+\left( {\frac{3}{2}+\frac{\sqrt 3 }{8}-\frac{21\gamma }{16}} \right)k_{2} } \right)} \right) \\ &&+\left( {\omega_{1}^{\prime} \frac{\partial^{2}B_{1}^{1,0} }{\partial \varphi_{1} }-nB_{1}^{0,1} } \right)\left( {2nD} \right.-\left( {\frac{3\sqrt 3 \gamma }{4}-\frac{15\sqrt 3 k_{1} }{8}} +\frac{11k_{1} \gamma }{8}+ {\frac{3\sqrt 3 k_{2} }{16}} \right), \\ V&=&\frac{\partial }{\partial \varphi_{2} }\left( {\omega_{2}^{\prime} \frac{\partial B_{1}^{1,0} }{\partial \varphi_{1} }-nB_{1}^{0,1} } \right)\left( {2nD} \right.-\left( {\frac{3\sqrt 3 \gamma }{4}-\frac{15\sqrt 3 k_{1} }{8}} +\frac{11k_{1} \gamma }{8}+{\frac{3\sqrt 3 k_{2} }{16}} \right) \\ &&-\left( {\omega_{2}^{\prime} \frac{\partial^{2}B_{2}^{1,0} }{\partial \varphi_{2} }-nB_{1}^{1,0} } \right)\left( {D^{2}-\left( {\frac{3}{4}+\frac{3k_{1} }{8}+\left( {\frac{3}{2}+\frac{\sqrt 3 }{8}-\frac{21\gamma }{16}} \right)k_{2} } \right)} \right), \end{array} $$

and X 3 and Y 3 are homogeneous components of order 3 obtained on substituting \(x=B_{1}^{1,0} +B_{2}^{1,0} ,\,y=B_{1}^{0,1} +B_{2}^{0,1} ,\) in \(\frac {\partial {\Gamma }_{3} }{\partial x}+\frac {\partial {\Gamma }_{4}}{\partial x}\) and \(\frac {\partial {\Gamma }_{3} }{\partial y}+\frac {\partial {\Gamma }_{4} }{\partial y}\).

The components \(B_{3}^{0,1}\) and \(B_{3}^{1,0}\) are not required to be found out. We find the coefficients of c o s φ 1, s i n φ 1,c o s φ 2 and s i n φ 2 in the right-hand sides of Eq. 10, they are the critical terms. We eliminate these terms by properly choosing the coefficients in the polynomials

$$f_{2} ={f}_{2,0}^{\prime} I_{1} +{f}_{0,2}^{\prime} I_{2} ,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,g_{2} ={g}_{2,0}^{\prime} I_{1} +{g}_{0,2}^{\prime} I_{2} , $$

where

$$\begin{array}{@{}rcl@{}}{f}_{2,0}^{\prime} &=& \frac{\text{coefficient}\,\text{of}\,cos\varphi_{1} \,\text{in}\,\,{\Phi}_{3}}{2\left( {\text{coefficient}\,\text{of}\,cos\varphi_{1} \,\text{in}\,P} \right)}, \\ &=&f_{2,0} \left[ {1+\left( {\xi -\eta } \right)k_{1} +\left( {{\xi }^{\prime}-{\eta}^{\prime}} \right)k_{2} } \right]=A(\text{say}), \end{array} $$
(11)
$$\begin{array}{@{}rcl@{}}{f}_{0,2}^{\prime} &=&{g}_{2,0}^{\prime} =\frac{\text{coefficient}\,\text{of}\,cos\varphi_{2} \,\text{in}\,{\Phi}_{3} }{2\left( {\text{coefficient}\,\text{of}\,cos\varphi_{2} \,\text{in}\,Q} \right)}, \\ &=&f_{0,2} \left[ {1+\left( {\zeta -\eta } \right)k_{1} +\left( {{\zeta }^{\prime}-{\eta }^{\prime}} \right)k_{2} } \right]=B(\text{say}), \end{array} $$
(12)
$$\begin{array}{@{}rcl@{}} {g}_{0,2}^{\prime} &=& \frac{\text{coefficient}\,\text{of}\,cos\varphi_{2} \,\text{in}\,\psi_{3} }{2\left( {\text{coefficient}\,\text{of}\,cos\varphi_{2} \,\text{in}\,Q} \right)}, \\ &=&g_{0,2} \left[ {1+\left( {\sigma -\rho } \right)k_{1} +\left( {{\sigma }^{\prime}-{\rho }^{\prime}} \right)k_{2} } \right]=C(\text{say}), \end{array} $$
(13)
$$\begin{array}{@{}rcl@{}} f_{2,0} &=&\frac{{\omega_{2}^{2}} \left( {81-696{\omega_{1}^{2}} +124\omega_{1}^{2} } \right)}{72\left( {1-2{\omega_{2}^{2}} } \right)^{2}\left( {1-5{\omega_{1}^{2}} } \right)},\\ g_{0,2} &=&\frac{{\omega_{1}^{2}} \left( {81-696{\omega_{2}^{2}} +124\omega_{2}^{2} } \right)}{72\left( {1-2{\omega_{2}^{2}} } \right)^{2}\left( {1-5{\omega_{2}^{2}} } \right)},\\ f_{0,2} &=&-\frac{\omega_{1} \omega_{2} \left( {43+64{\omega_{1}^{2}} \omega_{2}^{2} } \right)}{6\left( {1-2{\omega_{1}^{2}} } \right)\left( {1-2\omega_{2}^{2} } \right)\left( {1-5{\omega_{1}^{2}} } \right)\left( {1-5\omega_{2}^{2} } \right)}, \end{array} $$
$$\begin{array}{@{}rcl@{}} \xi &=&\frac{1}{32}\left[ {32} \right.\left( {\sum\limits_{i=0}^{6} {\left( {L_{i+1} L_{i+1,1} \,T_{i+15} } \right)\sum\limits_{i=1}^{7} {\left( {L_{i+1} T_{i+14} } \right)} } } \right.+\left\{ {-\left( {15+\sqrt 3 p} \right)6\sqrt 3 T_{22} } \right. \\ &&+12\left( {85-22p\gamma } \right)T_{23} -\left( {15} \right.\left. {+2p} \right)18\sqrt 3 T_{24} \text{+3}\left( {\text{465+82}p} \right)T_{25} \\ &&-45\left( {35} \right.\left. {+6p\gamma } \right)\sqrt 3 T_{26} +\left( {255+6p} \right)T_{27}-6\left( {2\sqrt 3 T_{22,1} +44T_{23,1} } \right.\\\\ &&+6\sqrt 3 T_{24,1} -44T_{25,1} -45\sqrt 3 \gamma T_{26,1} \left. {-\left. {T_{27,1} } \right\} \omega_{1} } \right], \end{array} $$
$$\begin{array}{@{}rcl@{}} \zeta &=&\frac{1}{32}\left[ {32} \left( {\sum\limits_{i=0}^{6} {\left( {L_{i+1} L_{i+1,1} T_{i+28} } \right)\sum\limits_{i=0}^{6} {\left( {L_{i+1} T_{i+28,1} } \right)} } } \right.+\left\{ {-\left( {15+2p} \right)6\sqrt 3 T_{35} } \right. \right.\\ &&+12\left( {85-22p\gamma } \right)T_{36} -\left( {15} \right.\left. {+2p} \right)18\sqrt 3 T_{37} \text{+3}\left( {{465+82}p} \right)T_{38} \\ &&-45\left( {35} \right.\left. {+6p\gamma } \right)\sqrt 3 T_{39} +\left( {255+6p} \right)T_{40}-6\left( {2\sqrt 3 T_{35,1} +44T_{36,1} } \right.\\ &&\left.+6\sqrt 3 T_{37,1} -44T_{38,1} -45\sqrt 3 \gamma T_{39,1} - {T_{40,1} } \right\} \left.{\vphantom{\sum\limits_{i=0}^{6}}}\omega_{1} \right], \end{array} $$
$$\begin{array}{@{}rcl@{}} \eta &=&\frac{1}{8k^{2}{l_{1}^{2}} }\left[ {3\left( {11+60p} \right.-45\gamma +6\alpha_{13} -48\alpha_{21} +18\left. {\alpha_{23} } \right)} \right.+4\left( {33} \right.+78p \\ &&+20\alpha_{13} +16\alpha_{21} -\left. {8\alpha_{23} } \right){\omega_{1}^{2}} +32\left( {5p+\alpha_{13} +\alpha_{23} } \right)\left. {{\omega_{1}^{4}} } \right], \end{array} $$
$$\begin{array}{@{}rcl@{}} {\eta }^{\prime}&=&\frac{1}{8k^{4}{l_{1}^{4}} z_{1} \omega_{2} }\left[ {-27\gamma {l_{1}^{2}} z_{1} } \right.+8\gamma \omega_{2} \left( {24+37{\omega_{1}^{2}} -236} \right.{\omega_{1}^{4}} -27\omega_{2} +123{\omega_{1}^{2}} \omega_{2}\\ &&+\left. {60{\omega_{1}^{4}} \omega_{2} } \right)+k^{2}{l_{1}^{2}} z_{1} \omega_{2} \left\{ {36\left( {27+4{\omega_{1}^{2}} } \right)} \right.-4{p}^{\prime}\left( {45+78{\omega_{1}^{2}} +40{\omega_{1}^{4}} } \right) \\ &&-16{\alpha }_{21}^{\prime} \left( {-9+4{\omega_{1}^{2}} } \right)-2{\alpha }_{23}^{\prime} \left( {27-16{\omega_{1}^{2}} +16{\omega_{1}^{4}} } \right)-2{\alpha }_{13}^{\prime} \left. {\left. {\left( {1+4{\omega_{1}^{2}} } \right)} \right\} } \right], \end{array} $$

and the values of \({\xi }^{\prime },\,{\zeta }^{\prime }\) are obtained from the values of ξ, ζ respectively by replacing L i ’s, T i ’s by corresponding \({L}_{i}^{\prime }\)’s, \({T}^{\prime }_{i}\)’s and p, q by \({p}^{\prime }\), \({q}^{\prime }\) respectively keeping ω 1,ω 2, l 1,l 2,z 1,z 2,z 3,z 4, unchanged. Also the values of \(\sigma ,\rho ,{\sigma }^{\prime },{\rho }^{\prime }\) can be obtained from \(\xi ,\eta ,{\xi }^{\prime },{\eta }^{\prime }\) respectively by replacing ω 1 by −ω 2, l 1 by l 2,k 2 by −k 2,z 1 by z 2 whenever they occur and the values of \(L_{i} \left ( {i=1,2,\ldots ,14} \right )\!, L_{i,1} \left ( {i=1,2,\ldots ,7} \right )\!,T_{i} \left ( {i=15,16,\ldots ,40} \right )\) and \(T_{i,1} \left ( {i=15,16,\ldots ,40} \right )\) are given in Appendix 3.

Stability

Now we verify that this condition is satisfied. The condition is \(K_{1} \omega _{1}^{\prime } +K_{2} \omega _{2}^{\prime } \ne 0\), for all pairs \(\left ( {K_{1} ,K_{2} } \right )\) of rational integers such that \(\left | {K_{1} } \right |+\left | {K_{2} } \right |\le 4\).

We calculate

$$ K_{1} \omega_{1}^{\prime} +K_{2} \omega_{2}^{\prime} =0,\Leftrightarrow \frac{\omega_{1}^{\prime}}{\omega_{2}^{\prime} }=-\frac{K_{2} }{K_{1}} $$
(11)

Here as

$$0<\omega_{2} <\frac{1}{\sqrt 2 }<\omega_{1} <1, $$

and so

$$0<\omega_{2}^{\prime} <\frac{1}{\sqrt 2 }<\omega_{1}^{\prime} <1\left( {\left| {K_{1} } \right|<<1,\left| {K_{2} } \right|<<1} \right). $$

So we have

$$\begin{array}{@{}rcl@{}} \frac{\omega_{1}^{\prime} }{\omega_{2}^{\prime} }>1, \end{array} $$

For Eq. 11 to be true, K 1 and K 2 are of opposite signs and \(-\frac {K_{2} }{K_{1} }>1\).

Therefore, K 1,K 2, can have the following values,

$$K_{1} =1,\,\,K_{2} =-2\,\,\,\,;K_{1} =-1,\,\,K_{2} =2\,. $$
$$K_{1} =1,\,\,K_{2} =-3\,\,\,\,;K_{1} =-1,\,\,K_{2} =3\,. $$

Case (i)

K 1=1, K 2=−2 ;K 1=−1, K 2=2 .

Equation 11 gives

$$ \frac{\omega_{1}^{\prime} }{\omega_{2}^{\prime} }=2\,\,\,\text{i.e.\thinspace }\omega_{1}^{\prime} -2\omega_{2}^{\prime} =0. $$
(12)

Solving the Eqs. 6, 7 and 12 and putting γ=1−2μ,we get

$${\mu }^{\prime}_{1} =\left( {0.024293897...} \right)+\left( {0.0201245...} \right)k_{1} +\left( {-0.00602906...} \right)k_{2} , $$

Case (ii)

K 1=1, K 2=−3 ;K 1=−1, K 2=3 .

Equation 11 gives

$$ \frac{\omega_{1}^{\prime} }{\omega_{2}^{\prime} }=3\,\,\,\text{i.e.\thinspace }\omega_{1}^{\prime} -3\omega_{2}^{\prime} =0. $$
(13)

Solving the Eqs. 6, 7 and 13 and putting γ=1−2μ,we get

$${\mu }^{\prime}_{2} =\left( {0.013516016...} \right)+\left( {0.503812...} \right)k_{1} +\left( {-0.228826...} \right)k_{2} , $$

Hence for the values \({\mu }^{\prime }_{1} \) and \({\mu }^{\prime }_{2} \) of the mass ratio condition (i) of Moser’s theorem is not satisfied.

The normalized Hamiltonian up to fourth order is written as

$$H=\omega_{1} I_{1} +\omega_{2} I_{2} +\frac{1}{2}\left( {A{I_{1}^{2}} +2BI_{1} I_{2} +C{I_{2}^{2}} } \right)+.... $$

After substituting the values of A, B and C from Eq. ??, b and c, the determinant D occurring in condition (a) of Moser’s theorem becomes

$$D=-\frac{9\left( {36-541{\omega_{1}^{2}} {\omega_{2}^{2}} +644{\omega_{1}^{4}} {\omega_{2}^{4}} } \right)+Rk_{1} +{R}^{\prime}k_{2} }{72\left( {1-4{\omega_{1}^{2}} {\omega_{2}^{2}} } \right)\left( {4-25{\omega_{1}^{2}} {\omega_{2}^{2}} } \right)}, $$

where

$$\begin{array}{@{}rcl@{}} R&=&-{\omega_{2}^{4}} \left( {81-696{\omega_{1}^{2}} +124{\omega_{1}^{4}} } \right)\left( {\xi -\eta +2q} \right)z_{2} -24u^{2}\left( {43+64u^{2}}\right) \\ &&\times\left( {\xi -\eta +p+q} \right)-{\omega_{1}^{4}} \left( {81-696\omega_{2}^{2} +124{\omega_{2}^{4}} } \right)\left( {\sigma -\rho +2p} \right)z_{1} , \end{array} $$

And \({R}^{\prime }\) can be obtained from R by replacing ξ,η,ζ,σ,ρ,p,q by \({\xi }^{\prime },{\eta }^{\prime },{\zeta }^{\prime },{\sigma }^{\prime },{\rho }^{\prime },{p}^{\prime },{q}^{\prime }\) respectively.

Moser’s second condition is violated for the unperturbed problem (i.e. for k 1 = k 2=0) when μ 3=.0109137....

When k 1,k 2≠0 , we take \({\mu }^{\prime }_{3} =\mu _{3} +Xk_{1} +{X}^{\prime }k_{2} \) such that D=0. It is seen that the condition (b) of Moser’s theorem is satisfied i.e. D ≠ 0 if in the interval 0<μ<μ c , the mass ratio does not take the value \({\mu }^{\prime }_{3} =\mu _{3} +6069.97...k_{1} +733.824...k_{2}\).

Therefore, the libration point L 4 is stable in the range of linear stability μ<μ c , except at three values of mass parameters μ 1,μ 2 and μ 3.

Conclusion

We have investigated the non linear stability of the triangular libration point L 4 in the restricted three body problem where the Moser theorem is satisfied, by taking the smaller primary with mass m 2 an oblate spheroid with three layers having different densities ρ i and axes \(\left ( {a_{i} ,c_{i} } \right )\left ({i=1,2,3} \right )\). Till now, researchers and scientists have investigated this problem by taking different shapes of the primaries but all of them have taken homogeneous primaries. We found the triangular point L 4 is stable in the range of linear stability 0<μ<μ c , where, \(\left ( {\mu _{c} =\mu _{0} +\frac {k_{1} }{18}\left ({41-\frac {11\sqrt {69} }{3}} \right ),\mu _{0} =0.03852....} \right )\), except for three mass ratios

$$\begin{array}{@{}rcl@{}} {\mu }^{\prime}_{1} &=&\left( {0.024293897...} \right)+\left( {0.0201245...} \right)k_{1} +\left( {-0.00602906...} \right)k_{2} ,\\ {\mu }^{\prime}_{2} &=&\left( {0.013516016...} \right)+\left( {0.503812...} \right)k_{1} +\left( {-0.228826...} \right)k_{2} ,\\ {\mu }^{\prime}_{3} &=&\left( {0.0109366...} \right)+\left( {6069.97...} \right)k_{1} +\left( {733.824...} \right)k_{2}, \end{array} $$

at which Moser’s theorem does not apply. k 1 and k 2 are the parameters which depend upon the oblateness and densities ρ 1,ρ 2 and ρ 3. If we take k 1 = k 2=0, then the values of \({\mu }^{\prime }_{1} ,{\mu }^{\prime }_{2}\) and \({\mu }^{\prime }_{3}\), agree well with the classical case which was discussed by Deprit and Deprit [6]. We observe that μ c is independent of k 2 and its range increases if k 1 increases. We also observe that the value of \(\mu _{1}^{\prime }\) and \(\mu _{2}^{\prime }\) increases as k 1 increases and decreases as k 2 increases, while value of \(\mu _{3}^{\prime }\) always increases.