1 Introduction

At the heart of the modern study of waves is their behaviour and stability. The last century has heralded many studies and successes into these avenues, but there is much left to be understood. Particularly, the stability of monochromatic wavetrains in hydrodynamics generated large interest after the experiments of Benjamin and Feir demonstrated that such a state was unstable in experiments [4, 5]; some analytical insight has since been gained using various mathematical techniques [44]. Such instabilities have been speculated to lead to the formation of rogue waves [34, 43], or the decrease in the frequency of the monochromatic wave [7, 28, 41]. However, even when such waves are stable to an instability like this, they can undergo different transitions to generate structures such as dark solitary waves [24, 30, 32], whose mechanism for formation remains unclear. Therefore, even such a heavily studied problem has a wealth of dynamics we have yet to understand.

The role of mean flow coupling on both the dynamics and stability of water waves has long been emphasised. There have been many contributions towards accurately providing a description between this interplay. Early work by Davey and Stewartson [13] provided a system which coupled a nonlinear Schrödinger equation to an irrotational fluid flow. This work has been furthered by several authors since. Of particular note is the work of Dysthe et al. [15, 46], who have derived several higher order models which capture the effects of this mean flow-wave coupling with a large degree of success. For example, these have helped to more accurately investigate the stability of uniform wavetrains [45,46,47], rogue wave formation [27, 42], and frequency downshifting [16]. It is because of this rich variety of behaviours that this paper will concern itself with the Dysthe equation with only its leading order dispersive term, as written in Ref. [46]. Thus, the governing equations are given by

$$\begin{aligned} \begin{aligned}&iA_t+A_{xx}-|A|^2A+\alpha \left. \Phi _x\right| _{z=0}A+i\beta |A|^2A_x = 0,\\&\left. \Phi _z\right| _{z=0} = \alpha (|A|^2)_x,\\&\nabla ^2\Phi = 0\quad \text{ in } \quad z\in (-h,0),\\&\left. \Phi _z\right| _{z = -h} = 0, \end{aligned} \end{aligned}$$
(1)

for complex-valued wave envelope A(xt), velocity potential \(\Phi (x,z,t)\) of the flow, and h the depth of the fluid. The constants \(\alpha \) and \(\beta \) characterise the magnitude of mean-flow and higher order self-steepening effects, respectively. In the work of Trulsen and Dysthe [46], the values of these parameters are taken to be \(\alpha = 4\) and \(\beta = 8,\) but we shall leave these free to see their explicit role in the emergence and evolution of defects.

To investigate systems which admit wavetrains and discuss their stability properties, one may utilise a phase dynamical approach. The origins of such studies can be traced back to Whitham [48, 50], with several years of subsequent work building upon these ideas. The premise is to assume that one has a wavetrain solution of the form \(\hat{u}(kx+\omega t) \equiv \hat{u}(\theta ;k,\omega )\) for phase \(\theta \), wavenumber k and frequency \(\omega \). Then, the key idea is to assume that these wave variables are not fixed, but instead slowly vary in time. Under this assumption, either by an averaging principle or direct asymptotic analysis, one generates the system of equations:

$$\begin{aligned} A(k,\omega )_T+B(k,\omega )_X = 0, \quad k_T = \omega _X, \end{aligned}$$
(2)

for the now slowly varying wavenumber k(XT), frequency \(\omega (X,T)\), and slow variables \(X = {\varepsilon }x,\,T = {\varepsilon }t\) for \({\varepsilon }\ll 1\). The functions A and B turn out to be the wave action and wave action flux, respectively, for the original system, averaged over one period of the original wavetrain \(\hat{u}\). This set of equations then govern how the wavenumber and frequency evolve, which lead to deformations in the wavetrain from which they are derived. There have been several extensions to the Whitham methodology, such as the extension to problems with multiple phases [1, 35, 50] and to the more general setting of relative equilibria [10, 49, 50]. It is these extensions that will allow us to explore the phase dynamics of the Dysthe system (1), as one not only modulates the emergent wavetrain but also the mean flow, which itself forms a relative equilibrium.

The Whitham equation (2) has the additional usage that by investigating the linearised stability problem for a fixed wavenumber and frequency, one can infer stability properties of the original wave. This is diagnosed by the properties of the eigenvalues of such a linearisation, which are denoted as the characteristics of the system. For real characteristics, the system is hyperbolic and the associated wavetrain is stable. Alternatively, in cases, where the characteristics become complex the Whitham system is elliptic and the underlying wave is unstable, with perturbations to it growing exponentially. One of the most famous examples of this usage arises when studying the Whitham equations which emerge from the nonlinear Schrödinger equation. One can show in this case that the Benjamin–Feir instability criterion emerges at the point when the characteristics become complex [29]. This result was the moment when “the penny dropped” for Whitham regarding the connection between characteristics and stability [33].

A key development in the setting of this paper is how the dispersionless Whitham system morphs to incorporate dispersive effects at singularities, which was achieved via the work of Bridges et al. [9, 39]. The main idea is to modify Whitham’s original ansatz with a different set of scalings at points where the Whitham modulation equations develop singularities. In such cases, one instead adopts an ansatz at a new solution of the form:

$$\begin{aligned} u = \hat{u}\big (\theta +{\varepsilon }\phi (X,T);k+{\varepsilon }^2\phi _X(X,T),\omega +{\varepsilon }^4\phi _T(X,T)\big )+{\varepsilon }^3 W(\theta ,X,T;{\varepsilon }), \end{aligned}$$

with instead \(X = {\varepsilon }x,\, T = {\varepsilon }^3 t\). Doing so allows one to resolve the singularity which leads to the emergence of dispersion and the appearance of the well-known nonlinear wave equation, the Korteweg–de Vries (KdV), which governs the evolution of the slowly varying wavenumber. Most strikingly, the KdV which emerges does so in universal form, in the same sense as the Whitham equations. This is to say that its coefficients relate to abstract properties of the original Lagrangian that generates the problem. Extensions to this methodology, for example, when resolving additional singularities which may arise, demonstrate how other nonlinear equations such as the two-way Boussinesq and modified KdV (mKdV) emerge. In particular, a recent advancement has shown that when the phase modulation is done in a moving frame, with a speed equal to one of the characteristics emerging from the Whitham system, nonlinear phase dynamics emerge automatically with dispersion [37]. Furthermore, when the characteristics pass through certain values or coalesce, more complex phase dynamics may occur [11]. It is for this reason that this paper will adopt the modulation approach in the moving frame, to link both the properties of the conservation laws and characteristics of the Whitham equations to the resulting dynamics.

In summary, this paper aims to utilise a modification of the Whitham modulation theory to derive nonlinear dispersive equations to investigate the phase dynamics of the Dysthe system (1). This will require a modulation of a two-parameter relative equilibrium, namely, the plane wave solution coupled to a uniform current, which adopts a theory adept at treating multiple phases as well as a moving frame. This is achieved by combining existing modulation approaches, and will ultimately be used to show how the KdV equation governing the phase dynamics emerges from the Dysthe equation. Moreover, a secondary aim will be to provide criteria for when one expects the behaviour of defects in the wave to be qualitatively different, occurring precisely when one of these reductions fails to be adequate and should be replaced by another phase dynamical equation. In the setting of this paper, we consider only one example of this, which will focus on how the Gardner, and consequently the mKdV equation, arise whenever the nonlinearity of the KdV equation is sufficiently small and vanishing. This leads to increased nonlinear effects within the evolution of phase defects, which will be highlighted in the body of the paper.

From this phase dynamical description, we aim to provide a possible qualitative picture for the emergence of various coherent structures, such as dark and bright solitary waves. In the study of this paper, we will utilise the various nonlinear dispersive reductions and illustrate how these structures can be interpreted as a deformation under the perturbation of the wave variables by the solitary wave solutions of these systems. This technique has been utilized in other works [3, 8, 23, 26]; however, it has yet to be considered in the context of a wave–mean flow system like the Dysthe equation. Thus, another contribution of this work will be to determine how this coupling manifests within this approach as well as to identify its effect on the qualitative predictions. Depending on the phase dynamical description, which itself depends on the properties of the moving frame, we will illustrate that various structures are predicted to form from this viewpoint. Moreover, we will demonstrate that each regime implies that the original wave is affected to differing degrees. Thus, the characteristic speeds from the Whitham equations may in fact be used as a diagnostic regarding the degree in which the phase dynamical description distorts the original wavetrain.

The outline of this paper is as follows. In Sect. 2, we review how the existing phase modulation approach may be applied to the Dysthe system (1) to obtain a KdV equation which governs how the phases of the waves evolve over space and time. This is followed in Sect. 2.1 by an illustration of how such a KdV may be utilised to predict the bifurcation behaviour of the original plane wave solution, with an emphasis on processes that lead to the formation of dark or bright solitary waves. Subsequently, in Sect. 3, further phase reductions are discussed, noting when such equations arise and their effect on the plane wave solution. Concluding remarks are presented in Sect. 4.

2 From Dysthe to KdV

There are a multitude of techniques in which to investigate the phase dynamics of equations like (1). Primarily, the approach usually taken is to use a standard multiple scales analysis and solve at each order of the small parameter [8, 26]. A variant of this, and the closest reduction procedure to the modulation considered here, is to undertake this approach after transforming the system using the Madelung transform [3, 17, 23]. This approach is utilised in Appendix B to derive the KdV and show that it agrees with that obtained via the modulation methodology. Prior to this paper, it does not appear that such a study using the Madelung approach has been undertaken in the context of wave–mean flow systems such as the Dysthe equation, and so the reduction using this method is also another novelty of this paper. Within this paper the focus will be on the modulation approach, with the details of how the KdV equation may be obtained using this procedure appearing in Appendix A. This method allows one to draw a connection between not only the conservation laws the Dysthe system possesses, but also highlights how the hyperbolicity of the system plays a role in the evolution of the phase defects.

To apply the modulation approach detailed in Appendix A to the Dysthe system (1), we must first note that it is generated by a Lagrangian, namely

$$\begin{aligned} \mathcal {L}= & {} \iint \Bigg [\frac{i}{2}(AA^*_t-A^*A_t)+|A_x|^2+\frac{i \beta }{4}|A|^2(AA^*_x-A^*A_x)\nonumber \\&-\alpha |A|^2\left. \Phi _x\right| _{z=0}-\frac{1}{2}\int _{-h}^0\Phi _x^2+\Phi _z^2\, {\text {d}}z\Bigg ]\ {\text {d}}x\, {\text {d}}t. \end{aligned}$$
(3)

To discuss how defects in the phase evolve for uniform plane waves with a current in this equation, we consider the relative equilibria associated with the product of an affine and toral symmetry. The first of these arises from the fact that one may add arbitrary constants to the velocity potential \(\Phi \) and leave the Lagrangian invariant, and the second is the one associated with the \(S^1\) symmetry of the amplitude A. This corresponds to the uniform wavetrain state in A and the uniform flow solution of \(\Phi \), which explicitly is given by

$$\begin{aligned} A = A_0 e^{i \theta }, \quad \Phi = u_0x+\gamma t, \quad \text{ with } \quad |A_0|^2 =\Psi = \frac{\alpha u_0-k^2-\omega }{1+\beta k}. \end{aligned}$$
(4)

The amplitude \(|A_0|^2\) must be positive and nonsingular, meaning that \(k \ne -\frac{1}{\beta }\).

For the analysis of Appendix A we require the relevant conservation law vectors. These arise from averaging the Lagrangian (3) over a period of the wave, as determined in (19) and can be readily determined as

$$\begin{aligned} \mathbf{A} = \begin{pmatrix} \widehat{\mathcal {L}}_\omega \\ \widehat{\mathcal {L}}_{\gamma } \end{pmatrix} = \begin{pmatrix} \Psi \\ 0 \end{pmatrix}, \quad \mathbf{B} = \begin{pmatrix} \widehat{\mathcal {L}}_k\\ \widehat{\mathcal {L}}_{u_0} \end{pmatrix} = \begin{pmatrix} 2k\Psi +\frac{\beta }{2}\Psi ^2\\ -hu_0-\alpha \Psi \end{pmatrix}. \end{aligned}$$

The first row of these vectors are associated with the conservation of wave action of the periodic wave and is akin to the conservation of mass for the wave itself. The second emerges as the conservation of mass of the flow beneath the wave, albeit with a negative sign. This is due to the way one obtains it via the averaging principle as detailed in (19). With these quantities in hand, we are in a position to compute both the criterion for emergence and the coefficients of the KdV associated with this wavetrain. The relevant criticality condition for this to emerge in a moving frame, with characteristic speed c, from the solution (4) is given by (21) in Appendix A. For the Dysthe equation, this requires that the determinant of the Jacobian

$$\begin{aligned} \mathbf{E}(c) = \frac{1}{1+\beta k} \begin{pmatrix} 2(1+\beta k)\Psi -(2k+c+\beta \Psi )^2&{}\alpha (2k+\beta \Psi +c)\\ \alpha (2k+c+\beta \Psi )&{}-h(1+\beta k)-\alpha ^2 \end{pmatrix}, \end{aligned}$$

vanishes. This is the case whenever the characteristic speed satisfies

$$\begin{aligned} \Delta (c)=hc^2+2h(\beta \Psi +2k)c+\beta ^2h\Psi -2\big (h(1-\beta k)+\alpha ^2\big )\Psi +4hk^2 = 0, \end{aligned}$$
(5)

and so the characteristic speeds are given by

$$\begin{aligned} c = -2k-\beta \Psi \pm \sqrt{2\Psi \bigg (\beta k+1+\frac{\alpha ^2}{h}\bigg )}. \end{aligned}$$
(6)

The necessity for hyperbolicity requires that \(k>-\frac{1}{\beta }-\frac{\alpha ^2}{\beta h}\). In such cases, where (5) is satisfied and c is real, we can define the relevant eigenvector of \(\mathbf{E}(c)\) required for the theory, \({\varvec{\zeta }}\), as

$$\begin{aligned} {\varvec{\zeta }}= \begin{pmatrix} \zeta _1\\ \zeta _2 \end{pmatrix} = \frac{1}{1+\beta k} \begin{pmatrix} h(1+\beta k)+\alpha ^2\\ \alpha (2k+c+\beta \Psi ) \end{pmatrix}. \end{aligned}$$

We may now compute the coefficients of the resulting KdV equation, which is done by following the methodology in Ref. [12] and Appendix A. The first to be computed is the coefficient of the time derivative term, giving that

$$\begin{aligned} {\varvec{\zeta }}^T\mathbf{E}'(c){\varvec{\zeta }}= -\frac{2\big ((\beta k+1)h+\alpha ^2\big )(\beta \Psi +c+2k)h}{(bk+1)^2}. \end{aligned}$$

The next calculation is for the coefficient of the nonlinear term, and gives that

$$\begin{aligned} \begin{aligned} {\varvec{\zeta }}^T\mathbf{H}({\varvec{\zeta }},{\varvec{\zeta }})=\frac{3\big ((\beta k+1)h+\alpha ^2\big )(\beta \Psi +c+2k)h\big (h(\beta ^2\Psi +\beta c-2)-2\alpha ^2 \big )}{(bk+1)^3}. \end{aligned} \end{aligned}$$

The final coefficient required is that of the dispersive term. Typically within the modulation approach, this is acquired by undertaking a Jordan chain analysis; however, it is in fact more readily obtained from the dispersion relation computed about the solution (4), as is typically done in hydrodynamic settings [2, 19, 20]. This is since the dispersion relation for the KdV must match up with the dispersion relation from the original problem (1) in the long wave limit. This can be found from a simple linear analysis of (1) about the plane wave solution (4), and leads to the dispersion relation:

$$\begin{aligned} \sigma (\kappa ) = -(2k+\beta \Psi +c)\kappa \pm \kappa \sqrt{2 \Psi (1+\beta k)+\kappa ^2+\frac{\alpha ^2\Psi }{h}\kappa \coth (\kappa h)}. \end{aligned}$$

Thus, the relevant coefficient of dispersion arises from the cubic term in \(\kappa \) in the long wave expansion of the above relation, multiplied by the coefficient of the KdV’s time derivative term. This leads to the required coefficient of dispersion:

$$\begin{aligned} \frac{1}{6}\left. \sigma _{\kappa \kappa \kappa }\right| _{\kappa =0}{\varvec{\zeta }}^T\mathbf{E}'(c){\varvec{\zeta }}=-\frac{\big ((\beta k+1)h+\alpha ^2\big )h(3+2\alpha ^2h\Psi )}{3(bk+1)^2}. \end{aligned}$$

This can be verified by following the approach outlined in Appendix B, up to a scaling factor. Combining the above results and simplifying gives the relevant KdV as

$$\begin{aligned} (2k+\beta \Psi +c)\bigg (U_T- \frac{3}{2(1+\beta k)}\big (h(\beta ^2\Psi+ & {} \beta c-2)-2\alpha ^2 \big )UU_X\bigg )\nonumber \\+ & {} \frac{1}{6}(3+2\alpha ^2h\Psi )U_{XXX} = 0. \end{aligned}$$
(7)

One can notice that the mean flow effects, characterised by \(\alpha \), appear explicitly within the dispersive coefficient, whereas both the characteristic speed c and the self-steepening effects determined by \(\beta \) appear within the coefficients of the time and nonlinear terms. Thus, these effects have a non-negligible effect on the phase dynamics which emerge from the plane wave solution. The aim now will be to investigate how these effects influence the phase dynamics and lead to the emergence of coherent, localised structures from the original plane wave.

2.1 The Evolution of Phase Defects

With the relevant KdV (7) in hand, we may now discuss how it may be used to determine the bifurcating behaviour of the periodic wave solution (4). There are a large family of solutions admitted by the KdV equation, such as cnoidal waves [14] and multipulse solutions [22, 31], which could be utilised to explore the phase dynamics as predicted by (7). However, for simplicity, we will focus on how the KdV equation in this context can describe the formation of dark and bright solitary waves from the uniform plane wave solution. The strategy to illustrate this is to use the solitary wave solution to the KdV equation (7):

$$\begin{aligned} U(\xi ) = a\,\mathrm{sech}^2\bigg (\frac{\xi }{W}\bigg ), \end{aligned}$$

with \(\xi = X-VT\), and where the amplitude a and width W are given by

$$\begin{aligned} \begin{aligned} a =&-\frac{2V(1+\beta k)}{\big (h(\beta ^2\Psi +\beta c-2)-2\alpha ^2 \big )}, \qquad W = 2 \ \sqrt{\frac{3+2\alpha ^2h\Psi }{6(2k+\beta \Psi +c) V}}. \end{aligned} \end{aligned}$$
(8)

This is then used to reconstruct the solution to the Dysthe equation according to the ansatz

$$\begin{aligned} \begin{aligned} A&= A_0(k+{\varepsilon }^2\zeta _1U,u_0+{\varepsilon }^2\zeta _2U,\omega +c{\varepsilon }^2\zeta _1U-V{\varepsilon }^4\zeta _1U){\text {e}}^{i(\theta +{\varepsilon }\zeta _1\phi )},\\&\text{ where } \quad \phi = \int U d\xi , =aW \tanh \bigg (\frac{\xi }{W}\bigg ), \end{aligned} \end{aligned}$$
(9)

which is utilized to derive the KdV (7). We then determine how the original solution (4) is impacted. This, to leading order, has the corresponding effect on the envelope:

$$\begin{aligned} \begin{aligned} |A|^2&= \Psi +{\varepsilon }^2 \ \frac{2(c+2k+\beta \Psi )V}{\beta ^2 \Psi +\beta c-2-\frac{2 \alpha ^2}{h}}\mathrm{sech}^2 \bigg (\frac{\xi }{W}\bigg )+\mathcal {O}({\varepsilon }^4). \end{aligned} \end{aligned}$$
(10)

It is clear at this stage that both the mean flow and steepening influence the amplitude and width of the solitary wave and will lead to a wide range of possible localised structures. However, we note that the current \(u_0\) does not explicitly enter any of these expressions, and so for a fixed amplitude \(\Psi \) the resulting dynamics is independent of the mean flow velocity, although the strength of these effects does influence the observed phenomenon.

We can already determine a great deal analytically. In particular, we can see that as the strength of the mean flow \(\alpha \) is increased the width of the solution U, and thus the width of the observed disturbance to the wave, increases. This is the case for either choice of the characteristic speeds, as made clear by (6) which highlights that

$$\begin{aligned} c +2k+\beta \Psi = \pm \sqrt{2\Psi \bigg (\beta k+1+\frac{\alpha ^2}{h}\bigg )}, \end{aligned}$$

and so this only affects the choice of sign for V. Thus, the mean flow is the dominant effect in the width, as the steepening only asymptotically decreases the width as \(\beta ^{-\frac{1}{2}}\), whereas the mean flow causes this to grow as \(\alpha \).

The effect of the phase dynamics on the amplitude of the plane wave is less clear and involves an interplay between steepening and depth. However, by studying (10), one is able to see that overall the effects of steepening and the mean flow decrease the resulting amplitude of the structure which forms, and the asymptotic decay is algebraic of the order \(\alpha ^{-1},\,\beta ^{-1}\). As such, one expects the largest amplitudes for these disturbances to occur for \(\alpha = 0\) and for values of \(\beta \) approaching the singularity in the amplitude of the solution U as given in (8). However, in this proximity the KdV equation (7) begins to become an invalid model of the phase dynamics and other descriptions become operational, as will be described within the next section, so the discussion here will not apply to such choices of \(\beta \) that are sufficiently close to these points.

In summary, this approximate picture provides us with a qualitative description for how these localised structures appear and what form these are expected to take. In the presence of strong mean flow interactions between the wave and the current, the distortion to the envelope is expected to have a longer range. When the wave is subjected to increased self-steepening effects, the magnitude of the disturbance is muted so long as a degeneracy in the derived KdV equation is not approached in parameter space. Therefore, the KdV dynamics derived in this setting suggest that the original plane wave is in fact stable to the majority of defects which can form, and their effect on the wavetrain is very limited.

We combine these findings to illustrate what effects the mean flow and self-steepening have upon the bifurcation behaviour of the original plane waves. This is done by setting \(\alpha = \beta = 0\) and in essence becomes a study of the phase dynamics of the Nonlinear Schrödinger equation [8, 26]. This is then compared to cases, where \(\alpha ,\,\beta \) are nonzero, but with the same choice of wavenumber and frequency so that the effects of the mean flow and steepening can be assessed. The mean flow \(u_0\) is chosen so that the amplitudes in all cases are the same. An example of such a comparison is depicted in Fig. 1.

Fig. 1
figure 1

Comparison between the resulting bifurcation of the plane wave solution (4) under the phase dynamics for the fast (left column) and slow (right column) characteristic speeds for \({\varepsilon }= 0.2\), \(k = 1,\,\omega = -\frac{3}{2},\,h = 4,\, V = 1\) and a, b\(\alpha = \beta = 0\), c, d\(\alpha = \beta = 1\), e, f\(\alpha = -4,\,\beta = -\frac{3}{2}\) and g, h\(\alpha = 4,\,\beta = 8\). The dashed red lines indicate the original plane wave, whereas the solid blue lines denote the solution reconstructed via (9), with the dotted blue lines corresponding to the envelope for this solution (Color figure online)

We now discuss the results of this investigation. One can observe that the mean flow and steepening effect can lead to qualitative differences in the dynamics, changing the polarity of the resulting structure from dark to bright, and viceversa. This is to be expected, since the steepening effects enter the nonlinear coefficient and allow for it to change sign. Moreover, we can see that these effects can also lead to the previously discussed suppression of the solitary structures, suggesting that the original wave persists even in the presence of the nonlinear phase dynamics and hints at increased stability in these regimes. Additionally, it becomes clear that the presence of mean flow and self-steepening within the system can lead to increased nonlinear effects in the predicted bifurcating behaviour, where these disturbances are enhanced. Overall, the presence of both of these effects leads to a greater range of behaviours for the localised structures.

3 Further Singularities and Higher Order Phase Dynamics

Although the KdV itself provides some insight into the evolution of defects and the formation of coherent structures from the uniform wavetrain, there are parameter values for which the KdV stops being operational due to vanishing terms. At such points, there is the potential for even more interesting nonlinear phenomena to emerge in a way analogous to a secondary instability via higher order phase equations. We discuss one such case of this below, leading to the modified KdV equation, and by extension the Gardner equation, with a discussion of their effects on the evolution of the original plane wave solution.

3.1 Modified KdV

The singularity of interest is the case, where only the quadratic nonlinearity vanishes, which is whenever the characteristic speed takes the value

$$\begin{aligned} c = \frac{1}{\beta }\bigg (2-\beta ^2\Psi +\frac{2\alpha ^2}{h}\bigg ). \end{aligned}$$
(11)

For this to occur, the amplitude must satisfy

$$\begin{aligned} \Psi = \frac{2}{\beta ^2}\bigg (\beta k+1+\frac{\alpha ^2}{h}\bigg ). \end{aligned}$$
(12)

Using this value of the amplitude in (11) gives the much simpler condition that \(c = -2k\). From this, we note that without the moving frame one would have to impose \(k=0\) to obtain the mKdV from the Dysthe equation, limiting its applicability as well as its ability to emerge from plane wave solutions.

In such cases, where (11) holds, a rescaling of the modulation approach must occur to reintroduce a nonlinear term into the analysis. The subsequent modulation reduction procedure then follows very similarly to [35], with the modifications for the moving frame appearing in other works [11, 12] and Appendix A. Alternatively, one considers a rescaling of the ansatz used in Appendix B as is done to obtain the mKdV in similar contexts (for example, [25, 40]). By following either procedure, one obtains the modified KdV equation

$$\begin{aligned} \beta \Psi U_T+\frac{3\beta ^2\Psi ^2}{2(1+\beta k)^2}U^2U_X+\frac{3+2\alpha ^2h\Psi }{6}U_{XXX} = 0. \end{aligned}$$
(13)

Of note is that the effects of steepening now have an increased role in the dynamics through the coefficient of the cubic nonlinearity.

To investigate dark and bright solitary waves in this regime, the relevant solitary wave solution family to the above is given by

$$\begin{aligned} U = \pm \sqrt{\frac{V(1+\beta k)^2}{\beta \Psi }} \ \mathrm{sech}\bigg (\sqrt{\frac{6 \beta V \Psi }{3+2\alpha ^2\Psi h}} \ \xi \bigg ). \end{aligned}$$
(14)

The requirement that this solution be real imposes that \(\beta V>0\). Note that solutions of both polarities are permissible from the phase dynamics now, and so one expects both dark and bright solitary waves to emerge in this scenario at the same parameter values. The solution is then reconstructed according to

$$\begin{aligned} A = A_0(k+{\varepsilon }\zeta _1U,u_0+{\varepsilon }\zeta _2U,\omega +c{\varepsilon }\zeta _1U-V{\varepsilon }^3\zeta _1U){\text {e}}^{i(\theta +\zeta _1\phi )}, \end{aligned}$$
(15)

where the relevant scalings in \({\varepsilon }\) are chosen so that the corresponding modulation approach leads to the mKdV [36]. To leading order, the effects on the envelope of the wave can be deduced via a Taylor expansion, revealing that

$$\begin{aligned} |A|^2 = \Psi \pm {\varepsilon }\sqrt{\frac{2h(\beta h k+h+\alpha ^2)V}{\beta }}\mathrm{sech}\bigg (\sqrt{\frac{6 \beta V \Psi }{3+2\alpha ^2\Psi h}} \ \xi \bigg )+\mathcal {O}({\varepsilon }^2). \end{aligned}$$

From this, we can make some inferences regarding how the mean flow and steepening alter the phase dynamical observations. First, the fact that the solution admitted is sech to a unitary power reveals that the dark or bright structures are already expected to be wider than those observed under the KdV dynamics. We can also see that \(\beta \) now has a decreased role in the amplitude of the disturbance, and instead the mean flow and depth become the main contributing factors to its size. Moreover, the expansion itself leads to a correction an order lower than that of the KdV case, and so the effects of this are expected to be more readily observed. The width of the localised structure is very similar to that discussed in the KdV equation, which is expected as the linear dispersive properties of the phase dynamics have not changed, and thus has the same asymptotic decay properties.

Fig. 2
figure 2

Examples of how the solution (14) distorts the original plane wave for a, b\(\alpha = -0.5,\,\beta = 3,\,k=1,\,\omega = -4.32,\,h = 3,\,u_0 = 1,\,c=1,\, {\varepsilon }= 0.06\) and c, d\(\alpha = 0.7,\,\beta = -1.2,\,k=1.2,\,\omega = -0.69,\,h = 0.5,\,u_0 = 1,\,c=-1.6,\,{\varepsilon }= 0.2\) for both positive (left) and negative (right) polarities (Color figure online)

These observations are confirmed by the examples of how these solutions modify the original plane wave, depicted in Fig. 2. Overall, the reconstructions in this case tend to have a more pronounced but typically broader effect on the plane wave than the KdV equation, causing more apparent versions of the emergent dark and bright solitary waves. This is expected, as the mKdV soliton is both narrower and larger in amplitude than the one which is admitted by the KdV. Moreover, the scalings within the ansatz to obtain the mKdV are larger than the KdV case. Thus, the resulting bifurcation that the phase dynamics predicts should be of higher magnitude, as well as being sharper in regimes, where the mKdV (13) is operational.

3.2 Gardner Equation

Fig. 3
figure 3

Examples of how the solution (16) affects the plane wave solution in the regimes, where a\(B = 9.19\) and b\(B=1.63\) (Color figure online)

The fact that solutions of both polarities are admissible in the mKdV presents the issue of how this may be selected in practice. One way to ensure a certain polarity is selected is to break this symmetry, which may be achieved by instead choosing a wavespeed close to \(-2k\). We achieve this by setting

$$\begin{aligned} c = -2k+{\varepsilon }\gamma , \end{aligned}$$

for \(\gamma = \mathcal {O}(1)\). The modulation in this case instead leads to the Gardner equation:

$$\begin{aligned} \beta \Psi U_T- \frac{3\beta \Psi \gamma }{2(1+\beta k)} UU_X+\frac{3\beta ^2\Psi ^2}{2(1+\beta k)^2}U^2U_X+\frac{3+2\alpha ^2h\Psi }{6}U_{XXX} = 0. \end{aligned}$$

The solitary waves admitted by this system differ from those arising from (13) and have more interesting forms due to the presence of \(\gamma \). These solutions are given by [18]

$$\begin{aligned} U =-\frac{4 V(1+\beta k)}{\gamma \beta \Psi \bigg [1+B \cosh \bigg (\sqrt{\frac{6\beta \Psi V}{3+2\alpha ^2h\Psi }} \ \xi \bigg )\bigg ]}, \quad B^2 = 1+\frac{4 V \beta \Psi }{\gamma ^2}, \end{aligned}$$
(16)

with again the requirement that \(\beta V>0\). From this, it is clear that \(\gamma \) now controls the polarity of the solution, rather than both signs being admissible. There are two limits of interest for this solution. The first is as \(B \rightarrow \infty \), namely, as the quadratic coefficient becomes small, where the classical \(\mathrm{sech}\) solitary wave of the mKdV (14) is recovered. The other is as \(B\rightarrow 1\), for which the solitary wave becomes broader. One also notes that the tabletop solitary wave solution of the Gardner equation is no longer possible, since \(B^2\ge 1\). Examples of both regimes and their effect on the plane wave solution, as reconstructed according to (15), are depicted in Fig. 3. As one might expect, the regime, where B is large is reminiscent of the mKdV case, with more pronounced bifurcating structures than for the KdV. For lower B (corresponding to larger \(\gamma \)) one observes slightly wider packets of lower amplitude, closer to those of the KdV dynamics. Thus, the Gardner equation sheds light on the dynamics in the intermediate regime between the KdV and mKdV equations, and thus the way in which the solitary waves become tighter and increase in amplitude, as one may expect.

4 Concluding Remarks

This paper has demonstrated that there is a great wealth of phase dynamics emerging from the Dysthe equation, primarily owing to both mean flow and self-steepening effects. The simplest of these is the KdV, which predicts how plane waves can bifurcate to modulated, or dark/bright structures. Additionally, when certain criterion are met, richer dynamics can occur by the increment of nonlinearity within the phase equations.

There are other phase reductions that may be admitted from the study of the Dysthe equation. For example when the characteristics of the Whitham equation coalesce, both the time derivative and nonlinear terms in (7) vanish, and instead a modified version of the two-way Boussinesq equation arises:

$$\begin{aligned} U_{TT}&+\bigg (\frac{\beta ^2\Psi ^2}{2(1+\beta k)^2}U^3+\frac{1}{6}(3-2\alpha ^2 \Psi h)U_{XX}\bigg )_{XX}\nonumber \\&-\frac{\beta \Psi }{1+\beta k}(2UU_T+U_X\partial _X^{-1}U_T)_X = 0. \end{aligned}$$
(17)

The dynamics of this equation are quite complicated, and it does not appear to admit solitary waves for the coefficients which emerge. Thus, a more delicate analysis of this equation would be necessary and would involve the study of the periodic solutions it supports. Moreover, when the Dysthe system (1) possesses higher order dispersive effects, we expect the phase dynamics to also have increased dispersive properties within certain parameter regimes. Such scenarios are expected to lead to the fifth order KdV equation emerging from the modulation, and perhaps lead to more interesting bifurcating structures.

Although the phase dynamics provides a qualitative picture as to the formation of solitary structures within the original plane wave, the next step would be to compare the results obtained here to those from direct numerical simulation. This would quantify the ability of the nonlinear dynamics discussed here to capture the true bifurcating behaviour, and possibly help to lend these analytic techniques more credibility when discussing these scenarios.

There are other systems which exhibit a mean flow coupling within a set of nonlinear equations with a free surface. Examples include a version of (1) with higher order terms in A or systems such as Benney–Roskes [6] or Hasimoto–Ono [21] equations. These can also be explored using a phase dynamical approach to investigate how the flow beneath a wave influences how the uniform wave is modulated in the presence of defects. The novel features of the latter systems is that the wave action vectors these problems admit are nondegenerate, unlike the Dysthe equation, and so richer time dynamics are possible.