Covariant M5-brane action with self-dual 3-form

In this work, we extend a theory which describes a linearly self-dual exact 3-form field in six dimensional spacetime within the formalism of arXiv:1511.08220 and arXiv:1903.12196 to an action which fully describes an M5-brane in the eleven-dimensional supergravity background. The action we obtain has all the required symmetries including kappa-symmetry. Although the derivation of the action and the proof the symmetries do not require one to express the action explicitly in terms of the original field contents, we also give an initial attempt to give this expression.


Introduction
In order to construct a low-energy effective action for a single M5-brane, it is important to base this on an action which describes a chiral 2-form field, i.e. a 2-form field with self-dual 3-form field strength, living in a six-dimensional spacetime. This is because a chiral 2-form field is one of the field content of M5-brane. Furthermore, it is already not a straightforward matter to construct an action for a chiral 2-form field.
In the literature, there have been two main types chiral 2-form theories in six dimensions and the corresponding complete actions of single M5-brane in elevendimensional supergravity background. In one type, the actions are diffeomorhism invariant but not manifestly. Chiral 2-form theories of this type are given in [1], [2], [3], [4]. Complete M5-brane actions of this type are constructed in [5], [6]. In the other type, the actions are manifestly diffeomorphism invariant, but auxiliary fields should be introduced. One the of symmetries ensure that the auxiliary fields have no dynamics. The approach to construct theories of this type is called the PST formalism. Actions for chiral 2-form of this type are given in [7], [8], [9], [10]. Complete M5-brane actions of this type are constructed in [11], [12], [13], [14]. Actions for theories within either of these types contains terms which describe dynamics of the chiral 2-form field. Self-duality condition for its 3-form field strength is obtained at the level of equations of motion. This is made possible due to a gauge symmetry usually called PST1 symmetry.
It could be possible to extend other kinds of chiral 2-form actions to obtain complete M5-brane actions. We find that one of the most promising recent constructions of a chiral 2-form theory is given in [15], [16]. See also [17] and [18] for further developments. The theory which is relevant to us describes a 2-form field P and a 3-form field Q in a six-dimensional curved spacetime. The 2-form field P has the wrong sign of the kinetic term. The 3-form field Q is self-dual with respect to flat metric. A certain combination of the 3-form field Q with the curved metric gives rise to a 3-form field which is self-dual with respect to the curved metric. At the level of equations of motion, the latter 3-form field is closed. If the spacetime has a trivial topology, then this 3-form field is exact and hence, since it is also self-dual, it is equivalent to a chiral 2-form field with self-dual 3-form field strength. Also at the level of equations of motion the 2-form field P is decoupled. Further investigation using Hamiltonian analysis also shows that the 2-form field P is decoupled. We will call the idea of this construction as Sen formalism. The analysis of the linear self-dual 3-form theory in Sen formalism has been quite throughout [15], [16], [17], [18] which provides good background for us to seek to extend the theory.
The main goal of this paper is to extend the action in Sen formalism by nonlinearising the self-duality condition and extending the resulting action to fully describe an M5-brane in the eleven-dimensional supergravity background.
Our strategy is to start by constructing an action for a nonlinear self-dual 3form theory by using diffeomorphism invariance to fix the form of the action. Then a 3-form source field can naturally be included in a similar way as in the linear selfdual case. One of gauge symmetries of this resulting theory has the gauge parameter which is subject to some condition. By including an appropriate 6-form source field, it is possible to remove this restriction. In fact, the 3-form and 6-form source fields can be realised as pull-backs of C 3 and C 6 fields from the eleven-dimensional supergravity background to the M5-brane worldvolume. Based on these results, a complete action of an M5-brane in the eleven-dimensional supergravity background can be constructed. The action has all the required properties especially kappasymmetry.
The rest of this paper is organised as follows. In section 2, we review the action of [15], [16]. In particular, we focus on the prove of diffeomorphism invariance. In section 3, we proceed to generalise to nonlinear self-dual case. In section 4, we further generalise the action to a complete M5-brane action in eleven-dimensional supergravity background. In the analysis explained so far, we do not need to know the exact expression of the action as a function of original field content. As long as all the properties are known, it is sufficient to prove the symmetry. Nevertheless, to prepare for other purpose, it might be useful to be able to explicitly express the action in terms of the original field content. In linear self-dual case, This task has been done [15], [16], [18]. In section 5, we will provide an initial attempt in nonlinear self-dual case. Finally, in section 6, we give conclusions and discuss possible directions for future works.

A covariant action for a linear self-dual field
In this section, we review an action given in [15], [16] which describes a self-dual 3-form field in six dimensions. In particular, we review basic properties which includes diffeomorphism invariance.
Consider a six-dimensional spacetime with coordinates x µ , µ = 0, 1, · · · , 5 and metric g µν . We define a differential p−form as Exterior derivatives and interior products are defined to act from the right. In Sen formalism, there are two kinds of Hodge star operator. The first kind is the standard Hodge star operator, which we label as * . It acts as * dx µ 1 ∧ · · · ∧dx µp = (−1) p+1 2) where ǫ µ 1 ···µ 6 is a Levi-Civita symbol defined such that ǫ 012···5 = 1. The other kind which we label as * ′ acts as 3) where η µν is the flat metric. In other words, * ′ is defined with respect to the flat metric.
Let us now review an action given in [15], [16], which is where P is a 2-form field, and Q is a * ′ -self-dual 3-form field: * ′ Q = Q. Let the variation of L I with respect to Q be of the form The condition (2.6) can be thought of as an extra condition on R. It is shown by construction [15], [16], [18] that this condition along with * ′ -anti-self-duality of R can be used to completely determine R.
Varying the action (2.4) gives Therefore, equations of motion are Eliminating dP from eq.(2.9) and substituting into eq.(2.8) gives In spacetime with trivial topology, H is exact. It can be seen from eq.(2.6) and eq.(2.10) that, in spacetime with trivial topology, combining the metric g µν and the field Q in a certain way gives rise to a field H which is * -self-dual off-shell and is exact on-shell. The field P, which is unphysical due to the wrong sign of its kinetic term, is decoupled. In particular, it has been shown [16] that a combination is decoupled and is invariant under a diffeomorphism and gauge transformations. Furthermore, after extending the theory to an abelian (2, 0) theory, H (s) is a singlet under supersymmetry transformation [17], [18]. From the review above it can be seen that the field content P and Q do not have standard properties. In fact, as we will also review later, these fields do not transform as standard differential forms under diffeomorphism transformations. Hence, they are called pseudoforms.
It is interesting to compare Sen formalism with PST formalism in the construction of an action for a 2-form field with self-dual 3-form field strength. In short, Sen formalism gives a 3-form field which is self-dual off-shell and is closed on-shell. In a spacetime with trivial topology, this field is exact. On the other hand, the construction of PST formalism describes a 2-form field. So its 3-form field strength is exact off-shell. The self-duality of 3-form field strength is obtained on-shell. The analysis can be done in the case of spacetime with trivial topology as well as spacetime with nontrivial topology [19], [20].
More detailed discussions are as follows. We have seen that in Sen formalism, the action (2.4) of the pseudoforms P and Q = * ′ Q is constructed such that the form of L I allows a certain combination of the metric and Q to give rise to H which is * -self-dual. At the level of the equation of motion, the field H is closed and the field P is decoupled. If the spacetime has trivial topology, the field H is then exact on-shell. In the PST formalism, the 2-form field B is introduced whose field strength will be self-dual at the level of equations of motion. In order to achieve this while making the action manifestly diffeomorphism invariant, one or more auxiliary scalar fields are introduced. In this formalism, apart from the standard tensor gauge symmetry, there are two important gauge symmetries called PST1 and PST2 symmetries. The PST1 symmetry is used to ensure that the equation of motion of B reduces to self-duality condition for dB.
In Sen formalism, the presence of a non-standard Hodge star operator * ′ suggests that the action (2.4) is not manifestly diffeomorphism invariant. In [15], [16], [18], the expression of L I is given at the start. Then the non-standard diffeomorphism transformation of P and Q which leaves the action invariant is determined. For us, in order to gain some insights as a preparation for nonlinearisation, let us proceed in the opposite direction. We will simply impose diffeomorphism transformations on P and Q as given in [15], [16], [18], then proceed to determine the form of L I .
Let P and Q transform under infinitesimal diffeomorphism x µ → x µ + ξ µ as Diffeomorphism variation on the action (2.4) is then (2.14) Using Leibniz rule, we obtain where tot. are total derivative terms. On the other hand, by using self-duality condition, eq.(2.14) becomes Therefore, adding eq.(2.15) and eq.(2.16) together gives So the action is diffeomorphism invariant if the condition Direct calculation using δ ξ H = −L ξ H gives (2.20) On the other hand, using H = Q − R on eq.(2. 19) gives Comparing eq.(2.20) and eq.(2.21) gives Applying wedge product with H to the above equation and using eq.(2.18) gives δL I δg µν δ ξ g µν = Q∧ δR δg µν δ ξ g µν , (2.23) which can be integrated to give By taking derivative with respect to Q and using H∧δ Q H = 0, we obtain ζ(Q) = 0, and hence L I = Q∧R, (2.25) which is as required.

A covariant action for a nonlinear self-dual field
In this section, we present a construction of a covariant action for a nonlinear selfdual field. Apart from the fields P, Q, and g, let us also introduce a 3-form source. The action should take the form where P and Q are as defined in the previous section, and J is a 3-form source. Let us write the variation of L I with respect to Q as So R is * ′ -anti-self-dual. The equation of motion is modified from eq.(2.10) to where H J = Q−R+J. Furthermore, as in the linear self-dual case, the combination H (s) is also decoupled. We require H J to be nonlinear * -self-dual, that is * H J = V(H J , g).
Again, eq.(3.4) can be thought of as a condition given to fix the form of L I . It is not an on-shell condition.
For the purpose of this paper, whose main goal is to construct an M5-brane action, it is sufficient to consider V of the form which can be obtained by considering (H J ) µνρ V µνρ and using eq.(3.4) and eq.(3.5).
As to be given in details later, eq.(3.4)-(3.5) with a specific choice of F 1 and F 2 describes nonlinear * -self-duality condition for M5-brane.

Useful identities
Before proceeding to derive the action, it would be useful to first present some identities from nonlinear self-duality conditions. It can be seen that eq.(3.4) with V given by eq.(3.5) satisfies and and Note that these conditions can alternatively be written as Alternatively, the condition (3.9) can be expressed using differential forms as (3.14) Note also that there is an identity Note that these identities are derived based on the form of V as given in eq.(3.5). This form alone, however, is not enough to ensure that eq.(3.4) is consistent. There should be extra restrictions to ensure this. By counting the number of components, eq.(3.4) is a system of 20 equations. In order for it to truly be a nonlinear self-duality condition, the system should contain 10 independent equations. This requirement gives consistency conditions which would fix the form of V. Note, however, that we do not yet need to impose these extra conditions to derive the action. So we postpone the discussion of the consistency conditions to subsection 5.2.

Derivation of the action using diffeomorphism
Let us now discuss the derivation of the action. We are going to do this by demanding that the action is diffeomorphism invariant. The consideration is in parallel to the linear self-dual case. Let P, Q transform as The action then transforms as Imposing eq.(3.8)-(3.9) on eq.(3.18), we obtain So after substituting into eq.(3.17), we see that there are two types of expressions. The first type contains terms proportional to δ ξ g µν . The second type contains terms proportional to δ ξ J µνρ . Let us suppose that these two types should vanish separately. So as part of this, we should set It is possible to simplify the first term on RHS. For this, let us use the condition (3.14) and the identity This gives By using eq.(3.14) and eq.(3.19), we obtain and hence (3.24) Integrating gives Taking derivative with respect to J gives On the other hand, demanding terms proportional to δ ξ J µνρ in eq.(3.17) to vanish gives Comparing the two expressions of δ J L I gives and hence ζ = Q∧J +ζ(Q). (3.29) Taking derivative with respect to Q gives and hence δ Qζ = 0. (3.32) So from the consideration of diffeomorphism invariance, we obtain the action where U is a function of H J (Q, J, g) and g. The form of U (H J (Q, J, g), g) and R(Q, J, g) should be determined from * -self-duality condition eq.(3.4) provided that it is consistent with * ′ -self-duality of Q and that the conditions (3.8) and (3.9) are satisfied. Note that in the linear * -self-dual case, the conditions (3.8) and (3.9) are satisfied with U = 0. In this case, the action (3.33) also reduces to the form given in [15], [16], [18]. It can be seen that just like the linear self-dual case, the field P is decoupled at the level of equations of motion. In the nonlinear self-dual case, the decoupling of P can also be seen from Hamiltonian analysis [16], [18]. In fact, the result of the analysis of [16] holds for actions of the form (3.1). Since our nonlinear action falls in this case, the result of Hamiltonian analysis of [16] also valid for our case.
Although the Hamiltonian analysis of [16], [18] treat only P and Q in the canonical formulation, it can be expected that even if every field is treated in the canonical formulation the field P still decouples from the theory as long as it does not couple to the metric and J. This is indeed the case for the action (3.33).
In order to gain further insights on the action (3.33), it would be useful to consider the case of flat spacetime 1 . In this case, the two types of metrics g µν and η µν should coincide and hence one would expect that the fields P and Q should transform in the standard way under Lorentz transformation. It turns out that this standard Lorentz transformation is not obtainable from the nonstandard diffeomorphism transformation (2.12). In fact, the theory (3.33) in flat spacetime has two types of Lorentz symmetries: the standard one and the nonstandard one which is obtained from diffeomorphism transformation (2.12) with ξ µ = ω [µν] x ν . In order for the action to possess the nonstandard Lorentz symmetry, the conditions (3.8) and (3.9) should still be satisfied. When promoted to curved spacetime, it turns out that only the nonstandard Lorentz symmetry survives and is indeed promoted to the nonstandard diffeomorphism transformation (2.12).

Symmetries
Apart from being invariant under diffeomorphism by construction, the action (3.33) also enjoys the gauge symmetries as in the case of linear * -self-dual case [16], [18]. The invariance under P → P + dλ is straightforward. To check another gauge symmetry, let us note that subject to the variation of the action is given by where Λ is subject to Λ∧dJ = 0. (3.37) 1 We thank Dmitri Sorokin for the correspondence on this issue In fact, as shall be seen in section 4, a 6-form source field with appropriate transformation can be included into the action so that the resulting action is gauge invariant without the need to impose extra condition. By considering eq.(3.35), it can be seen that energy-momentum tensor can easily be derived and can be shown to be conserved. The analysis is similar to the case of linear self-duality given in [18]. In the case of nonlinear self-duality, energy-momentum tensor is given by (3.38) The conservation of the energy-momentum tensor can also be shown by using the diffeomorphism invariance of the action. With the help of eq.(3.35), we obtain Imposing equation of motion dH J = dJ, we obtain So when there is no source, the energy-momentum tensor is conserved.
In the case of linear * -self-duality, in which U = 0, the energy-momentum tensor reduces to the one given in [18]. The conservation then also follows.

A covariant M5-brane action in Green-Schwarz formalism
In this section, we are going to present an extension of the action (3.33) with an appropriate U to describe a complete M5-brane action embedded in the elevendimensional supergravity background.

Nonlinear * -self-duality condition from M5-brane
Let us start by showing that the bosonic chiral 2-form field which is one of the fields on M5-brane worldvolume can be described by the action (3.33) with appropriate U, to be given below. Our strategy is to first obtain the form of eq.(3.4) in flat sixdimensional spacetime and without source. Since this equation should be Lorentz covariant, the subsequent introduction of the curved metric and source should be straightforward. One way to obtain nonlinear self-duality condition for the 3-form field strength of the chiral 2-form is by considering [21], which derives the condition by using superembedding formalism. In this description, there are two kinds of 3-form fields. One of them is h, which is linear self-dual but is not a field strength of any 2-form field. Another is H, which is a field strength of some 2-form field, but H itself is not linearly self-dual. Instead, it satisfies a nonlinear self-dual condition which can be obtained by solving In order to obtain the self-duality condition in the form (3.4), the field h µνρ should be eliminated. One way to do this is to first make a 5 + 1 split of spacetime indices [21] to obtain V µν5 in terms of H µν5 . One may then try to propose an ansatz for the covariant form and demand that it is consistent with the form obtained from a 5 + 1 split. Another way to do this is by covariantly eliminating the field h µνρ [22], [23].
Having been able to put eq.(4.1) in the form of eq.(3.4), one can then include the curved metric and the source. This finally gives where By using eq.(3.7) and eq.(3.10), we obtain  Alternatively, we may reexpress U M5 (H J , g) to the form introduced in [23]. In this case, eq.(3.15) is given by (4.7) In fact, we also need to ensure that the analysis so far in this section is consistent after expressing H in terms of Q. We will discuss in subsection 5.2 that this is indeed the case.

Extension to a complete M5-brane action
The action (3.33) with U = U M5 can be promoted to a full M5-brane action in Green-Schwarz formalism. For this, let us first state a convention which we use to describe the target superspace.
The eleven-dimensional target superspace is parametrised by Z M = (X M , θ), where X M are eleven bosonic coordinates and θ are 32 real fermionic coordinates. The supervielbeins are E A (Z) = dZ M E M A (Z) and E α (Z) = dZ M E M α (Z) where A = 0, 1, · · · , 10 is a bosonic tangent space indices and α = 1, 2, · · · , 32 are fermionic tangent space indices. The kappa-symmetry would require that the superspace torsion has to satisfy a constraint The target superspace also contains a 3-form gauge superfield and its C 6 (Z) dual. Kappa-symmetry would also require the field strengths of C 3 (Z) and C 6 (Z) to satisfy the following constraints where · · · are terms which are not relevant for kappa-symmetry calculation.
The embedding of an M5-brane worldvolume into the target superspace is described by the embedding functions Z M (x). The induced metric on the M5-brane worldvolume is given by The M5-brane worldvolume still contains the fields P (x) and Q(x). As for source field J(x), it is realised as the pull-back of C 3 to the M5-brane worldvolume. So let us denote J as C 3 and redefine the notation of H J from the previous section to F. That is As a first step to promote the action (3.33), we need to include a Wess-Zumino term. In fact, it could be expected that only C 6 term can be added into the action. An inclusion of F ∧C 3 which is the other part of the Wess-Zumino term would modify the equation of motion of Q which is already in the desirable form. Correspondingly, the need to add only C 6 term can be seen from a gauge symmetry requirement. If there is no condition imposed on the gauge parameter, the action (3.33) would not be invariant under the transformation (3.36). The transformation on the extra term C 6 cancels out the non-zero contribution from the variation of the action (3.33). The resulting action is then given by (4.14) Note that the last two terms are combined into Wess-Zumino term. Note also that the field F appearing in the action is actually a function of Q, C 3 , g. The action (4.14) is invariant under the gauge transformation which is promoted from eq.(3.36), that is where the gauge transformations on C 3 and C 6 are such that their field strengths are gauge invariant. The action (4.14) should be invariant under the local fermionic kappa-symmetry with parameters κ α (x) which transforms the pull-back of the embedding functions such that The induced metric then transforms as Up to this point, when considering δ κ S M5 , there are still unknown quantities which are δ κ P, δ κ Q, and di κ C 3 . It turns out that by imposing the expression di κ C 3 no longer appears in δ κ S M5 . We then see that the action transforms as The quantity (1 +Γ)/2 is a projector of rank 16 with [22], [23] 1 It would be useful to compare the action (4.14) with M5-brane action from other formalisms. In particular, let us compare on-shell actions. The action (4.14) is given on-shell as An action which is closely related to our construction is the one presented in [23]. It involves a 2-form field B whose equation of motion is required to be (in our convention) d(dB + C 3 ) = dC 3 . This requirement gives rise to the nonlinear selfduality condition for M5-brane. The action has kappa-symmetry provided that self-duality condition is imposed. After appropriate scalings, the on-shell action of [23] is exactly the same as the action (4.24). Explicitly, the identification is , where primed quantities are the quantities from [23].
It is also possible to compare with various M5-brane actions which use PST formalism. For example, it can easily be seen that the action (4.24) is the same as the on-shell action of [11], [12], [13], [14] provided that the actions are scaled appropriately.
The agreement of the on-shell actions implies the agreement of some properties of solitonic solutions. For example, the value of the on-shell action determines the tension of a self-dual string solution.

On the form of R
We have seen that the action (3.33) which describes nonlinear self-dual 3-form field (as well as the action (4.14) which describes M5-brane) possesses all the required symmetries. Furthermore, it is interesting that in order to obtain the symmetries, we do not need to know the form of R as a function of Q, J, g. Nevertheless, in order to understand more about the action (3.33), it could be useful to be able to explicitly write the form of R. Let us give an initial investigation along this direction.
In this section, we restrict ourselves to theories which describe a * -self-dual 3-form. The * -self-duality is either linear or nonlinear. In the latter case, we do not restrict to just M5-brane.

Linear self-dual case
Let us first consider the linear self-dual case. The task to determine R is equivalent to the task to determine M from the equation subject to * -self-duality of H, * ′ -self-duality of Q, and * ′ -anti-self-duality of R. Furthermore, M is assumed to be symmetric in the sense that ω 1 ∧Mω 2 = ω 2 ∧Mω 1 for any * ′ -self-dual 3-forms ω 1 and ω 2 .
There have been two approaches to derive M. The first one [15], [16] is to essentially write down M formally using the help of some inverse operator. The second one [18] is to first choose the basis and write M in that basis. For our purpose, we find it more convenient to derive a form of M alternative to these two approaches. Let us present our way to obtain this.
The quantity M can be thought of as a map from a * ′ -self-dual 3-form to a * ′ -anti-self-dual 3-form. The requirements from self-duality and anti-self-duality where ω ± ≡ (1 ± * ′ )ω/2 for any 3-form ω. It would be convenient to extend the domain of M to a set of any 3-form. In principle, there are many possible ways to do this. A simple example way is given in [18], This clearly differs from the choice chosen in [18]. Nevertheless, it is possible to show that the choice of M that we have chosen indeed satisfies (5.2) and is also symmetric. The check that our M satisfies eq.(5.2) is straight forward. As for the check of symmetry, let us approach as follows. We first note that the solution to eq.(5.3)-(5.4) is given by This form is in a similar sense as that given in [15], [16] since it involves inverse operator. Note however that eq.(5.5) is not simply a formal expression. The operator * * ′ is a map from 3-forms to 3-forms. So it can be represented by a square matrix.
With the help of Cayley-Hamilton formula, the expression (1 + * * ′ ) −1 should in fact be given as a power series in * * ′ up to 19th order. The coefficients involve expressions of the form tr(( * * ′ ) n ) for non-negative integers n. It is complicated to write it in this way and does not provide useful insights for our purpose. Instead, it is more convenient to write eq. Then by noting that ω 1 ∧( * * ′ ) n ω 2 = −ω 2 ∧( * ′ * ) n ω 1 , (5.7) for any 3-forms ω 1 , ω 2 and any non-negative integer n, it is easy to see that M is symmetric. It can easily be seen that when restricted to flat spacetime, we have M = 0 as expected.

Nonlinear self-dual case
Let us now turn to nonlinear self-dual case. Consider nonlinear * -self-duality condition (3.4) with V given by eq.(3.5). We first need to make sure that these equations are consistent after we express H J = Q − R + J with Q = * ′ Q and R = − * ′ R. This would give a relationship between F 1 ([(K J ) 2 ]) and F 2 ([(K J ) 2 ]), which in turn restrict the form of U.
Let us first discuss the flat case. Imposing an ansatz where (K J Q ) µ λ ≡ (Q J ) µνρ (Q J ) λνρ , and substituting into eq.(3.5) gives Note that Q J is * ′ -self-dual but (Q J ) 3 is * ′ -anti-self-dual. So separately considering self-dual and anti-self-dual part of eq.(5.9) gives Furthermore, by substituting the ansatz (5.8) into [(K J ) 2 ], we obtain Eliminating G and [(K J Q ) 2 ] from eq.(5.10)-(5.12) gives The condition (5.13) relates F 1 and F 2 . It can be seen that F 2 = 0 corresponds to linear * ′ -self-duality condition. By setting F 2 = 0 in the condition (5.13), we obtain F 1 = 0, 1, −1. However, only F 1 = 1 satisfies the original equations (5.10)-(5.12). So we expect that setting F 2 = 0 in the condition (5.13) would correspond to nonlinear self-duality condition provided that we only choose the solutions which reduce to linear self-duality condition in the weak field limit as these solutions would satisfy the original equations (5.10)-(5.12).
An example of nonlinear self-duality condition which satisfies eq.(5.13) and reduces to linear self-duality condition in the weak field limit is the condition (4.3) which is inspired from M5-brane. In this case, the form of R can be obtained and is given by This result can in turn be used to write down eq.(4.14) explicitly in terms of P, Q, g, C 3 , C 6 . It would also be interesting to consider eq.(5.13) perturbatively in [(K J ) 2 ]. Let us assume that F 1 and F 2 can be expanded as Then the leading order of eq.(5.13) gives F 1 (0) = 0, ±1. Again, let us only consider F 1 (0) = 1, since this is the only case which reduces to linear self-duality condition in the weak field limit. So after proceeding to solve eq.(5.13) perturbatively, we obtain (5.19) The expressions (5.17)-(5.18) can also be substituted into eq.(3.10) which gives So theories for nonlinear self-dual 3-form in six dimensional flat spacetime should possess U which takes the universal form (5.20). At this point, let us compare this feature of universality to the case of PST formalism. In PST formalism, an action describing a 2-form field with nonlinear self-dual field strength can also be constructed. This is with the help of auxiliary fields. Actions in this formalism possess PST1 and PST2 symmetry. The presence of PST1 symmetry ensures that nonlinear self-duality condition can be obtained as an equation of motion. As for PST2 symmetry, its presence ensures that auxiliary fields indeed contains no dynamics. In order to see what kind of nonlinear theories are possible, one starts from writing down an action which contains an undetermined potential term. The form of this potential term would correspond to the form of nonlinear self-duality condition. Requiring that the action possesses PST2 symmetry would give rise to the condition called the PST consistency condition which is imposed on the form of the potential term. This condition can then be solved which gives rise to the universal form of the potential term [24]. Note that the PST consistency condition is related to the diffeomorphism invariant (or Lorentz invariant in the case of flat spacetime) requirement in the following sense. After gauge fixing the auxiliary fields, the theory would be non manifestly diffeomorphism invariant. The gauge fixed version of the PST consistency condition ensures that the theory is indeed diffeomorphism invariant.
It can be seen that the situation in Sen formalism is analogous to this provided that we realise U as a potential term. Conditions which restrict the form of U comes from requiring the nonlinear theory to be invariant under the nonstandard Lorentz invariant and from the relationship between the physical field H and the unphysical * ′ -self-dual field Q.
Let us now turn to discuss Sen formalism in curved spacetime. We expect that in this case, due to nonstandard transformation of Q J under diffeomorphism, promoting equations (5.10)-(5.12) to include the metrics would be very complicated since they depend explicitly on Q J . So instead of directly attempting to promote these conditions, let us turn to the condition (5.13). This condition has Q J appeared only implicitly through the field H J , which transforms in the standard way under both types of Lorentz transformations. So eq.(5.13) could then be directly promoted to the case of curved spacetime. It is then easy to see that the promoted condition is covariant.
Since eq.(5.13) can be promoted to the case of curved spacetime, results analysed in the flat case also hold in the curved case. That is, the linear self-duality condition and the M5-brane nonlinear self-dual condition are consistent. Furthermore, the potential U for consistent nonlinear self-dual theories should be perturbatively expressible as in eq.(5.20).
However, when considering the form of R, the extension is not so trivial. That is, in the curved case R J would no longer be of the form (5.8). In order to get some insights, let us consider perturbative expansion. Let us start from where α 1 , α 2 , β 1 , β 2 , · · · are functions of F 2 (0), F ′ 2 (0), F ′′ 2 (0), · · · . The coefficients can be worked out from eq.(5.17)-(5.18). However, for the following discussion we do not need explicit forms of these coefficients. Next, by writing we may obtain the expressions at each order as follows. At the leading order, we have At the third order, we have where we have used * ′ -anti-self-duality of R J 3 . This gives At the fifth order, we obtain (5.27) The calculation can be done up to any desirable orders. This allows us to see a general feature that R J depends on g only through the map M and through raising and lowering of indices. This is interesting that the nonlinearisation does not introduce extra kinds of dependency on g.

Conclusion
In this paper, we have extended the action in Sen formalism [15], [16] to a complete M5-brane action in Green-Schwarz formalism. The action is shown to have all the required symmetries.
The action from Sen formalism which is relevant to us is an action of a 2-form field P and a 3-form field Q, both of which transform in a non-standard way under diffeomorphism transformation. The action is given in such a way that a certain combination H of metric and Q satisfies a self-duality condition off-shell and that H is closed on-shell. A 3-form source J can also be introduced into the theory.
We have shown by explicit construction that it is possible to extend to theories for nonlinear self-dual 3-form field. The theories have all the required gauge symmetries as well as diffeomorphism symmetry. It turns out remarkably that in order to prove the symmetries, one only needs to know the properties of the fields as well as the form of the action (3.33) where U is a function of H J (Q, g, J), g. The explicit form of H J (Q, g, J) is not needed to be expressed. Furthermore, the symmetries are satisfied as long as the conditions (3.8) and (3.9) are met. We expect that the analysis in section 3 can easily be extended to describe nonlinear action for chiral 2n−form in 4n + 2 dimensions in Sen formalism.
We then focus on the action with an appropriate form of U = U M5 . After including a 6-form gauge field, this action can be extended to the action (4.14) which fully describes an M5-brane action in an eleven-dimensional supergravity background. The prove of gauge symmetries and kappa-symmetry can be done without the need to explicitly express H J (Q, g, J). We have also compared the action that we have obtained with some alternative forms of the action in the literature. By appropriate scaling and changing to the same conventions, the actions agree on-shell. The agreement suggests that they provide the same physics of solitonic solutions.
In the analysis explained so far one does not need the explicit form of the action in terms of Q, g, J. Nevertheless, in principle it is important to be able to at least see whether this can consistently be done. In the linear case, as shown in [15], [16], [18], and further confirmed in subsection 5.1, the explicit form of the action can be given. In the flat nonlinear case, in order to ensure the consistency, the conditions (5.10)-(5.12) should be satisfied. These conditions ultimately restrict the form of the nonlinear action. It turns out that the nonlinear self-dual theory obtained from M5-brane indeed satisfy these conditions. As for other nonlinear self-dual theories, we demonstrate a universal perturbative form of the potential U. This universality is analogous to the analysis [24] in PST formalism.
Note that while the conditions (5.10)-(5.12) do not transform in the standard way under standard Lorentz transformation, these conditions give rise to the condition (5.13) which does transform in the standard way. The condition (5.13) can also be thought of as the condition which restricts the form of the nonlinear selfduality condition provided that we choose only the theories which reduce to linear self-dual theory in the weak field limit. When turned to the curved nonlinear case, we expect that promoting eq.(5.10)-(5.12) would be complicated. On the other hand, promoting eq.(5.13) would be straightforward. This promoted equation can then be used as the condition which restricts the form of the action. So the results at this part would be similar to the flat case, namely that the M5-brane nonlinear self-dual theory is allowed and that the potential U on any allowed theories should take the universal perturbative form as presented in eq. (5.20).
In order to write the action explicitly in terms of Q, g, J, one needs to be able to determine R. For the case of flat nonlinear theories, we have solved the conditions (5.10)-(5.12) to obtain the explicit form for M5-brane nonlinear self-dual theory and the perturbative expression for other consistent nonlinear self-dual theories. As for the curved nonlinear theories, although we have shown that M5-brane theory is consistent, it would still be interesting to write the action explicitly in terms of P, Q, g, C 3 , C 6 . We have made an initial attempt along this direction by making perturbative analysis. It turn out that the appearances of the metric g in nonlinear theories are only in raising and lowering indices as well as through the quantity M in the form we present in eq.(5.5). We expect that in order to be able to provide the full analysis, the conditions (5.10)-(5.12) should be promoted to include the metric g. This is a highly nontrivial task, so leave this as a future work.
The explicit form of the nonlinear action in terms of Q, g, J would be required in order to perform the analysis of dimensional reduction, which will provide further physical insights to the theories. In fact, although the explicit full form of nonlinear theory in terms of Q, g, J in curved spacetime is not yet known, dimensional reduction can still be analysed. This is because in this kind of analysis one specifies the form of the metric. So although the metric appears in quite a complicated way in the full theory, dimensional reduction could be relatively simpler. As a future work, we will most likely proceed along this direction to see what theories would arise from the dimensional reductions.
For the analysis in this paper, we assume that the six-dimensional spacetime has a trivial topology. This is to ensure that being closed on-shell, H is exact. It would be interesting to extend the analysis to the case of nontrivial topology and to see if it is possible to compare and contrast with PST formalism in spacetime with nontrivial topology [19], [20].
Finally, there is also another interesting recent construction [25] of chiral 2-form theory. This construction is equivalent to PST formalism. So it could be possible to also obtain a complete M5-brane action based on this construction.