QCD phase transition and equation of state of stellar strong interaction matter via Dyson–Schwinger equation approach

Abstract

We study the phase structure and phase transition of cold dense QCD matter via the Dyson–Schwinger equation approach. We take the rainbow approximation and the Gaussian-type gluon model. In order to guarantee that the quark number density begins to appear at the nuclear liquid-gas phase transition chemical potential, we propose a chemical potential dependent modification factor for the gluon model. We find that for the iso-symmetric quark matter, the modification reduces the chemical potential of the phase coexistence region of the first-order phase transition. We also implement the relativistic mean field theory to describe the hadron matter, and make use of the Maxwell and Gibbs construction method to study the phase transition of $\beta$-equilibrium and charge neutral matter in compact stars. The results show that the phase transition will not happen in case of the Gaussian-type gluon model without any modification. The results also indicate that the upper boundary of the coexistence region should be larger than the current Nambu solution existing region. We also calculate the mass-radius relation of the compact stars, and find that the hadron-quark phase transition happens at too high chemical potential so that the maximum mass of the compact star is hardly affected by the hadron-quark phase transition.

1 Introduction

The quantum-chromodynamics(QCD) is an asymptotic theory. In vacuum, the quarks, which are the fundamental particles of the theory, are confined inside hadrons and cannot be detected directly. At high temperature or/and high density, however, the interaction becomes weak, the quarks can be released from hadrons. The interaction responsible for the confinement also generates over $98\%$ mass of visible matter, and this effect is known as the dynamical chiral symmetry breaking (DCSB) (see, e.g., Refs. [1,2,3,4,5,6,7,8]). The complexity of QCD leads to an abundance of phase structures. At zero chemical potential and finite temperature, it is believed that the transition from hadron matter to quark matter is a crossover [9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26]. At low temperature and high chemical potential, although there are arguments that the transition may also be a crossover [27,28,29,30,31], most model calculations show that this phase transition is of first-order [15, 25, 26, 32,33,34,35,36]. Therefore, there should exist a critical end point (CEP) on the $T-\mu$ plane, which connects the crossover region with the first-order phase transition, and the searching for the CEP is a hot topic in the study of QCD phase structure [15,16,17,18,19,20,21,22,23,24,25,26, 37, 38]. At low temperature and high chemical potential, the phase structure is even more complicated. At baryon chemical potential $\mu _{B}\sim 923\;$MeV, there is a first-order liquid-gas phase transition where the nuclear matter emerges from vacuum [39,40,41,42,43,44,45]. After that, there is the phase transition from nuclear matter to quark matter. It is also possible that the hadron-quark phase transition will lead to color-superconducting quark phase (see Ref. [46] for review and some recent works in Refs. [47,48,49,50,51]).

At sufficiently high temperature or/and large chemical potential, since the interaction is weak, the perturbative QCD approach can provide reliable results on the property of QCD matter [52, 53]. However, in the phase transition region, the interaction is strong, and perturbative approach becomes invalid. One must then implement non-perturbative approaches to study the phase structure. There are phenomenological models such as NJL model, the quark meson model and their Polyakov loop improved versions [35, 54,55,56,57,58,59,60]. Also, there are first-principle methods such as lattice QCD [9,10,11,12,13,14, 61, 62], functional renormalization group theory [7, 8, 22,23,24, 63,64,65,66] and Dyson–Schwinger equation approach (for some reviews, see, e.g., Refs. [21, 67, 68]).

It is well known that lattice QCD is principally only valid at zero chemical potential, since it endures the notorious “sign problem” at finite chemical potential. Although there are extrapolation methods such as the Taylor expansion [69,70,71,72,73], the imaginary chemical potential [74,75,76,77,78,79,80], and other approach (e.g., Ref. [81]), the lattice QCD can still only deal with small chemical potential $\mu /T<2$. Therefore, the theoretical study of the phase transition at zero temperature and finite chemical potential is not as clear as that in zero chemical potential.

The problem also exists on the experimental side. There are colliders such as RHIC, FAIR, NICA and HIAF which aim at studying the structure of hot or warm dense QCD matter, but terrestrial experiments are now not able to create cold dense QCD matter. The possible way to study the phase transition at large chemical potential is by resorting to the astronomical observations. There have been several very heavy compact stars with a mass over $\sim 2M_{\odot }$ discovered [82,83,84,85,86,87], and the detection of gravitational wave from binary neutron star merger also provides constraints on the equation of state (EOS) and hence the possible phase transition inside compact stars [88,89,90,91,92,93,94,95,96,97]. Still, the detail of the phase transition is hidden inside compact stars.

It has been known that the Dyson–Schwinger (DS) equations approach (see, e.g., Refs. [67, 68, 98,99,100,101]) is a continuous field theory of QCD, which can simultaneously deal with the confinement and dynamical chiral symmetry breaking, and is available on the entire $T-\mu$ plane as well as zero current quark mass limit. The DS equations include in fact an infinite number of coupled integral equations, and in order to solve them, one must take truncation [102,103,104,105,106,107,108,109] and model the dressed gluon propagator [110,111,112,113]. The DS equation approach has been taken to study the phase transition and the CEP [15,16,17,18,19,20,21,22, 25, 26, 114,115,116,117,118,119,120], and also been used to study the cold dense matter [121,122,123,124,125,126,127].

However, in spite of the fact that the DS equation approach recovers successfully many properties of the DCSB phase, it is still far from being able to describe the real-world hadron matter. For example, one of the most important properties of hadron matter is the emergence of matter from vacuum, which corresponds to the first-order nuclear liquid–gas phase transition. In DS equation calculations, it has been shown that the quark number density and quark condensate remains the same as in vacuum up to a critical chemical potential [21, 47, 128, 129], but this critical chemical potential is not in accordance with the liquid–gas phase transition chemical potential. Therefore, in this paper, we propose that a chemical potential dependent modification should be applied on the coupling strength of the model in DSE approach. We consider two types of the modification function. The first one is a monotonically decreasing function, which has already been used in previous works [122,123,124, 126, 127]. However, in the earlier works, the modification function includes a free parameter which controls the speed of quark matter approaching the asymptotic freedom. In this work, we will make use of the liquid–gas phase transition to determine the parameter. We will also propose a second type of modification function, which allows for a non-monotonic behavior of the chemical potential dependence.

Even if we correctly recover the liquid–gas phase transition chemical potential, the DSE results is not as accurate as the hadron model in describing the properties of hadron matter. Therefore, in this paper, we will use the hadron model for the hadron sector, and take different construction schemes to describe the phase transition in the cold dense matter. We will also construct the EOS of the hybrid star matter, and we will show that the mass-radius relation of the highly massive compact star can be described well.

The remainder of this paper is organized as follows. After this introduction, in Sect. 2, we reiterate briefly the DS equation approach at zero temperature and finite chemical potential, and propose the chemical potential dependent modification on the model. In Sect. 3, we present the numerical results with the DS equation being solved, including the phase transition of iso-symmetric matter as well as the $\beta$-equilibrium and charge neutral cold dense matter. We also calculate the EOS and the mass-radius relation of the compact star. In Sect. 4, we give a summary and some remarks.

2 Dyson–Schwinger equation approach

2.1 Gap equation at zero temperature and finite chemical potential

In this section, we describe briefly the DS equation approach at zero temperature and finite chemical potential.

The starting point of the DS equation approach is the gap equation:

$$\begin{aligned} S(p;\mu )^{-1}=Z_{2}^{} [i\mathbf {\gamma } \cdot \mathbf {p} + i\gamma _{4}^{} {\tilde{p}}_{4}^{} +m_{q}^{}]+\Sigma (p;\mu ), \end{aligned}$$

(1)

where $S(p;\mu )$ is the quark propagator, ${\tilde{p}}_4=p_4+i\mu$, $\Sigma (p;\mu )$ is the renormalized self-energy of the quark:

$$\begin{aligned} \Sigma (p;\mu )= & {} Z_{1}^{} \int ^{\Lambda }\frac{\text {d}^{4} q}{(2\pi )^{4}} g^{2}(\mu )D_{\rho \sigma }^{} (p-q;\mu ) \nonumber \\&\times \frac{\uplambda ^{a}}{2} \gamma _{\rho } S(q;\mu ) \Gamma _{\sigma }^{a}(q,p;\mu ), \end{aligned}$$

(2)

where $\int ^{\Lambda }$ is the translationally regularized integral, $\Lambda$ is the regularization mass-scale. $g(\mu )$ is the strength of the coupling, $D_{\rho \sigma }^{}$ is the dressed gluon propagator, $\Gamma _{\sigma }^{a}$ is the dressed quark-gluon vertex, $\uplambda ^{a}$ is the Gell–Mann matrix, and $m_{q}^{}$ is the current mass of the quark. $Z_{1,2}^{}$ are the renormalization constants. In this paper, we take a model that ultraviolet integration is finite, so we can move the renormalization point to infinity and take $Z_{1,2}^{}=1$.

At finite chemical potential, the quark propagator can be decomposed according to the Lorentz structure as:

$$\begin{aligned} S(p;\mu )^{-1}= & {} i\mathbf {\gamma } \cdot \mathbf {p} A(\mathbf {p}^2, {\tilde{p}}_4^2) + B(\mathbf {p}^{2},{\tilde{p}}_4^{2}) \nonumber \\&+i\gamma _{4}^{} {\tilde{p}}_{4} C(\mathbf {p}^{2},{\tilde{p}}_{4}^{2}). \end{aligned}$$

(3)

A complete decomposition should include another term proportional to $\sigma _{\mu \nu }$, but this term contributes little, and is usually omitted [21, 67].

At zero chemical potential, a commonly used ansatz for the dressed gluon propagator and the dressed quark-gluon interaction vertex is:

$$\begin{aligned}&Z_{1}^{} g^{2} D_{\rho \sigma }^{}(p-q)\Gamma _{\sigma }^{a}(q,p) \nonumber \\&\quad ={\mathcal {G}}((p-q)^{2} )D_{\rho \sigma }^{\text {free}}(p-q) \frac{\uplambda ^a}{2} \Gamma _{\sigma }^{}(q,p) , \end{aligned}$$

(4)

where

$$\begin{aligned} D_{\rho \sigma }^{\text {free}}(k\equiv p-q)=\frac{1}{k^{2}} \left( \delta _{\rho \sigma }^{} - \frac{k_{\rho }^{} k_{\sigma }^{}}{k^{2}} \right) , \end{aligned}$$

(5)

${\mathcal {G}}(k^{2})$ is the effective interaction to be introduced in the model, and $\Gamma _{\sigma }^{}$ is the quark-gluon vertex. In this paper, we adopt the rainbow approximation which is the leading-order term in a symmetry preserving approximation scheme [130, 131],

$$\begin{aligned} \Gamma _{\sigma }^{}(q,p) = \gamma _{\sigma }^{}. \end{aligned}$$

(6)

Even though the rainbow approximation does not represent the complicated nature of the strong interaction completely, when one studies the properties of cold dense matter, the effect of the tensor and other higher order vertices can be absorbed into the parameter for the gluon model, as shown in Ref. [124]. However, the effect of the higher order vertices on the liquid-gas phase transition has not yet been considered in Ref. [124], we will leave this to our future work and restrict ourselves to the rainbow approximation in this paper.

For the effective interaction, ${\mathcal {G}}(k^2)$ plays the role of the dressing of gluon propagator. Ideally, it should be determined by solving the DSEs of the gluon and the ghost, together with the quark’s gap equation. However, for simplicity, it is also quite common to take an analytic function to model this effective interaction. One of the most widely used model is the Maris-Tandy model [112]:

$$\begin{aligned} \frac{{\mathcal {G}}(k^{2})}{k^{2}}=\frac{4\pi ^{2} D}{\omega ^{6}}k^{2} \text {e}^{-k^{2}/\omega ^{2}} +4\pi \alpha _\mathrm{pQCD}, \end{aligned}$$

(7)

where D and $\omega$ are the parameters of the model. $\alpha _\mathrm{pQCD}$ is the ultraviolet perturbation term. At zero temperature, contributions of the infrared region dominates, and one could neglect the perturbation term. To distinguish from the full Maris–Tandy model, we will, in this paper, denote our model as “Gaussian type” model. The parameters of the model are $\omega =0.5\,$GeV and $D=1.0\,\text { GeV}^2$ which are determined with the pion mass $m_{\pi }=0.14\;$GeV and decay constant $f_{\pi }=0.091\;$GeV with $m_{q}=5\;$MeV for u and d quark [132]. And the current mass of the strange quark is $m_{q}=115\;$MeV by fitting the kaon mass $m_{K} =0.492\;$GeV [133].

The parameters of our model is fixed by fitting the properties of mesons in vacuum. At finite chemical potential, however, it is certain that this approximation loses its accuracy and requires modification. We will discuss the modification in Sect. 2.3.

The DS equation has multiple solutions. In chiral limit (zero current quark mass) and zero chemical potential, there is a solution with $B(p)\equiv 0$, and a solution with $B(p)>0$, which are called Wigner solution, Nambu solution, respectively. The Wigner solution corresponds to the dynamical chiral symmetry (DCS) phase, where the quarks are massless. The Nambu solution corresponds to dynamical chiral symmetry breaking (DCSB) phase, since the mass function $M(p)=B(p)/A(p)$ acquires a non-zero value. Although there are hints that there might exist a quarkyonic phase where the quark is confined but the chiral symmetry is preserving, it is usually believed that the DCSB (Nambu) solution corresponds to the confined hadron phase, and the DCS (Wigner) solution corresponds to the deconfined quark phase. For both the DCSB and DCS solutions, the scalar functions A(p) and C(p) have values of order 1 for all momentum p, and $A(p)=C(p)=1$ when p approaches infinity. At finite chemical potential, all the three functions A, B and C become complex, but their imaginary part is significantly smaller than the real part. We still have $\text {Re} A(|\mathbf {p}|,p_4)\sim \text {Re} C(|\mathbf {p}|,p_4)\sim 1$, and the $\text {Re} B(|\mathbf {p}|=0,p_4=0)$ can be used to distinguish the DCSB solution from the DCS one.

At zero temperature and small quark chemical potential, the DCSB phase should stay in the vacuum ground state until the chemical potential is larger than some critical value. This property is known as “Silver–Blaze” property of QCD. At baryon chemical potential $\mu _{B}^{}=923\;$MeV, which is the proton mass minus the binding energy, there will be a liquid–gas phase transition where nucleons begin to emerge from vacuum, and the Silver–Blaze property will be broken beyond this chemical potential.

2.2 Quark number density and Silver–Blaze property

After solving the DS equation at some quark chemical potential, the number density of the quarks can be obtained through [121]:

$$\begin{aligned} n_{q}^{}(\mu ) = 6\int \frac{\text {d}^{3}p}{(2\pi )^{3}} f_{q}^{}(|\mathbf {p}|;\mu ), \end{aligned}$$

(8)

where $f_{q}^{}$ is the distribution function and reads

$$\begin{aligned} \begin{aligned} f_{q}^{}(|\mathbf {p}|;\mu )&= \frac{1}{4\pi }\int _{-\infty }^{\infty }\text {d}p_{4}^{} \text {tr}_{\text {D}}^{}[-\gamma _{4}^{} S_{q}^{}(p;\mu )] , \end{aligned} \end{aligned}$$

(9)

where the trace is for the spinor index.

The integration in Eq. (9) can be converted to a contour integral on the complex plane of ${\tilde{p}}_4$:

$$\begin{aligned} f_{q}^{}(|\mathbf {p}|;\mu )= & {} \frac{1}{\pi }\int _{-\infty }^{\infty }\text {d}p_{4}^{} \frac{i\left( p_{4}+i\mu \right) C\left( |\mathbf {\,p}|^2,\left( p_{4}+i\mu \right) ^2\right) }{{\mathcal {M}}}\nonumber \\= & {} \frac{1}{\pi }\int _{-\infty +i\mu }^{\infty +i\mu }\text {d}{\tilde{p}}_{4} \frac{i{\tilde{p}}_{4}C(|\mathbf {\,p}|^2,{\tilde{p}}_{4}^2)}{{\mathcal {M}}}\nonumber \\= & {} \frac{1}{\pi }\Bigg (\oint _{(-\infty +i\mu )\rightarrow (\infty +i\mu )\rightarrow (\infty )\rightarrow (-\infty )\rightarrow (-\infty +i\mu )} \nonumber \\&-\int _{\infty }^{-\infty } - \int _{-\infty }^{-\infty +i\mu } - \int _{\infty +i\mu }^{\infty }\Bigg )\frac{i{\tilde{p}}_{4}C(|\mathbf {\,p}|^2,{\tilde{p}}_{4}^2)\text {d}{\tilde{p}}_{4}}{{\mathcal {M}}},\nonumber \\ \end{aligned}$$

(10)

where

$$\begin{aligned} {\mathcal {M}}=\mathbf {\,p}^{2}A^{2}(|\mathbf {\,p}|^2,{\tilde{p}}_{4}^2)+{\tilde{p}}_{4}^{2}C^2(|\mathbf {\,p}|^2,{\tilde{p}}_{4}^2) + B^2(|\mathbf {\,p}|^2,{\tilde{p}}_{4}^2).\nonumber \\ \end{aligned}$$

(11)

The first equality in Eq. (10) is obtained by carrying out the trace in Eq. (9), and the second one in Eq. (10) is simply a replacement of the dummy variable. The integration path of the second line of Eq. (10) is a horizontal line on the complex plane. It corresponds to the horizontal solid red line with non-zero imaginary value in the upper panel of Fig. 1. This horizontal line is part of the rectangular contour, given in the third line in Eq. (10), which corresponds to the red solid rectangular in the upper panel of Fig. 1. The value of the contour integral can be calculated using the residue theorem, and the value of the integral in the second line of Eq. (10) equates the contour integral minus the other three edges of the rectangular. This leads to the third equality of Eq. (10).

In the last line of Eq. (10), the integral $\int _{\infty }^{-\infty }$ is zero since the kernel is an odd function, and the last two integrals are also zero because the denominator ${\mathcal {M}}$ diverges at infinity. Therefore, the value of distribution function $f_{q}$ is determined by the poles inside the contour.

Fig. 1

Schematic feature of the contour and the singularities. Upper panel: by increasing the chemical potential, the contour may include the pole, and the number density becomes nonzero. Lower panel: for a fixed chemical potential, by decreasing the coupling constant D, the imaginary part of the pole becomes small and moves inside the contour, and the number density becomes nonzero

The pole corresponds to the zero in the denominator ${\mathcal {M}}$:

$$\begin{aligned} {\tilde{p}}_{4}=\pm \frac{i\sqrt{\mathbf {p}^2A^2+B^2}}{C}. \end{aligned}$$

(12)

A schematic figure of the pole and the contour is shown in the upper panel of Fig. 1. For Nambu solution, since B is non-zero, and C is of order 1, we can see from Eq. (12) that the imaginary part of the first pole, $\mu _{a}$, is finite for arbitrary value of $\mathbf {p}^2$, and for the chemical potential $\mu <\mu _{a}$, the integral in Eq. (10) is zero since there is no singularities inside the contour, and the quark number density remains zero. When $\mu > \mu _{a}$, the contour begins to include the pole, and the number density becomes nonzero. This is a manifestation of the Silver–Blaze property.

2.3 Modification for the Gluon model

In DS equation calculations, the Silver–Blaze property is maintained. The condensation and number density of the Nambu solution remains the same as in vacuum for small quark chemical potential (see, e.g. Refs. [121, 129, 134]). However, the critical chemical potential where quark number density becomes non-zero relies on the vertex and the gluon model, and is usually not in accordance with the nuclear liquid-gas phase transition chemical potential.

We notice that, by altering the coupling constant D in Eq. (7), the critical chemical potential can be changed. The schematic characteristic of the procedure is shown in the lower panel of Fig. 1. For a fixed chemical potential, the pole may lie outside the contour of Eq. (10), and the number density is zero. However, by decreasing the coupling constant D, the imaginary part of the pole should also decrease. When the pole moves into the contour, the number density will become nonzero.

In order to correctly recover the liquid-gas phase transition chemical potential, we are going to add a chemical potential dependence to the coupling D by multiplying a modification function $h(\mu )$ to the coupling constant D defined in Eq. (7), and we have

$$\begin{aligned} D(\mu )=D(\mu =0)h(\mu ), \end{aligned}$$

(13)

where $D(\mu =0)=1.0\;\text {GeV}^{2}$.

There are three constraints on the modification function $h(\mu )$: First, we should have $h(\mu =0)=1$ since the vacuum value is fixed by meson properties. Second, $h(\mu =\infty )=0$ to take into account the asymptotic freedom at large chemical potential. And we also require that $h(\mu )$ guarantees that the critical chemical potential is $\mu _{B,c}=923\,$MeV which is in accordance with the emergence of nuclear matter.

Previously, in the study of compact star matter, an exponentially damping function was proposed [122,123,124, 126, 127] as:

$$\begin{aligned} h_{\alpha }(\mu _{q})=\exp (-\alpha \mu _{q}^2/\omega ^2), \end{aligned}$$

(14)

where $\alpha$ is a parameter and $\omega$ is the same as in the original gluon model. This function is a monotonically decreasing function. In general, a larger coupling constant D corresponds to a larger quark mass function M(p), and a larger critical chemical potential. Meanwhile, it reduces the value $\mu _{B,c}^{}$ for the phase transition to happen [127].

However, for some vertex and gluon propagator, the $\mu _{B,c}^{}$ might be less than $923\,$MeV (see, e.g. Ref. [134]), which indicates that a non-monotonic modification function is required. In this work, we introduce another type of modification function:

$$\begin{aligned} h_{\beta }(\mu _{q})=\left( 1+\frac{\mu _{q}^2}{\mu _{q,c}^2}\right) \exp (-\beta \mu _{q}^2/\omega ^2), \end{aligned}$$

(15)

The value of the parameters $\alpha$ and $\beta$ are fixed by requiring $\mu _{B,c}^{}=923\,$MeV (i.e., $\mu _{q,c}^{}=307.6\,$MeV). In the following, we will denote the modification function in Eq.(14), in Eq.(15) as $\alpha$ type, $\beta$ type, respectively.

2.4 Numerical calculation

Since what we need to solve are the functions A, B and C in terms of the momentum, the momentum must be discretized at first. The most direct way to carry out the discretion is taking $|\mathbf {p}_{1}^{}|, |\mathbf {p}_{2}^{}|,\cdots |\mathbf {p}_{N}^{}|$ and $p_{4,1}^{},p_{4,2}^{},\cdots p_{4,N}^{}$ with average spacing. However, since the main feature of the propagator is in the infrared domain, we take then the logarithmic spacing to guarantee enough grid points in the infrared, i.e., we choose $\log |\mathbf {p}_{1}^{}|, \cdots \log |\mathbf {p}_{N}^{}|$ and $\log p_{4,1}^{},\cdots \log p_{4,N}^{}$ to be average spacing. The smallest, largest momentum point is $|\mathbf {p}|_\mathrm{min}^{} = p_{4,\text {min}}^{} =0.01\,$GeV, $|\mathbf {p}|_\mathrm{max}^{} = p_{4,\text {max}}^{}=100\,$GeV, respectively. The number of the momentum points N varies for different chemical potential, and ranges from $N\sim 50$ at $\mu =1\;$GeV to $N\sim 100$ near the phase boundary.

There is another way to discretize the momentum. The A, B and C can also be regarded as functions of P and $\cos \theta$, where $P=\sqrt{\mathbf {\,p}^2+p_{4}^2}$ is the module of the four momentum, and $\theta =\arccos (p_{4}/P)$ is the angle between the four momentum and its 4th-direction projection. In our calculation, P is discretized with logarithmic average spacing on the range $(0.01,100)\;$GeV, the $\cos \theta$ is discretized averagely on the range $(-1,1)$. The number of points for P is about $N_{P}\sim 200$, and the number of points for $\cos \theta$ is about $N_{\theta }\sim 50$.

In this paper, we make use of the first discretion method to searching the Wigner solution, since it has a better numerical behavior when calculating the number density. And we implement the second discretion method to determine the Nambu solution, since this method preserves the O(4) symmetry of the vacuum.

In order to solve the gap equation, Eq. (1) is usually simplified with some trace technique.

From Eq. (3) and the properties of gamma matrix, we have:

$$\begin{aligned} \begin{aligned}&\text {tr}_{\text {D}}\left[ S^{-1}(\mathbf {p},p_{4};\mu )\right] =4B(\mathbf {\,p}^2,{\tilde{p}}_{4}^2),\\&\text {tr}_{\text {D}}\left[ i\mathbf {\gamma }\cdot \mathbf {p}S^{-1}(\mathbf {p},p_{4};\mu )\right] =-4\mathbf {p}^2A(\mathbf {\,p}^2,{\tilde{p}}_{4}^2),\\&\text {tr}_{\text {D}}\left[ i\gamma _{4}{\tilde{p}}_4S^{-1}(\mathbf {p},p_{4};\mu )\right] =-4{\tilde{p}}^2C(\mathbf {\,p}^2,{\tilde{p}}_{4}^2),\\ \end{aligned} \end{aligned}$$

(16)

and by multiplying gamma matrix and taking trace on both side of Eq. (1), the gap equation becomes:

$$\begin{aligned} \begin{aligned} B(\mathbf {\,p}^2,{\tilde{p}}_4^2)&=m_q+\frac{1}{4}\text {tr}_{\text {D}}\left[ \Sigma (\mathbf {p},p_{4};\mu )\right] ,\\ A(\mathbf {\,p}^2,{\tilde{p}}_4^2)&=1-\frac{1}{4\mathbf {\,p}^2}\text {tr}_{\text {D}}\left[ i\mathbf {\gamma }\cdot \mathbf {p}\times \Sigma (\mathbf {p},p_{4};\mu )\right] ,\\ C(\mathbf {\,p}^2,{\tilde{p}}_4^2)&=1-\frac{1}{4{\tilde{p}}_4^2}\text {tr}_{\text {D}}\left[ i\gamma _{4}{\tilde{p}}_4 \times \Sigma (\mathbf {p},p_{4};\mu )\right] . \end{aligned} \end{aligned}$$

(17)

Then, after discretizing the momentum, we can choose arbitrary initial values for A, B and C, and implement Eq. (17) to do iteration with proper algorithm (for example, Broyden iteration), until the solution reaches some required precision.

3 Results and discussions

3.1 Determination of parameter

The Nambu solution of the quark’s DS equation has a relatively large mass function. Therefore, at small chemical potential, the number density of quark in Nambu solution should be zero. The mass function is related to the coupling constant D, so for a fixed chemical potential, the quark number density relies definitely on the coupling strength D.

For $D/D_{0}=1$, where $D_{0}=1.0\,\text {GeV}^2$ is the coupling strength in vacuum and fixed with the meson properties, the number density should be zero [121], which satisfies the Silver-Blaze property. However, because of numerical error, the calculated number density is non-zero and numerically unstable. Therefore, it is hard to determine the critical coupling strength via the number density directly.

Another way to determine the critical coupling strength is by searching for the mass pole. As we have stated in the last section, the emergence of the quark number density corresponds to a mass pole in the propagator. The denominator ${\mathcal {M}}$, as defined in Eq. (11), is a function of $|\mathbf {\,p}|$ and $p_{4}^{}$, for a fixed value of $\mu =\mu _{q,c}$. When the pole enters the contour of the integral in Eq. (10), as illustrated in the lower panel of Fig. 1, there should be a zero ${\mathcal {M}}$ for a certain value of $|\mathbf {\,p}|$ and $p_{4}$, or equivalently, the maximum value of $1/|{\mathcal {M}}|$ should be divergent. The obtained maximum value of the $1/|{\mathcal {M}}|$ as a function of coupling constant is shown in Fig. 2. From the figure, it is apparent that $1/|{\mathcal {M}}|$ has a pole at $D_{c}=0.716D_{0}$, where the quark number density begins to appear.

Fig. 2

Calculated coupling strength D dependence of the denominator $1/|{\mathcal {M}}|$ defined in Eq. (11), with the critical chemical potential $\mu _{q,c}=307.6\;$MeV, and dressed gluon propagator in the Gaussian-type with $D = D_{0}=1.0\;\text {GeV}^{2}$

The parameter $\alpha$ and $\beta$ in Eqs. (14) and (15) is fixed by requiring that $h(\mu _{q,c})\times D_{0}=D_{c}$, and we have $\alpha =0.883$ and $\beta =2.714$. The calculated variation behavior of the modification coefficient h with respect to the quark chemical potential $\mu _{q}$ (i.e., $\mu _{B}^{}/3$) is shown in Fig. 3. The black solid line corresponds to the result with the $\alpha$-type modification function defined in Eq. (14) and the red dashed line corresponds to the $\beta$-type modification function defined in Eq. (15). Both of the two presently obtained modification functions are monotonically decreasing. However, at small chemical potential, the modification $h_{\alpha }(\mu )$ is smaller than the $h_{\beta }(\mu )$, while at large chemical potential $h_{\beta }(\mu )$ decreases to almost zero more rapidly. The two lines cross at $\mu _{q}=307.6\;$MeV, which is required by our assumption. After fixing the parameters, we can solve the DS equations with the Gaussian-type dressed gluon model defined in Eq. (7) without any modification, with the $\alpha$-type modification and the $\beta$-type modification, respectively.

Fig. 3

Calculated quark chemical potential dependence of the modification coefficient h. The black solid line corresponds to the $\alpha$-type modification function defined in Eq. (14), and the red dashed line corresponds to the $\beta$-type modification function defined in Eq. (15)

3.2 Phase transition of the iso-symmetric matter and the coexistence region

Because of the asymptotic nature of QCD, there must exist a deconfined phase transition at high density (chemical potential). Regardless of the possible existence of quarkyonic phase, this phase transition is also the chiral phase transition from DCSB phase to DCS phase.

The order parameter of the chiral phase transition reads usually the condensate of quark, which is the trace of the quark propagator. The chiral susceptibility, which is the derivative of the quark condensate with respect to the current quark mass, is taken as the criteria of the phase transition.

The chiral susceptibility is related to the stability of the system [15, 26]. If the chiral susceptibility is positive, the system is stable or meta-stable, and the system is unstable if the chiral susceptibility is negative.

In vacuum, the chiral susceptibility of the Nambu solution, which corresponds to the DCSB phase, is positive, while the chiral susceptibility of the Wigner solution, which corresponds to the DCS phase, is negative(see, e.g. Ref. [15]). This means that the DCS quark matter is unstable in vacuum, and the system should be described by the Nambu solution. When the chemical potential gets large enough, the Nambu solution will disappear, and the chiral susceptibility of the Wigner solution is positive, this means that the quarks gets deconfined from hadrons and the matter should be in quark phase.

In chiral limit, the quark condensate and the chiral susceptibility is well-defined. In case of nonzero current quark mass, however, both the quark condensate and the chiral susceptibility with the original definition in tracing the propagator diverges drastically. There have been several subtraction scheme to eliminate the divergence (see, e.g., Refs. [2, 3, 12, 15, 21, 26, 66]), but it is quite convenient to simply make use of the Lorentz scalar part of the quark propagator at zero momentum, i.e., taking $B(p=0)$ as a representative of the order parameter, and $\chi _{m}\equiv \frac{\text {d}B(p=0)}{\text {d}m_{0}}$ as chiral susceptibility, as has been done in Refs. [15, 26, 121]. Specifically, it has been shown that, for the phase transition at zero chemical potential and high temperature, the value of the pseudo-critical temperature would be slightly different by using this criterion [26]. At zero temperature and high chemical potential, however, the chiral susceptibility will diverge at the phase boundary, and the phase transition chemical potential determined with this definition is not different from those fixed via other forms of the definition.

In Fig. 4, we show the calculated chiral susceptibilities of both the Nambu and the Wigner solutions with the different gluon models, and we have neglected the solution where the chiral susceptibility is negative. As we can see from the figure, for all the three models, there exists a (quark) chemical potential region where both the Nambu solution and the Wigner solution have positive chiral susceptibility, and this region is usually referred to as the “phase coexistence region”, where the DCSB hadron matter and the DCS quark matter appear simultaneously.

Fig. 4

Calculated chiral susceptibility with different gluon models. The black lines correspond to the results in the Gaussian-type model without modification, the red lines stand for the results implementing the $\alpha$-type modification function defined in Eq. (14), and the blue lines represent the results using the $\beta$-type modification function defined in Eq. (15). The solid lines correspond to that of the Nambu solution, and the dashed lines correspond to that of the Wigner solution

In details, in case of the Gaussian-type gluon model without chemical potential dependence, the coexistence region is $\mu _{q} \in [0.322, 0.500]\,$GeV. In case of the Gaussian-type gluon model with the $\alpha$-type modification, the coexistence region is $\mu _{q} \in [0.271 , 0.309]\,$GeV, and for the gluon model with the $\beta$-type modification, the coexistence region is $\mu _{q} \in [0.276, 0.308]\,$GeV. The red solid and blue solid lines in Fig. 4 are then not exactly the same, but appear very close to each other. More concretely, at zero chemical potential, the results in the $\alpha$-type and $\beta$-type are exactly the same, because there should be no modification in vacuum. And the chiral susceptibilities in the two models both become divergent soon at the liquid-gas phase transition. At low chemical potential, the silver-blaze property guarantees that the system is very similar to the vacuum state, and the two models give similar results. Both the upper and lower boundaries of the coexistence region are smaller than the corresponding one in case of without the modification, and the coexistence region gets narrower. This is because our modification coefficients generally reduce the interaction strength, and it will be easier for the quarks to be released from hadrons.

3.3 $\beta$-equilibrium quark matter

In the last subsection, we have shown the calculated phase boundary and phase transition for the iso-symmetric matter, i.e., the chemical potentials of the u and d quarks are the same. However, it is now impossible for us to get high density matter in terrestrial experiment. The most common way to study the phase structure and phase transition in the high density region is then to resort to the studying of the matter in compact stars. The compact star matter is usually in $\beta$-equilibrium and charge neutral. Therefore, we are now going to study the $\beta$-equilibrium and charge neutral quark matter.

To study of the property of compact star matter composed of both hadrons and quarks (hybrid matter), it is presently common to implementing the theoretical approach of hadron matter and that of quark model separately to study hadron, quark phase, respectively, and combine them with construction schemes.

In this section, we calculate the property of quark matter by solving the DS equation of quark, and only keep the Wigner solution, since the Nambu solution should correspond to the hadron phase. For the hadron phase, however, we take the relativistic mean field (RMF) theory with TW-99 parameterization [135], which is described in the appendix.

Fig. 5

Calculated number density (in unit of the saturation nuclear matter density $n_{S}$) of the u and d quarks as a function of quark chemical potential. The black solid line corresponds to the result using the Gaussian-type gluon model without modification, the red dashed line stands for the result by implementing the Gaussian-type gluon model with the $\alpha$-type modification, and the blue dotted line represents the result with the $\beta$-type modification

The calculated quark chemical potential dependence of the number density of u and d quarks in the Wigner solution is shown in Fig. 5. It is clear that the result with the $\beta$-type modification has the largest number density. This is physically reasonable, because the $\beta$-type modification has the smallest coupling strength in the relevant chemical potential region, and the dressed mass is generally smaller corresponding to a smaller coupling strength, and therefore the number density of the quarks with smaller dressed mass is larger.

In the study of compact star matter, it is also necessary to take into consideration the effect of strange quarks. The study of the solutions of DS equation has revealed that, for a fixed coupling strength, there exists a critical current quark mass, above which the Wigner solution will disappear [2, 3, 6]. For a coupling strength fixed in vacuum, the strange quark mass is well above the critical mass, which means that the strange quark does not have the Wigner solution for the Gaussian-type gluon model without modification, whose coupling strength is the same as that in vacuum at any chemical potential. In case of the gluon model with the modifications, however, since the Wigner solution must exist at large enough chemical potential where the coupling strength is quite small, we can then get the strange quark number density above some critical chemical potential.

Fig. 6

Calculated number density (in unit of the saturation nuclear matter density $n_{S}$) of the s quark as a function of the quark chemical potential. The red dashed line corresponds to the result using the Gaussian-type gluon model with the $\alpha$-type modification, and the blue dotted line stands for the result with the $\beta$-type modification

The calculated number density of the strange quarks as a function of quark chemical potential is shown in Fig. 6. One can notice from Fig. 6 that, in case of the gluon model with the $\alpha$-type modification, the critical chemical potential for the strange quark to appear is $\mu _{s,c}=0.461\,$GeV, while for the gluon model with the $\beta$-type modification, the critical chemical potential is $\mu _{s,c}=0.420\,$GeV. The critical chemical potential in case of the $\beta$ type modification is smaller, because it has smaller coupling strength in the relevant chemical potential region. Also, the strange quark density for the $\beta$-modification is larger, with the same reason for that the u and d quark number density is larger for the $\beta$-type modification.

After calculating the number density of the quarks, the pressure of each flavor of the quark at zero temperature can be obtained by integrating the number density:

$$\begin{aligned} P_{q}^{}(\mu _{q}^{}) = P_{q}^{}(\mu _{q,0}^{}) + \int _{\mu _{q,0}^{}}^{\mu _{q}^{}} \text {d}\mu n_{q}^{}(\mu ). \end{aligned}$$

(18)

Mathematically, the starting point of the integration $\mu _{q,0}$ can be any value, but in our calculation, it is more convenient to choose $\mu _{u/d,0}$ to be the value of the left boundary of the coexistence region, and $\mu _{s,0}$ to be the critical chemical potential where the Wigner solution for the s quark to appear. In Ref. [121], the pressure difference between the Nambu and Wigner solutions is calculated by using the “steepest-descent” approximation, and since the Nambu solution corresponds to the vacuum at $\mu _{q,0}$, the initial value of the integration can be chosen as $P_{q}^{}(\mu _{q,0}) = P_{W}(\mu _{q,0}) - P_{N}(\mu _{q,0})$, where $P_{W}$ and $P_{N}$ is the pressure of the Wigner solution, Nambu solution, respectively. By adopting the result from Ref. [121], we have $P_{q}^{}(\mu _{q,0})=2.58\times 10^{-4}$, $4.32\times 10^{-4}$ and $4.19\times 10^{-4} \text {GeV}^{-4}$ in case of the Gaussian-type gluon model without modification, with the $\alpha$-type modification and with the $\beta$-type modification, respectively. For the s quark, we simply choose $P_{s}(\mu _{s,0})=0$ as in Ref. [122].

The total pressure of the quark matter is the sum of the pressure of each flavor of the quarks:

$$\begin{aligned} P_{Q}^{}(\mu _{u}^{},\mu _{d}^{},\mu _{s}^{}) = \sum _{q=u,d,s}{\tilde{P}}_{q}^{}(\mu _{q}^{}) - B_{\text {DS}}^{}, \end{aligned}$$

(19)

where

$$\begin{aligned} {\tilde{P}}_{q}^{}(\mu _{q}^{})\equiv \int _{\mu _{q,0}^{}}^{\mu _{q}^{}}\text {d}\mu n_{q}^{}(\mu ), \end{aligned}$$

(20)

and

$$\begin{aligned} B_{\text {DS}}^{} \equiv -\sum _{q=u,d,s} P_{q}^{}(\mu _{q,0}^{}). \end{aligned}$$

(21)

In compact star matter, the contributions from leptons is also important. Usually, the electron and muon are considered. The number density for the leptons is:

$$\begin{aligned} n_{l}^{} = \frac{k_{Fl}^{3}}{3\pi ^{2}}, \end{aligned}$$

(22)

where $k_{Fl}^{2} = \mu _{l}^{2} - m_{l}^{2}$ for $l=e^{-},\, \mu ^{-}$. In this work, we take $m_{e}=0.511\;$MeV and $m_{\mu ^{-}}=105\;$MeV.

The quark matter in a compact star should be in $\beta$-equilibrium and electric charge neutral, so we have:

$$\begin{aligned}&\mu _{d}^{}= \mu _{u}^{} + \mu _{e}^{} = \mu _{s}^{}, \nonumber \\&\quad \mu _{\mu ^{-}}=\mu _{e}\end{aligned}$$

(23)

$$\begin{aligned}&\frac{2n_{u}^{} - n_{d}^{} - n_{s}^{}}{3} - n_{e}^{} - n_{\mu ^{-}}^{} = 0. \end{aligned}$$

(24)

And we have the baryon density and chemical potential as:

$$\begin{aligned} n_{B}^{}= & {} \frac{1}{3}(n_{u}^{} + n_{d}^{} + n_{s}^{} ), \end{aligned}$$

(25)

$$\begin{aligned} \mu _{B}^{}= & {} \mu _{u}^{} + 2\mu _{d}^{}. \end{aligned}$$

(26)

With these relations, we can calculate the properties of the $\beta$-equilibrium and charge neutral quark matter with a given baryon chemical potential (baryon density).

Except for solving the DS equation for the quark matter and the RMF for the hadron matter, we also need to make use of some construction scheme to describe the hadron-quark phase transition. The condition for the phase transition to occur is the chemical potential and the pressure in the two phases are equal, i.e.,

$$\begin{aligned} p_{H}(\mu _{B})=p_{Q}(\mu _{B}), \end{aligned}$$

(27)

where the footnote H and Q denote the hadron and quark sector, and the pressure as a function of baryon chemical potential is determined by the hadron and quark model we implemented. By solving the above equation, we can get the chemical potential corresponding to the hadron-quark phase transition.

The above described construction scheme is called the “Maxwell construction”, which provides a straight forward way to describe the phase transition, and is widely used. However, it only considers one chemical potential, the baryon chemical potential. In the compact star matter, there are two conservation numbers, the baryon number and the charge number. In turn, there are two chemical potentials, the baryon chemical potential and the charge chemical potential. We should then implement the Gibbs construction to take into account both of the chemical potentials.

In Gibbs construction, it assumes that there is a mix phase region (phase coexistence region) where both quark and hadron exist simultaneously, and the baryon chemical potential ($\mu _{B}^{}$) and the charge chemical potential($\mu _{e}$) are the same in both the quark and the hadron phases.

In the phase coexistence region, the pressure of the two phases are the same. And though the two phases are no longer charge neutral separately, there will be a global charge neutral condition. If we define the quark fraction $\chi \in [0, 1]$, the phase transition conditions can be expressed as:

$$\begin{aligned}&p_H(\mu _B,\mu _e)=p_Q(\mu _B,\mu _e), \end{aligned}$$

(28)

$$\begin{aligned}&(1-\chi )n^c_H(\mu _B,\mu _e)+\chi n^c_Q(\mu _B,\mu _e)=0, \end{aligned}$$

(29)

where $p_{H}$, $p_{Q}$ is the pressure of the hadron, the quark phase, respectively, which is a function of both the $\mu _{B}$ and the $\mu _{e}$. $n^{c}_{H}$, $n^{c}_{Q}$ is the charge density of each of the two phases, which is determined by the corresponding hadron and quark model.

Then, combining Eqs. (28) and (29), together with the field equations of the two phases, we can obtain the $\mu _{B}^{}$ and $\mu _{e}$ with any given quark fraction $\chi$. By taking $\chi =0,\;1$, we can calculate the phase boundary of the coexistence region under the charge neutral and $\beta$-equilibrium condition.

Fig. 7

Calculated pressure as a function of baryon chemical potential. The black solid line corresponds to the pressure of hadron matter without hyperons using the RMF TW-99 model, the gray dotted line corresponds to that of hadron matter with hyperons in the RMF TW-99 model, the green line with diamond symbol corresponds to the quark matter with DS equation calculation with the Gaussian-type gluon model without modification, the red line with circle symbol stands for the quark matter with the DS equation calculation with the $\alpha$-type modification, the blue line with star symbol represents that of the quark matter with the DS equation calculation implementing the $\beta$-type modification, the orange dashed line with circle symbol corresponds to the Gibbs construction linking the RMF TW-99 hadron model and the DS equation using the $\alpha$-type quark model, and the pink dashed line with star symbol is that via the Gibbs construction linking the RMF TW-99 hadron model and the DS equation using the $\beta$-type quark model

The calculated baryon chemical potential dependence of the pressure of each kind matter is shown in Fig. 7. From the figure, we can see that, among the three quark models, the one with the $\beta$-type modification has the largest pressure, which is in accordance with that the $\beta$-type modification has the largest quark number density.

In Fig. 7, the black solid line and gray dotted line represent the results of the calculated hadron matter pressure via the TW-99 RMF model described in the appendix. The black solid line corresponds to the nuclear matter that only proton, neutron and leptons are considered, while the gray dotted line is the one with the contribution of the hyperons having been taken into consideration. As we have mentioned, for the hadron-quark phase transition to take place, the pressure and the chemical potential of hadron matter and those of the quark matter must be simultaneously the same, respectively. This means that the P–$\mu _{B}^{}$ curve of the hadron matter and that of the quark matter must have a cross point. However, Fig. 7 manifests clearly that, in case of the Gaussian-type gluon model without any modification, the pressure of the quark matter is too small and there is no cross point with the line of hadron matter. This means that, for the Gaussian-type gluon model without modification, the coupling strength in the relevant chemical potential region is too strong and should be excluded, since the hadron-quark phase transition is forbidden in the model, which is physically unlikely. This further indicates the necessity of taking into consideration the modification factor. For the Gaussian-type gluon model with the $\alpha$- and $\beta$-type modification, the phase transition chemical potential for the $\beta$-equilibrium and charge neutral matter is $\mu _{B}^{}=1.94\,$GeV, $1.71\,$GeV, respectively. And the gray dotted line does not have cross point with all the quark lines, which means that the phase transition will not happen after the inclusion of hyperons. Therefore, we will not include the hyperon effect in the following.

By implementing the Gibbs construction, we can obtain that the phase coexistence region of the $\beta$-equilibrium and global charge neutral matter is $\mu _{B}^{} \in [1.69 , 2.02]\,$GeV, $\mu _{B}^{} \in [1.48, 1.79]\,$GeV in case of the model with the $\alpha$-type, $\beta$-type modification, respectively, as shown in Fig. 7.

The obtained phase coexistence region and the phase boundary here is different from what we have shown in Sect. 3.2, but with some simple relations:

$$\begin{aligned} 3\mu _{W,c}\le \mu _{G,1}\le \mu _{G,2}\le 3\mu _{N,c}, \end{aligned}$$

(30)

where $\mu _{G,1}^{}$, $\mu _{G,2}^{}$ is the baryon chemical potential corresponding to the left, the right boundary of the coexistence region from the Gibbs construction, $\mu _{W,c}^{}$ is the quark chemical potential corresponding to the divergence of the chiral susceptibility of the Wigner solution (DCS phase), and $\mu _{N,c}^{}$ corresponds to the divergence of the chiral susceptibility of the Nambu solution (DCSB phase).

The first inequality in Eq. (30) is satisfied well in our calculation, while the last inequality is not satisfied. This deficiency comes from both the insufficiency of the hadron model and the quark model.

By solving the DS equation of the quark(s), we are calculating the property of uniform quark matter. However, the quarks in the hadron matter is not uniformly distributed since they are clustered as hadrons. So the Nambu solution of DS equation is different from the real world hadron matter after the appearance of quark number density.

The RMF model also has its deficiency. The parameters of the hadron matter model are fixed by fitting the property of the hadron matter at saturation density or lower densities where terrestrial experiments are able to create. Beyond the saturation density, different models give distinct results. Still, the RMF model is more reliable than the Nambu solution of the DS equation in describing the hadron matter, since the solution of the DS equation has not taken into account the effects of the non-uniform distribution of the quark, the surface of the cluster, and so forth. We then conclude that the right boundary of the phase coexistence region of the matter is at $\mu _{N,c}^{} \ge {\mu _{G,2}^{}}/3 = 0.673\,$GeV, $\mu _{N,c}^{} \ge 0.597\,$GeV for the $\alpha$-type modified, $\beta$-type modified gluon model, respectively.

3.4 Equation of state and mass-radius relation of compact star

In order to study the hadron-quark phase transition at zero temperature and high chemical potential (density), one has to make use of the compact star observations to check our theoretical results since it is not possible to generate such dense matter on earth.

The most important observable of compact stars is the maximum mass, which is related directly to the EOS $P=P(\varepsilon )$ of the dense matter.

The energy density $\varepsilon$ of the dense matter under Maxwell construction is:

$$\begin{aligned} \varepsilon _{\text {Maxwell}} = \left\{ \begin{array}{ll} \varepsilon _{H}, &{} \quad \text {if}\; \mu _{B}^{}<\mu _{B,c}^{}, \\[2mm] \varepsilon _{Q}, &{} \quad \text {if}\; \mu _{B}^{}>\mu _{B,c}^{}.\\ \end{array}\right. \end{aligned}$$

(31)

$\varepsilon _{H}$ and $\varepsilon _{Q}$ is the energy density of the charge neutral hadron matter and quark matter, respectively. $\mu _{B,c}^{}$ is the phase transition baryon chemical potential. The pressure of the dense matter under Maxwell construction is:

$$\begin{aligned} P_{\text {Maxwell}} = \left\{ \begin{array}{ll} P_{H}^{}, &{} \quad \text {if}\; \mu _{B}<\mu _{B,c}^{}, \\ P_{Q}^{}, &{} \quad \text {if}\; \mu _{B}>\mu _{B,c}^{} . \\ \end{array}\right. \end{aligned}$$

(32)

$P_{H}^{}$ is the pressure of the hadron matter and $P_{Q}^{}$ is the pressure of the quark matter.

For Gibbs construction, the EOS should be decomposed into three parts: $\mu _{B}\le \mu _{G,1}$ region, $\mu _{G,1}\le \mu _{B}\le \mu _{G,2}$ region and $\mu _{B}\ge \mu _{G,2}$ region, where $\mu _{G,1}$ and $\mu _{G,2}$ is the left and right boundary of the phase coexistence region (mixed phase).

The energy density of the mixed phase consists of the contribution of the two phases.

$$\begin{aligned} \varepsilon _{M} = \chi \varepsilon _{Q}(\mu _{B},\mu _{e}) + (1-\chi ) \varepsilon _{H} (\mu _{B},\mu _{e}), \end{aligned}$$

(33)

where the footnote M, Q and H corresponds to the mixed, the quark, the hadron phase, respectively. For a fixed $\mu _{B}$, $\mu _{e}$ is calculated by solving Eqs. (28) and (29). And the pressure for the mixed phase reads:

$$\begin{aligned} p_{M}^{}(\mu _{B}) = p_{H}^{} \left( \mu _{B},\mu _{e}(\mu _{B})\right) = p_{Q}^{} \left( \mu _{B},\mu _{e}(\mu _{B})\right) . \end{aligned}$$

(34)

The energy density and the pressure under the Gibbs construction are:

$$\begin{aligned} \varepsilon _{\text {Gibbs}}= & {} \left\{ \begin{array}{ll} \varepsilon _{H} &{} \quad \text {if}\; \mu _{B}<\mu _{G,1} , \\ \varepsilon _{M}&{} \quad \text {if}\; \mu _{G,1}\le \mu _{B}<\mu _{G,2}, \\ \varepsilon _{Q} &{} \quad \text {if}\; \mu _{B}>\mu _{G,2}, \\ \end{array}\right. \end{aligned}$$

(35)

$$\begin{aligned} P_{\text {Gibbs}}^{}= & {} \left\{ \begin{array}{ll} P_{H}^{} &{} \quad \text {if}\; \mu _{B}<\mu _{G,1}, \\ P_{M}^{} &{}\quad \text {if}\; \mu _{G,1}\le \mu _{B}<\mu _{G,2}, \\ P_{Q}^{} &{} \quad \text {if}\; \mu _{B}>\mu _{G,2} . \\ \end{array}\right. \end{aligned}$$

(36)

The calculated EOSs of the pure hadron, pure quark and hybrid (mixed phase) matter are illustrated in Fig. 8. As we can recognize from Fig. 8, the pure quark matter has a softer EOS than the pure hadron matter, and both the two different construction schemes connect the EOSs of the hadron matter and the quark matter. In case of the Maxwell construction, there is a density region where the pressure is constant, which corresponds to a phase transition with a constant chemical potential. However, in compact stars, the pressure must have a gradient in order to resist its own gravity, so that such a constant pressure region will not appear inside compact star. The phase transition under the Maxwell construction corresponds to a sudden change in the energy density between the quark core and the hadron shell.

Fig. 8

Calculated equation of state. The black solid line is the result of the pure hadron matter via the TW-99 RMF model, the red lines with circle symbol stand for the results using the $\alpha$-type modification, the blue lines with diamond symbol correspond the result implementing the $\beta$-type modification. The solid lines with symbols represent the results of the pure quark matter, the dashed lines with symbols correspond to the results via the Gibbs construction, and the dotted lines with symbols stand for the results with the Maxwell construction

The mass–radius relation of a compact star can be calculated by solving the Tolman–Oppenheimer–Volkov (TOV) equation:

$$\begin{aligned} \frac{\text {d}P}{\text {d}r}=-\frac{G}{r^2}(m(r)+4\pi Pr^3)(\epsilon +P)(1-2Gm(r)/r)^{-1},\nonumber \\ \end{aligned}$$

(37)

where G is gravitational constant and m(r) is the mass inside a radius r:

$$\begin{aligned} m(r)=\int _{0}^{r} 4\pi R^{2} \varepsilon \text {d}R. \end{aligned}$$

(38)

Then given the EOS as input, and with a given center density, we can integrate the TOV equation from inside out to get the mass and radius of the compact star.

For the pure hadron star and the hybrid star, at small density region, we make use of the Baym–Pethick–Sutherland (BPS) EOS [136]. For the pure quark star, we integrate to the surface where the pressure is zero.

The calculated mass-radius relations of the compact star in the cases of our consideration are shown in Fig. 9. From Fig. 9, one can notice apparently that the pure quark star has a much smaller maximum mass than the pure hadron star and hybrid star, and the maximum mass is $1.07M_{\odot }$, $0.95M_{\odot }$ for the $\alpha$-, $\beta$-type modification, respectively.

Fig. 9

Calculated mass-radius relation of the compact star in the cases we take consideration. The black solid line corresponds to the pure hadron star, the solid lines with symbols stand for the results via the Gibbs construction, the dashed lines with symbols represent the results with the Maxwell construction, and the dotted lines with symbols correspond to the results of the pure quark matter. The $\alpha$, $\beta$ in the legend denotes the DS equation result by implementing the $\alpha$-, $\beta$-type modification, respectively

For pure hadron star, the maximum mass is $2.06M_{\odot }$. From Fig. 9, we can see that at large radius and small mass, the mass-radius curves for hybrid star and pure hadron star are exactly the same, because for hybrid star with relatively small mass, the inner density is not large enough for the quark matter to appear.

When the mass of the hybrid star is large enough, the quark matter begins to appear in the composing matter of the star, and the mass–radius curves begin to deviate from the pure hadron one. As can be seen from Fig. 9, for both $\alpha$ and $\beta$ modification, and for both Gibbs and Maxwell construction, the deviation begin to appear at over $\sim 2M_{\odot }$, which means that the hadron-quark phase transition happens at very high density and the mass–radius relation is determined mainly by the EOS of the hadron sector. If the quark sector is described by DS equation with the $\alpha$-type modification, the maximum mass will be $2.06M_{\odot }$ for both Gibbs and Maxwell construction. If the quark sector is described by DS equation with the $\beta$-type modification, the maximum will be $2.00M_{\odot }$ for Gibbs construction and $2.05M_{\odot }$ for Maxwell construction.

4 Summary

The Dyson-Schwinger equation has been implemented to study the phase transition and the phase structure of QCD matter. However, the possible dependence of coupling strength on the quark chemical potential is not well studied. In this paper, we take into account several types of the chemical potential dependence, and make use of the nuclear liquid-gas phase transition to fix the parameter(s).

For the modification factor we propose, the boundary of the phase coexistence region of the iso-symmetric matter is reduced than that without modification. The phase coexistence region is $\mu _{q} \in [0.322 , 0.500]\,$GeV for the Gaussian-type gluon model without any modification, and becomes $\mu _{q} \in [0.271, 0.309]\,$GeV, $[0.276, 0.308]\,$GeV in case of the $\alpha$-, $\beta$-type modification, respectively.

However, by taking into account the $\beta$-equilibrium and charge neutral condition for the matter inside compact stars, and comparing the results from the DS equation calculation for the quark sector and the RMF hadron model, we find that the quark pressure is too low for the Gaussian-type gluon model without any modification, and the hadron-quark phase transition is prohibited, which is physically unlikely. This means that the adoption of modification is necessary.

By implementing the RMF model for hadron matter and the DS equation calculation for the quark sector, we find that the phase transition chemical potential under the Maxwell construction is $\mu _{B}=1.94$, $1.71\,$GeV for the $\alpha$-, $\beta$-type modification, respectively. The phase coexistence region under the Gibbs construction is $\mu _{B} \in [1.69, 2.02]\,$GeV, $[1.48, 1.79]\,$GeV in case of the $\alpha$-, $\beta$-type modification, respectively. The upper boundary of this mixed phase region is not compatible with the iso-symmetric result, and discrepancy comes from the uncertainty of both the DS equation approach and the RMF model in the phase coexistence region. Since the RMF model is more accurate in describing the hadron phase, we conclude that this boundary indicates that $\mu _{N,c}\ge 0.673\,$GeV, $0.579\,$GeV, for the $\alpha$- $\beta$-type modification, respectively, where $\mu _{N,c}^{}$ is the quark chemical potential at which the Nambu solution (DCSB phase) disappears.

We also obtain the mass-radius relation for compact stars in the cases we considered. The maximum mass of the pure hadron star is $2.06M_{\odot }$. If the quark sector is described using $\alpha$-type modification, the maximum mass of the hybrid star is the same as that of pure hadron star. If the quark sector is described by implementing $\beta$-type modification, the maximum mass of the hybrid star is $2.00M_{\odot }$ and $2.05M_{\odot }$ for the Gibbs construction, the Maxwell construction, respectively. Therefore, for hybrid stars, the maximum mass is almost the same since the phase transition happens at very high density. However, the maximum mass of the pure quark star is only about $1M_{\odot }$.

In this work, the possible appearance of hyperons in the dense star matter is not considered, because it prevents from the happening of the hadron-quark phase transition. This problem can be fixed by implementing the 3-window construction method instead of the Gibbs and Maxwell constructions, as has been done in our previous work [127].

Although we have made use of the liquid-gas phase transition chemical potential to fix the parameters of the modification function in our DS equation calculation, it is still impossible to recover all the properties of the hadron matter from first-principle calculation. For example, there should be a density gap at liquid-gas phase transition. Also, the saturation density of nuclear matter cannot be recovered by the Nambu solution of the DS equation. To fix these problems, taking into account the dressing effect of the quark-gluon interaction vertex or/and the interface effect between quarks and hadrons (clustered quarks) in the DS equation approach should be an efficient way. The related work is under progress.

Appendix A: Relativistic mean field theory

The relativistic mean field (RMF) theory [137,138,139,140,141,142,143] is a successful theory which describes the property of the hadron matter in the density region relevant in compact stars [144, 145]. There are many RMF models, with different parameterization which are calibrated by the properties of dense nuclear matter. However, most of the models satisfy only a few experimental data. In Ref. [146], hundreds of RMF models are taken to fit the experimental data, and the TW-99 model [135] is found to be one of the best model that can reproduce most of the nuclear matter properties. Also, TW-99 model generates an EOS that is stiff enough to support a 2-solar-mass neutron star [147], and we have taken such a model to construct a massive hybrid star in our previous work [127].

The Lagrangian of the TW-99 model for the hadron matter is

$$\begin{aligned} {\mathcal {L}}={\mathcal {L}}_{B}^{} + {\mathcal {L}}_{\text {lep}}^{} + {\mathcal {L}}_{M}^{} + {\mathcal {L}}_{\text {int}}^{}, \end{aligned}$$

(A1)

where ${\mathcal {L}}_{B}^{}$ is the Lagrangian of free baryons, which reads

$$\begin{aligned} {\mathcal {L}}_{\text {B}}^{} = \sum _{\text {i}}{\bar{\Psi }_{i}^{}} (i\gamma _{\mu }^{} \partial ^{\mu } - m_{i}^{}){\Psi _{i}^{}}, \end{aligned}$$

(A2)

where $i=p,n$ stands for proton and neutron. If we want to take into consideration the effect of hyperons, we can take $i=p,\,n,\,\Lambda ,\,\Sigma ^{\pm ,0},\,\Xi ^{-,0}$ for the baryon octet.

${\mathcal {L}}_M$ in Eq. (A1) is the Lagrangian of mesons,

$$\begin{aligned} {\mathcal {L}}_{M}^{}= & {} \frac{1}{2}\left( \partial _{\mu }^{} \sigma \partial ^{\mu } \sigma - m_{\sigma }^{2} \sigma ^{2} \right) -\frac{1}{4} \omega _{\mu \nu }^{} \omega ^{\mu \nu } - \frac{1}{2}m_{\omega }^{2} \omega _{\mu }\omega ^{\mu } \nonumber \\&-\frac{1}{4}\varvec{\rho }_{\mu \nu }^{} \varvec{\rho }^{\mu \nu } -\frac{1}{2}m_{\rho }^{2} \varvec{\rho }_{\mu } \varvec{\rho }^{\mu }, \end{aligned}$$

(A3)

where $\sigma$, $\omega _{\mu }^{}$, and $\varvec{\rho }_{\mu }^{}$ are the isoscalar-scalar, isoscalar-vector and isovector-vector meson field, respectively. $\omega _{\mu \nu }^{}=\partial _{\mu }\omega _{\nu } - \partial _{\nu }\omega _{\mu }$, $\varvec{\rho }_{\mu \nu }^{} =\partial _{\mu }\varvec{\rho }_{\nu } - \partial _{\nu }\varvec{\rho }_{\mu }$.

The ${\mathcal {L}}_{\text {int}}^{}$ in Eq. (A1) is the Lagrangian describing the interactions between baryons which are realized by exchanging the mesons:

$$\begin{aligned} {\mathcal {L}}_{\text {int}}^{}= & {} \sum _{B} g_{\sigma B}^{} {\bar{\Psi }_{B}^{}} \sigma {\Psi _{B}^{}} -g_{\omega B}^{} {\bar{\Psi }_{B}^{}} \gamma _{\mu }^{} \omega ^{\mu } {\Psi _{B}^{}} \nonumber \\&- g_{\rho B}^{} {\bar{\Psi }_{B}^{}} \gamma _{\mu }^{} \varvec{\tau }_{B}^{} \cdot \varvec{\rho }^{\mu } {\Psi _{B}^{}}, \end{aligned}$$

(A4)

where $g_{iB}^{}$ for $i=\sigma$, $\omega$, $\rho$ are the coupling strength parameters between baryons and mesons, which depend on the baryon density.

In some other literatures, the self-interaction of $\sigma$-meson, the cross interaction between different kind mesons and the effect of the isovector-scalar $\delta$-meson are included explicitly (see, e.g., the review in Refs. [145, 146]). In TW-99 parameterization, however, all these terms are taken zero, and their effects are represented in the density dependence of the coupling constants. For nucleons, the coupling constants are

$$\begin{aligned} {g_{iN}^{}}(n_{B}^{}) = {g_{iN}^{}} (n_{s}) f_{i}^{} (x), \quad \text {for} \quad i=\sigma ,\omega , \rho , \end{aligned}$$

(A5)

where $n_{B}^{}$ is the baryon density, $n_{s}$ is the saturation nuclear matter density and $x= {n_{B}^{}}/{n_{s}^{}}$. The density function can be chosen as [135]:

$$\begin{aligned} f_{i}^{}(x)= & {} a_{i}^{} \frac{1+b_{i}^{} (x + d_{i}^{})^{2}}{1 + c_{i}^{}(x+d_{i}^{})^{2}}, \quad \text {for} \quad i=\sigma ,\omega , \end{aligned}$$

(A6)

$$\begin{aligned} f_{\rho }^{}(x)= & {} \text {exp}\left[ -a_{\rho }^{} (x-1) \right] , \end{aligned}$$

(A7)

where the parameters $a_{i}^{}$, $b_{i}^{}$, $c_{i}^{}$, $d_{i}^{}$ and $g_{iN}^{}(\rho _{\text {sat}}^{})$ are fixed by fitting the properties of the nuclear matter at the saturation density, and their values are listed in Table 1.

Table 1 Parameters of the mesons and their couplings (taken from Ref. [135])

For hyperons, we represent them with the relation between the hyperon coupling and the nucleon coupling as: $\chi _{\sigma }^{}=\frac{g_{\sigma Y}^{}}{g_{\sigma N}^{}}$, $\chi _{\omega }^{}=\frac{g_{\omega Y}^{}}{g_{\omega N}^{}}$, $\chi _{\rho }^{}=\frac{g_{\rho Y}^{}}{g_{\rho N}^{}}$. On the basis of hypernuclei experimental data, we choose them as those in Refs. [144, 147]: $\chi _{\sigma }^{}=0.7$, $\chi _{\omega }^{}=\chi _{\rho }^{}=0.783$.

The ${\mathcal {L}}_{\text {lep}}^{}$ is the Lagrangian for leptons, which are treated as free fermions:

$$\begin{aligned} {\mathcal {L}}_{\text {lep}}^{}=\sum _{l}^{} {\bar{\Psi }_{l}^{}}(i\gamma _{\mu }^{} \partial ^{\mu } - m_{l}^{}) {\Psi _{l}^{}}, \end{aligned}$$

(A8)

and we include only the electron and muon in this paper.

The field equations can be derived by differentiating the Lagrangian. Under RMF approximation, the system is assumed to be in the static, uniform ground state. The partial derivatives of the mesons all vanish, only the 0-component of the vector meson and the 3rd-component of the isovector meson survive and can be replaced with the corresponding expectation values. The field equations of the mesons are then:

$$\begin{aligned} m_{\sigma }^{2} \sigma= & {} \sum _{B} g_{\sigma B}^{} \langle {\bar{\Psi }_{B}^{}} {\Psi _{B}^{}} \rangle , \end{aligned}$$

(A9)

$$\begin{aligned} m_{\omega }^{2} \omega _{0}^{}= & {} \sum _{B} g_{\omega B}^{} \langle {\bar{\Psi }_{B}^{}} \gamma _{0}^{} {\Psi _{B}^{}} \rangle , \end{aligned}$$

(A10)

$$\begin{aligned} m_{\rho }^{2} \rho _{03}^{}= & {} \sum _{B} g_{\rho B}^{} \langle {\bar{\Psi }_{B}^{}} \gamma _{0}^{} \tau _{3B}^{} {\Psi _{B}^{}} \rangle , \end{aligned}$$

(A11)

where $\tau _{3B}^{}$ is the 3rd-component of the isospin of baryon B.

The equation of motion (EoM) of the baryon is:

$$\begin{aligned} \left[ \gamma ^{\mu } (i\partial _{\mu }^{} - \Sigma _{\mu }^{}) - (m_{B}^{} - g_{\sigma B}^{} \sigma ) \right] {\Psi _{B}^{}} = 0, \end{aligned}$$

(A12)

where

$$\begin{aligned} \Sigma _{\mu }^{} = g_{\omega B}^{} \omega _{\mu } + g_{\rho B}^{} \varvec{\tau }_{B}^{} \cdot \varvec{\rho }_{\mu }^{} + \Sigma _{\mu }^{\text {R}}. \end{aligned}$$

(A13)

The $\Sigma _{\mu }^{\text {R}}$ is called the “rearranging” term, which appears because of the density-dependence of the coupling constant, and reads

$$\begin{aligned} \Sigma _{\mu }^{\text {R}}= & {} \frac{j_{\mu }^{}}{n_{B}} \bigg (\frac{\partial g_{\omega B}^{}}{\partial n_{B}}\bar{\Psi }_{B}^{} \gamma ^{\nu } \Psi _{B}^{} \omega _{\nu }^{} \nonumber \\&+ \frac{\partial g_{\rho B}^{}}{\partial n_{B}}\bar{\Psi }_{B}^{} \gamma ^{\nu } \varvec{\tau }_{B}^{} \cdot \varvec{\rho }_{\nu }^{} \Psi _{B}^{} -\frac{\partial g_{\sigma B}^{}}{\partial n_{B}}\bar{\Psi }_{B}^{} \Psi _{B}^{} \sigma \bigg ),\nonumber \\ \end{aligned}$$

(A14)

where $j_{\mu }^{} = \bar{\Psi }_{B}^{} \gamma _{\mu } \Psi _{B}^{}$ is the baryon current.

Under the EoM of Eq. (A12), the baryons behave as quasi-particles with effective mass

$$\begin{aligned} m_{B}^{*} = m_{B}^{} - g_{\sigma B}^{} \sigma , \end{aligned}$$

(A15)

and effective chemical potential:

$$\begin{aligned} \mu _{B}^{*} = \mu _{B}^{} - g_{\omega B}^{} \omega _{0}^{} - g_{\rho B}^{} \tau _{3B}^{} \rho _{03}^{} -\Sigma _{\mu }^{\text {R}}. \end{aligned}$$

(A16)

One can then get the baryon (number) density:

$$\begin{aligned} n_{B}^{} \equiv \langle {\bar{\Psi }_{B}^{}} \gamma ^{0} {\Psi _{B}^{}} \rangle =\gamma _{B}^{}\int \frac{\text {d}^3k}{(2\pi )^3} = \gamma _{B}^{}\frac{k_{FB}^{3}}{6\pi ^{2}}, \end{aligned}$$

(A17)

where $k_{FB}^{} =\sqrt{\mu _{B}^{*2} - m_{B}^{*2}}$ is the Fermi momentum of the particle, $\gamma _B=2$ is the spin degeneracy. And the scalar density is:

$$\begin{aligned} \rho _{B}^{s} \equiv \langle {\bar{\Psi }_{B}^{}} {\Psi _{B}^{}} \rangle&= \gamma _{B}^{}\int \frac{\text {d}^3k}{(2\pi )^{3}}\frac{m_{B}^{*}}{\sqrt{k^{2} + m_{B}^{*2}}}\nonumber \\&=\gamma _{B}^{}\frac{m_{B}^{*}}{4\pi ^{2}}\bigg [k_{FB}^{} \mu _{i}^{*} - m_{B}^{*2} \text {ln}\bigg (\frac{k_{FB}^{} + \mu _{B}^{*}}{m_{B}^{*}}\bigg )\bigg ].\nonumber \\ \end{aligned}$$

(A18)

The density of a kind of leptons is the same as that for baryons, except that the effective mass and the effective chemical potential should be replaced with the corresponding mass and chemical potential of the lepton.

The matter in the star composed of hadrons should be in $\beta$-equilibrium. Since there are two conservative charge numbers: the baryon number and the electric charge number, all the chemical potential can be expressed with the baryon chemical potential and the electron chemical potential:

$$\begin{aligned} \mu _{i}^{} = B \mu _{B}^{} - Q \mu _{e}^{}, \end{aligned}$$

(A19)

where B and Q is the baryon number, electric charge number for the particle i, respectively. Since neutron has one baryon number and zero charge number, we can take $\mu _{B}=\mu _{n}$ where $\mu _{n}$ is the neutron chemical potential.

Then, combining Eqs. (A9), (A10), (A11), (A14), (A15), (A16), (A17), (A18), (22) and (A19), together with the charge neutral condition:

$$\begin{aligned} n_{p}^{} + n_{\Sigma ^{+}}^{} = n_{e}^{} + n_{\mu ^{-}}^{} + n_{\Sigma ^{-}}^{} +n_{\Xi ^{-}}^{}, \end{aligned}$$

(A20)

one can determine the ingredients and the properties of the hadron matter with any given baryon density $n_{B}^{}$.

The EOS of the hadron matter can be calculated from the energy-momentum tensor:

$$\begin{aligned} T^{\mu \nu } = \sum _{\phi _i} \frac{\partial {\mathcal {L}}}{\partial (\partial _{\mu }\phi _{i}^{})}\partial ^{\nu }\phi _{i}^{} - g^{\mu \nu }{\mathcal {L}}. \end{aligned}$$

(A21)

The energy density $\varepsilon$ is:

$$\begin{aligned} \begin{aligned} \varepsilon =\langle T^{00}\rangle =\sum _{i=B,l}\varepsilon _{i}^{} +\frac{1}{2}m_{\sigma }^{2} \sigma ^{2} +\frac{1}{2}m_{\omega }^{2} \omega _{0}^{2} + \frac{1}{2}m_{\rho }^{2} \rho _{03}^{2}, \end{aligned}\nonumber \\ \end{aligned}$$

(A22)

where the contribution of the baryon B to the energy density is:

$$\begin{aligned} \varepsilon _{B}^{}&= \gamma _{B}^{}\int \frac{\text {d}^3k}{(2\pi )^3}\sqrt{k^{2} +m_{B}^{*2}}\nonumber \\&= \gamma _{B}^{}\frac{1}{4\pi ^{2}}\left[ 2\mu _{B}^{*3} k_{FB}^{} - m_{B}^{*2} \mu _{B}^{*} k_{FB}^{} - m_{B}^{*4}\text {ln}\left( \frac{\mu _{B}^{*} + k_{FB}^{}}{m_{B}^{*}} \right) \right] .\nonumber \\ \end{aligned}$$

(A23)

The contribution of the leptons to the energy density can be written in the similar form as baryons with a spin degeneracy parameter $\gamma _{l}^{} = 2$, except that the effective mass and effective chemical potential should be replaced with those of the leptons, respectively.

As for the pressure of the system, we can determine that with the general formula:

$$\begin{aligned} P =\sum _{i} \mu _{i}^{} \rho _{i}^{} - \varepsilon . \end{aligned}$$

(A24)