1 Introduction

The accelerated expansion of the cosmos has become clearly manifest after the discovery of unexpected reduction in the detected energy fluxes coming from supernovae of type Ia [1, 2]. Other observational data like cosmic microwave background radiations, large scale structures and galaxy red shift surveys [3, 4] also provide evidence in this favor. These observational data led one to propose an enigmatic form of force, dubbed dark energy (DE), which takes part in the expansion phenomenon and dominates overall the energy density of the cosmos. Despite some very solid claims about the existence of DE, its unknown nature is the substantial puzzle in cosmology. The idea of modified gravitational theories is obtained by extending the standard Einstein–Hilbert (EH) action, which has gained much fame in order to demonstrate the secrets of cosmic accelerating expansion.

There exist various theories of modified gravity such as f(R) gravity with R the Ricci scalar, \(f(\mathcal {T})\) gravity in which \(\mathcal {T}\) is the torsion scalar, f(R, T) gravity with T the trace of energy-momentum tensor, f(G) gravity in which G represents the Gauss–Bonnet invariant and f(R, T, Q) gravity (where \(Q=R_{\lambda \sigma } T^{\lambda \sigma }\)) etc. Nojiri and Odintsov [5] reviewed various versions of modified gravity models that could explain DE dominance in this accelerating cosmos. Cognola et al. [6] introduced some viable formulations of f(R) DE models and classified them into four main streams. Nojiri and Odintsov [7] studied some important aspects of f(R) gravity in order to make them well consistent with observational data. Bamba et al. [8] discussed the role of DE, through modified cosmic models, in the expansion of our accelerating cosmos. Durrer and Maartens [9] investigated the idea that some f(R) models could lead to new schemes to test out the credibility of general relativity itself on cosmological scales. Bhatti et al. [10] discussed the dynamical instability of a non-static cylindrical cosmic configuration by using \(f(\mathcal {T})\) gravity and found that additional curvature conditions generate the stability of an expanding stellar frame.

Harko et al. [11] used f(R, T) theory of gravity and presented the corresponding equations of motion for the massive particles through the variational principle in f(R, T) theory. The generalization of f(R, T) gravity is f(R, T, Q) gravity, where \(Q=R_{\lambda \sigma } T^{\lambda \sigma }\) shows the non-minimal coupling between matter and geometry [12]. Haghani et al. [13] obtained the field equations by using a Lagrange multiplier in the f(R, T, Q) theory of gravity. Odintsov and Sáez-Gómez [14] studied f(R, T, Q) gravity with a non-minimal association between matter and gravitational fields and concluded that the ensuing modified gravity contains additional points which would recast the possible cosmological evolution. Elizalde and Vacaru [15] evaluated some exact off-diagonal cosmological models in f(R, T, Q) gravity. Baffou et al. [16] used the perturbation technique and performed a stability analysis with the help of de Sitter and power law models through numerical simulations in f(R, T, Q) gravity.

Gravitational collapse is a fundamental and highly dissipative phenomenon leading to structure formation in our universe. Chandrasekhar [17] studied the dynamical instability of an oscillating spherically symmetric model by using a perfect fluid and found the instability limits in terms of the adiabatic index. Herrera et al. [18] analyzed the dynamical instability of dense stars with zero expansion scalar in a spherically symmetric configuration and found instability limits that are independent of adiabatic index. Cembranos et al. [19] studied gravitational collapse in f(R) gravity and found this phenomenon to possibly work as a key tool to constrain modified gravity models that describe late time cosmological acceleration. Yousaf et al. [20, 21] investigated the irregularity constituents for spherical self-gravitating stars in the presence of an imperfect matter distribution within f(R, T) gravity and found that the complexity of matter increases with the increase of anisotropic stresses. Yousaf [22] explored collapsing spherical models supporting a vacuum core in a \(\Lambda \)-dominated era within the stellar interior.

The subject of exploring the cosmic filamentary celestial objects has been a focus of great attention of many astrophysicists [23,24,25,26,27,28,29]. On a large cosmic scale, it has been analyzed that matter is usually configured to make large filaments. These stellar structures have been found to be very clear characteristics of the interstellar medium. They may give rise to galaxies upon contraction. Motivated by several simulations and observational results, the stability analysis of cosmic filaments with more realistic assumptions has received great interest. Binney and Tremaine [30] have linearized the Vlasov equation about the steady phase of the relativistic interior and solved the resulting eigenvalue equation in order to discuss the dynamical stability of collision-less celestial structures supported by the Vlasov–Poisson formulations. Chavanis [31] has extended their results in the context of non-linear dynamical stability and explored the problem of stability of barotropic as well as collision-less stellar systems via the maximization of a Casimir functional (or H-function) with fixed values of energy and mass. Quillen and Comparetta [32] assumed a constant linear mass density and approximately evaluated a dispersion relation in the background of the tidal galaxy tail.

Myers [33] has discussed the evolution of some observed characteristics of cores and filaments and concluded that during the contraction of host filaments, the core grows in mass and radius, and this phenomenon stops if the surrounding filament gas will no longer exist, making further accretion impossible. Breysse et al. [34] carried out an analytical approach with the detailed perturbation background and investigated the stability of polytropic fluid filaments. They found that the instabilities of the cosmic fluid filaments could be enhanced by introducing a tangential fluid motion of the system. Sharif and Manzoor [35] studied the dynamical instability of the axially symmetric stellar structure with reflection degrees of freedom coupled with locally anisotropic fluid configurations in self-interacting Brans–Dicke gravity and obtained stability conditions through the adiabatic index in both the N and the pN approximations. Birnboim et al. [36] performed a stability analysis in planar, filamentary and spherical infall geometries for the existence of a virialized gas in one-, two- and three-dimensional (3D) gravitational collapse and concluded that cosmic filaments are likely to host halos under some constraints.

Recently, we have investigated the anisotropic spherical collapse in the background of f(R, T, Q) gravity and discussed the stability of compact stars by taking into account the particular viable model with perturbation technique. We also examined that adiabatic index \(\Gamma _1\) has significant role in the dynamical instability of these massive stars [37]. The motivation of this paper is to explain the mathematical as well as physical features of self-gravitating cylindrical celestial objects within the framework of the f(R, T, Q) theory of gravity. Particularly, some properties of viable modified gravity model are discussed to create the expansion and DE consequences in cosmos. This paper is organized as follows: We provide the basic formalism of f(R, T, Q) gravity in Sect. 2. Section 3 deals with the dynamics of cylindrical self-gravitating collapsing model in which formation of field equations and conservation laws by linear perturbation technique and instability constraints at Newtonian (N) and post-Newtonian (pN) limits are investigated. Finally, we conclude our main results in the last section.

2 The formalism of f(R, T, Q) gravity

The formalism of f(R, T, Q) gravity is based on the contribution of non-minimal coupling of geometry and matter. Here R in the EH action is replaced with an arbitrary function of R,  T and \(R_{\gamma \delta }T^{\gamma \delta }\). In [13] modified EH action is demonstrated in the following way:

$$\begin{aligned} I_{f(R,T,Q)}=\frac{1}{2}\int \mathrm{d}^4x\sqrt{-g} [f(R,T,R_{\lambda \sigma }T^{\lambda \sigma })+ L_m], \end{aligned}$$
(1)

where \(L_m\) expresses the relative Lagrangian density of matter distribution then the respective energy-momentum tensor is expressed as

$$\begin{aligned} T_{\lambda \sigma }^{(m)}=-\frac{2}{\sqrt{-g}}\frac{\delta (\sqrt{-g}L_m)}{\delta {g^{\lambda \sigma }}}. \end{aligned}$$
(2)

On varying the modified action Eq. (1), with metric tensor \(g_{\lambda \sigma }\), the following field equations are obtained:

$$\begin{aligned}&-G_{\lambda \sigma }(f_{Q}\mathcal {L}_m - f_R) \nonumber \\&\quad - g_{\lambda \sigma } \left\{ \frac{f}{2}-\Box f_R -\frac{R}{2}f_R-\frac{1}{2}\nabla _\pi \nabla _\rho (f_{Q}T^{\pi \rho }) -\mathcal {L}_mf_T\right\} \nonumber \\&\quad +2f_QR_{\pi (\lambda }T_{\sigma )}^{~\pi }+\frac{1}{2}\Box (f_QT_{\lambda \sigma })-\nabla _\pi \nabla _{(\lambda } [T^\pi _{~\sigma )}f_Q]\nonumber \\&\quad -2\left( f_Tg^{\pi \rho }+f_QR^{\pi \rho }\right) \frac{\partial ^2\mathcal {L}_m}{\partial g^{\lambda \sigma }\partial g^{\pi \rho }}\nonumber \\&\quad - T_{\lambda \sigma }^{(m)}(f_T+\frac{R}{2}f_Q+1)-\nabla _\lambda \nabla _\sigma f_R=0, \end{aligned}$$
(3)

where \(\nabla _\pi \) and \(G_{\lambda \sigma }\) indicates covariant derivative and Einstein tensor, respectively, with \(\Box =g^{\lambda \sigma }\nabla _\lambda \nabla _\sigma \) as a d’Alembert operator. From Eq. (3), one can obtain the expression of trace as in [37]. In the framework of [11] the matter Lagrangian has no specific distinction for perfect fluid, and the corresponding second variation was neglected in their calculations. Equation (3) can be rewritten in GR perspective as follows:

$$\begin{aligned} R_{\lambda \sigma }-\frac{R}{2} g_{\lambda \sigma }=G_{\lambda \sigma }=\overset{~~\text {eff}}{{T}_{\lambda \sigma }}, \end{aligned}$$
(4)

where the effective energy-momentum tensor \({{T}_{\lambda \sigma }}^{\mathrm {eff}}\) has the following form:

$$\begin{aligned} {{T}_{\lambda \sigma }}^{\text {eff}}= & {} \frac{1}{(f_R-f_QL_m)}\bigg [\bigg (f_T+\frac{1}{2}Rf_Q+1\bigg )T^{(m)}_{\lambda \sigma }\nonumber \\&+ \bigg \{\frac{R}{2}\bigg (\frac{f}{R}-f_R\bigg )-L_mf_T-\frac{1}{2}\nonumber \\&\times \,\nabla _{\pi }\nabla _{\rho }(f_QT^{\pi \rho })\bigg \}g_{\lambda \sigma } -\frac{1}{2}\Box (f_QT_{\lambda \sigma })\nonumber \\&-\,(g_{\lambda \sigma }\Box -\nabla _\lambda \nabla _\sigma )f_R-2f_QR_{\pi (\lambda }T^\pi _{~\sigma )}\nonumber \\&+\nabla _\pi \nabla _{(\lambda } [T^\pi _{~\sigma )}f_Q]+2(f_QR^{\pi \rho }+f_Tg^{\pi \rho })\frac{\partial ^2L_m}{\partial g^{\lambda \sigma }\partial g^{\pi \rho }}\bigg ]. \end{aligned}$$

On taking \(Q=0\) in the above equation, f(R, T, Q) gravity would reduce to f(R, T) theory. However, in the case of vacuum it leads to f(R) gravity theory and consequently we will obtain GR results whenever \(f(R)=R\).

3 Anisotropic matter distribution and cylindrical field equations

We consider the three-dimensional (3D) timelike hypersurface, \(\Delta \), that would demarcate the 4D manifold \(\mathcal {W}\) into couple of regions, i.e., exterior \(\mathcal {W}^+\) and interior \(\mathcal {W}^-\). The interior region of relativistic stellar system is given by the following cylindrically symmetric spacetime:

$$\begin{aligned} \mathrm{d}s^2_-=-A^2(t,r)(\mathrm{d}t^{2}-\mathrm{d}r^{2})+B^2(t,r)\mathrm{d}z^{2}+C^2(t,r)\mathrm{d}\phi ^{2}. \end{aligned}$$
(5)

For the representation of cylindrical symmetry, the following ranges are imposed on the coordinates: \(-\infty \le t\le \infty , 0\le r, -\infty<z<\infty , 0\le \phi \le 2\pi .\) We number the respective coordinates \(x^0=t\), \(x^1=r\), \(x^2=z\) and \(x^3=\phi \). We assumed \(C = 0\) at \(r = 0\), which represents a non-singular axis. The spacetime for \(\mathcal {W}^+\) is [38]

$$\begin{aligned} \mathrm{d}s^2_+=-\mathrm{e}^{2(\gamma -\upsilon )}(\mathrm{d}\nu ^2-\mathrm{d}\rho ^2)+\mathrm{e}^{-2\upsilon }\rho ^2\mathrm{d}\phi ^2+\mathrm{e}^{2\upsilon }\mathrm{d}{z}^2, \end{aligned}$$
(6)

where \(\gamma \) and \(\upsilon \) are the functions of \(\nu \) and \(\rho \), while the coordinates are numbered as \(x^{\beta }=(\nu ,\rho ,\phi ,z)\). The corresponding vacuum field equations provide

$$\begin{aligned}&\rho (\upsilon _\nu ^2+\upsilon _\rho ^2)=\frac{\tilde{f}-\tilde{R}\tilde{f}_R}{2\tilde{f}_R}\mathrm{e}^{2(\gamma -\upsilon )},\end{aligned}$$
(7)
$$\begin{aligned}&2\rho \upsilon _\nu \upsilon _\rho =\gamma _\nu ,\end{aligned}$$
(8)
$$\begin{aligned}&\upsilon _{\nu \nu }-\frac{\upsilon _{\rho }}{\rho }-\upsilon _{\rho \rho }=\frac{\mathrm{e}^{2(\gamma -\upsilon )}}{4\rho } \left( \frac{\tilde{f}-\tilde{R}\tilde{f}_R}{\tilde{f}_R}\right) \left\{ \rho \mathrm{e}^{-4\upsilon }+\frac{\mathrm{e}^{2\gamma }}{\rho }\right\} , \end{aligned}$$
(9)

where subscripts \(\rho \) and \(\nu \) show partial differentiations with respect to \(\rho \) and \(\nu \), respectively and tilde indicates that the corresponding values are evaluated with constant R, T and Q conditions. It has been proved by Senovilla [39] that modified extra curvature terms on the boundary surface should be constant. Due to this, we have evaluated the above equations with constant R, T and Q. These equations suggest the existence of a gravitational field. We assume anisotropic and non-dissipative collapsing matter in the cylindrical geometry, whose energy-momentum tensor is

$$\begin{aligned} T_{\lambda \sigma }= & {} (P_{r}+\mu )V_{\lambda }V_{\sigma }+P_{r}g_{\lambda \sigma } -K_\lambda K_\sigma (P_{r}-P_{\phi })\nonumber \\&-S_{\lambda }S_{\sigma }(P_{r}-P_{z}), \end{aligned}$$
(10)

where \(\mu \) is the energy density which is the eigenvalue of \(T_{\lambda \sigma }\) for eigenvector \(V_\lambda \), while \(P_{\phi }, P_z, P_r\) are the principal stresses. The spacetime (5) is the canonical form for cylindrical symmetry, defined as usual by the 2D group that defines the cylindrical symmetry. The unitary vectors \(V_\lambda , L_\lambda , S_\lambda , K_\lambda \) are configured so as to render a canonical orthonormal tetrad in which a hypersurface orthogonal 4-velocity vector is \(V_\lambda \). Further, the two vectors \(S_\lambda \) and \(K_\lambda \) are tangent to the orbits of the 2D group that preserves cylindrical geometry and \(L_\lambda \) is orthogonal to the 4-velocity \(V_\lambda \) and to these orbits. It is worthy to stress that we are considering an Eckart frame where fluid elements are in the state of rest. The four-vectors obey the following relations:

$$\begin{aligned}&V^{\lambda }V_{\lambda }=-1, \quad K^{\lambda }K_{\lambda }=1=S^{\lambda }S_{\lambda }, \nonumber \\&\quad V^{\lambda }K_{\lambda }=V^{\lambda }S_{\lambda }=K^{\lambda }S_{\lambda }=0. \end{aligned}$$
(11)

We choose the fluid to be comoving in a given coordinate system; therefore, we have

$$\begin{aligned} V_{\lambda }=-A\delta ^{0}_{\lambda },\quad K_{\lambda }=C\delta ^{3}_{\lambda },\quad L_{\lambda }=A\delta ^{1}_{\lambda }\quad \mathrm{{and}}\quad S_{\lambda }=B\delta ^{2}_{\lambda }, \end{aligned}$$
(12)

The four-acceleration vector is \(a_\lambda =V_{\lambda ;\sigma }V^\sigma \), with \(a=\frac{A'}{A^2}\) as a scalar associated with the four-acceleration. The expansion scalar, (\(\Theta =V^{\lambda }_{;\lambda }\)), for our cylindrical spacetime leads to

$$\begin{aligned} \Theta =\frac{1}{A}\left( \frac{\dot{A}}{A}+\frac{\dot{B}}{B}+\frac{\dot{C}}{C}\right) , \end{aligned}$$
(13)

where an overdot represents the time derivative. The shear tensor \(\sigma _{\lambda \sigma }\) is

$$\begin{aligned} \sigma _{\lambda \sigma }=V_{(\lambda ;\sigma )}+a_{(\lambda }V_{\sigma )} -\frac{1}{3}\Theta h_{\lambda \sigma }, \end{aligned}$$

where \(h_{\lambda \sigma }\) is a projection tensor with \(h_{\lambda \sigma }=g_{\lambda \sigma }+V_{\lambda }V_{\sigma }\). The shear tensor can also be expressed as follows:

$$\begin{aligned} \sigma _{\lambda \sigma }=\sigma _s\left( S_\lambda S_\sigma -\frac{h_{\lambda \sigma }}{3}\right) +\sigma _k\left( K_\lambda K_\sigma -\frac{h_{\lambda \sigma }}{3}\right) , \end{aligned}$$

where

$$\begin{aligned} \sigma _s=-\frac{1}{A}\left( \frac{\dot{A}}{A}-\frac{\dot{B}}{B}\right) ,\quad \sigma _k=-\frac{1}{A}\left( \frac{\dot{A}}{A}-\frac{\dot{C}}{C}\right) . \end{aligned}$$
(14)

The non-zero modified gravitational field equations for our cylindrical line element associated with matter distribution (10) take the form

$$\begin{aligned}&\frac{1}{A^2}\left[ \frac{\dot{C}\dot{B}}{BC}-\frac{C''}{C}-\frac{B''}{B}-\frac{B'C'}{BC}+\alpha _1\right] =\overset{\text {eff}}{\mu },\end{aligned}$$
(15)
$$\begin{aligned}&\left( \frac{B'}{B}+\frac{C'}{C}\right) \frac{\dot{A}}{A}-\frac{ \dot{C'}}{C}-\frac{\dot{B'}}{B}+\left( \frac{\dot{B}}{B}+\frac{\dot{C}}{C}\right) \frac{A'}{A}=0,\end{aligned}$$
(16)
$$\begin{aligned}&\frac{B'{C'}}{BC}-\frac{\ddot{B}}{B}-\frac{\dot{B}\dot{C}}{BC} -\frac{\ddot{C}}{C}+\alpha _1=\overset{\text {eff}}{P_r},\end{aligned}$$
(17)
$$\begin{aligned}&\left( \frac{B}{A}\right) ^2\left[ \beta _1 +\frac{C''}{C}-\frac{\ddot{C}}{C}\right] =\overset{\text {eff}}{P_z},\quad \left( \frac{C}{A}\right) ^2\left[ \beta _1+\frac{B''}{B}-\frac{\ddot{B}}{B} \right] =\overset{\text {eff}}{P_\phi }, \end{aligned}$$
(18)

where

$$\begin{aligned}&\alpha _1=\left( \frac{\dot{C}}{C}+\frac{\dot{B}}{B}\right) \frac{\dot{A}}{A}+\left( \frac{{B'}}{B}+\frac{{C'}}{C}\right) \frac{{A'}}{A},\nonumber \\&\beta _1=\frac{\dot{A}^2}{A^2}-\frac{A'^2}{A^2}-\frac{\ddot{A}}{A} +\frac{A''}{A},\nonumber \\&\overset{\text {eff}}{\mu }=\frac{1}{f_R-f_Q\mathcal {L}_M} \left[ \mathcal {L}_Mf_T-\frac{1}{2}(f-Rf_R)+\mu \chi _1+\dot{\mu }\chi _2\right. \nonumber \\&\left. +\,\frac{\ddot{\mu }}{2A^2}f_Q+\frac{\mu ''}{2A^2}f_Q\right. +\,\mu '\chi _3+\frac{P''_r}{2A^2}f_Q \end{aligned}$$
(19)
$$\begin{aligned}&\left. +P_r\chi _4+P'_r\left\{ \frac{f_Q'}{A^2}-\frac{5A'}{2A^3}f_Q\right\} \right. \nonumber \\&\left. -\,\frac{f_Q}{2A^2B}(\dot{P_z}\dot{B}+P'_zB')\right. -\,\dot{P_r}\frac{\dot{A}}{A^3}f_Q+P_z\chi _5 \nonumber \\&\left. +P_\phi \chi _6-\frac{f_Q}{2A^2}\left( \dot{P_\phi }\frac{\dot{C}}{C} -{P_\phi '}\frac{C'}{C}\right) \right. \nonumber \\&\left. -\,\frac{\dot{f_R}}{A^2}\left( \frac{\dot{A}}{A}+ \frac{\dot{B}}{B}+\frac{\dot{C}}{C}\right) \right. \nonumber \\&\left. -\,\frac{{f_R'}}{A^2}\left( \frac{{A'}}{A}- \frac{{B'}}{B}-\frac{{C'}}{C}\right) +\frac{f_R''}{A^2}\right] ,\nonumber \\&\overset{\text {eff}}{P_r}=\frac{1}{f_R-f_Q\mathcal {L}_M}\left[ \frac{1}{2}(f-Rf_R) -\mathcal {L}_Mf_T+\frac{\ddot{f_R}}{A^2}\psi _1 \right. \nonumber \\&\left. +\,\dot{\mu } \left( \frac{5\dot{A}}{2A^3}f_Q-\frac{\dot{f_Q}}{A^2}\right) \right. \nonumber \\&\left. -\,f_R'\psi _2+P_r\chi _7-\frac{\ddot{\mu }}{2A^2}f_Q+\mu \chi _8+\frac{\mu 'A'}{2A^3}f_Q \right. \nonumber \\&\left. -\,\frac{\ddot{P_r}}{2A^2}f_Q+P_z\chi _9+P_\phi \chi _{10}\right. \nonumber \\&\left. +\,\frac{f_Q}{2A^2}\left\{ \dot{P_z}\frac{\dot{B}}{B}-P_z'\frac{B'}{B} +\dot{P_\phi }\frac{\dot{C}}{C}-P_\phi '\frac{C'}{C}\right\} +\dot{P_r}\chi _{11} +P_r'\chi _{12} \right] ,\nonumber \\&\overset{\text {eff}}{P_z}=\frac{1}{f_R-f_Q\mathcal {L}_M}\nonumber \\&\times \,\left[ \frac{1}{2}(f\!-\!Rf_R)\!-\!\mathcal {L}_Mf_T\!+\!\dot{\mu } \chi _{14}\!+\!\mu \chi _{13}\!-\!\frac{\ddot{\mu }}{2A^2}f_Q\!+\!\frac{\mu 'A'}{2A^3}f_Q\right. \end{aligned}$$
(20)
$$\begin{aligned}&\left. +\, P_r\chi _{15}+\frac{\dot{A}\dot{P_r}}{2A^3}f_Q+P_r'\left( \frac{5A'}{2A^3}f_Q-\frac{f'_Q}{A^2}\right) \right. \nonumber \\&\left. -\,\frac{P''_r}{2A^2}f_Q+P_z\chi _{16}+P_z'\chi _{17}-\dot{P_z}\right. \nonumber \\&\left. \times \,\chi _{18}-\frac{\ddot{P_z}}{2A^2}f_Q +\frac{P''_z}{2A^2}f_Q+P_{\phi }\chi _{19}\right. \nonumber \\&\left. +\,\frac{f_Q}{2A^2}\left( \dot{P_{\phi }}\frac{\dot{C}}{C} -P_{\phi }'\frac{C'}{C}\right) +\psi _3 \right] , \end{aligned}$$
(21)
$$\begin{aligned}&\overset{\text {eff}}{P_\phi } =\frac{1}{f_R-f_Q\mathcal {L}_M} \nonumber \\&\times \,\bigg [\frac{1}{2}(f-Rf_R)-\mathcal {L}_Mf_T+\dot{\mu } \chi _{14}+\mu \chi _{13}-\frac{\ddot{\mu }}{2A^2}f_Q+\frac{\mu 'A'}{2A^3}f_Q\nonumber \\&+\,P_r\chi _{15}+\frac{\dot{A}\dot{P_r}}{2A^3}f_Q+P_r'\bigg (\frac{5A'}{2A^3}f_Q-\frac{f'_Q}{A^2}\bigg )\nonumber \\&-\,\frac{P''_r}{2A^2}f_Q+P_z\chi _{20}+P_\phi '\chi _{23}+\dot{P_\phi }\nonumber \\&\times \,\chi _{22}-\frac{\ddot{P_\phi }}{2A^2}f_Q +\frac{P''_\phi }{2A^2}f_Q+P_{\phi }\chi _{21}+\frac{f_Q}{2A^2}\bigg (\dot{P_{z}}\frac{\dot{B}}{B} -P_{z}'\frac{B'}{B}\bigg )+\psi _4\bigg ],\nonumber \\ \end{aligned}$$
(22)

where a prime stands for the \(\frac{\partial }{\partial r}\) operator and the quantities \(\chi _i\) contain combinations of metric variables and their derivatives as mentioned in the appendix. The value of R for the spacetime is given as

$$\begin{aligned} R= & {} \frac{2}{A^2}\left[ \left( \frac{\ddot{A}}{A}+\frac{\ddot{B}}{B}+\frac{\ddot{C}}{C}\right) -\left( \frac{A''}{A}+\frac{B''}{B}+\frac{C''}{C}\right) +\frac{1}{A^2}(A'^2-\dot{A}^2)\right. \nonumber \\&\left. +\,\frac{1}{BC}(\dot{B}\dot{C} -B'C')\right] . \end{aligned}$$
(23)

3.1 Viability of f(R, T, Q) model and junction conditions

In this subsection, we shall deal with the hydrodynamics of cylindrical stellar collapse by using dynamical equations. The expression of covariant derivative of effective energy-momentum tensor is

$$\begin{aligned}&\nabla ^\lambda T_{\lambda \sigma }=\frac{2}{Rf_Q+2f_T+1}\nonumber \\&\times \,\bigg [\nabla _\sigma (\mathcal {L}_mf_T) +\nabla _\sigma (f_QR^{\pi \lambda }T_{\pi \sigma })-\frac{1}{2}(f_Tg_{\pi \rho }+f_QR_{\pi \rho })\nonumber \\&\times \,\nabla _\sigma T^{\pi \rho }-G_{\lambda \sigma }\nabla ^\lambda (f_Q\mathcal {L}_m)\bigg ], \end{aligned}$$
(24)

which would lead to two equations of motion in f(R, T, Q) theory. Making use of \(G^{\lambda \sigma }_{;\sigma }=0\) and Eqs. (15)–(18) along with \(\lambda =0,1\), the above equation gives

$$\begin{aligned}&\frac{\overset{~~\text {eff}}{\dot{\mu }}}{A}+\Theta \left[ \overset{~~\text {eff}}{\mu } +\frac{1}{3}\left( \overset{~~\text {eff}}{P_{r}}+\overset{~~\text {eff}}{P_{z}} +\overset{~~\text {eff}}{P_{\phi }}\right) \right] +\frac{1}{3}\left( \overset{~~\text {eff}}{P_{z}} -\overset{~~\text {eff}}{P_{r}}\right) (2\sigma _s-\sigma _k)\nonumber \\&\quad +\,\frac{1}{3}\left( \overset{~~\text {eff}}{P_{\phi }} -\overset{~~\text {eff}}{P_{r}}\right) (2\sigma _k-\sigma _s)+Z_1=0,\end{aligned}$$
(25)
$$\begin{aligned}&\nabla \overset{~~\text {eff}}{P_{r}}-\frac{1}{A}\left[ \left( \overset{~~\text {eff}}{P_{z}} -\overset{~~\text {eff}}{P_{r}}\right) \frac{B'}{B}+\left( \overset{~~\text {eff}}{P_{\phi }} -\overset{~~\text {eff}}{P_{r}}\right) \frac{C'}{C}\right] +\left( \overset{~~\text {eff}}{\mu } -\overset{~~\text {eff}}{P_{r}}\right) \nonumber \\&\quad \times \, a+Z_2=0, \end{aligned}$$
(26)

where the superscript “eff” indicates the presence of f(R, T, Q) terms in the matter variables and the expressions of \(Z_1\) and \(Z_2\) are mentioned in the appendix in Eqs. (A1) and (A1). The quantities \(Z_1\) and \(Z_2\) are due to the non-conserved divergence of the energy-momentum tensor. The dynamical equations could help to explain the hydrodynamics of locally anisotropic cylindrical relativistic massive bodies. It is worthy to mention that the theoretically designed stellar models are of importance if they are stable against instabilities and fluctuations. Now, we will explain the dynamic instability of anisotropic and non-dissipative relativistic cylindrical geometry by using a particular f(R, T, Q) model [40],

$$\begin{aligned} f(R,T,Q)=\alpha R^2+\beta Q, \end{aligned}$$
(27)

where \(\alpha \) and \(\beta \) are constants. The model with \(\alpha R^n+\beta Q^m\) is the generalization of the above-mentioned f(R, T, Q) model, in which m and n are constants. In order to deal with this theory free from Ostrogradski instabilities, one should take \(n\ne 1\). However, this model will generate a stable theory for \(m=1\), by giving the EH term including the canonical scalar field having a non-minimal variation coupling of the Einstein tensor. The model with \(n=2\) and \(m=1\) along with constant \(\beta \) could help to understand the dynamics and evolution of the inflationary cosmos. For the particular value of the constant \(\alpha \), i.e., \(\alpha =\frac{1}{6M^2}\) [41] with \(M=2.7\times 10^{-12}\) GeV, this model behaves as a substitute of DM. In the case of \(\alpha =0\), there is a geometry–matter association due to the coupling between the stress-energy tensor and the Ricci scalar. Yousaf et al. [37] studied this model with \(n=2, m=1\) and discussed the stability of compact stars in an anisotropic spherical configuration by taking \(\beta >0\) along with \(\alpha =\frac{1}{6M^2}\).

For the smooth matching of Eqs. (5) and (6) over \(\Delta \), we shall use junction conditions proposed by Darmois [42] as well as Senovilla [39] for f(R, T, Q) theory. Since we have assumed a timelike hypersurface, we impose \(r=\)constant in Eq. (5) and \(\rho (\nu )\) in the exterior metric (10). In this framework, the first fundamental form yields

$$\begin{aligned}&\mathrm{d}\tau \overset{\Delta }{=}\mathrm{e}^{2\gamma -2\upsilon }\left\{ 1-\left( \frac{\mathrm{d}\rho }{\mathrm{d}\nu }\right) ^2\right\} ^{1/2}\mathrm{d}\nu =A\mathrm{d}t,\end{aligned}$$
(28)
$$\begin{aligned}&B\overset{\Delta }{=}\mathrm{e}^{\upsilon },\quad C\overset{\Delta }{=}\mathrm{e}^{-\upsilon }\rho , \end{aligned}$$
(29)

with \(1-\big (\frac{\mathrm{d}\rho }{\mathrm{d}\nu }\big )^2>0\). Here, the notation overset \(\Delta \) indicates that the corresponding equations and quantities are evaluated on the hypersurface, \(\Delta \). The second fundamental form yields

$$\begin{aligned}&\mathrm{e}^{2\gamma -2\upsilon }[\nu _{\tau \tau }\rho _{\tau }-\rho _{\tau \tau }\nu _{\tau } -\{\nu _\tau (\gamma _\rho -\upsilon _\rho )+\rho _\tau (\gamma _\nu -\upsilon _\nu )\}\nonumber \\&\quad \times \, (\nu _\tau ^2-\rho _\tau ^2)] \overset{\Delta }{=}\frac{-A'}{A^2},\end{aligned}$$
(30)
$$\begin{aligned}&\mathrm{e}^{2\upsilon }(\rho _\tau \upsilon _{\nu }+\nu _\tau \upsilon _\rho )\overset{\Delta }{=}\frac{BB'}{A},\nonumber \\&\quad \mathrm{e}^{-2\upsilon }\rho ^2\left( \rho _\tau \upsilon _\nu +\nu _\tau \upsilon _\rho -\frac{\nu _\tau }{\rho }\right) \overset{\Delta }{=}\frac{-CC'}{A}. \end{aligned}$$
(31)

By making use of Eqs. (28)–(31), the field equations and after some manipulations, we obtain

$$\begin{aligned} \overset{\text {eff}}{P_r}\overset{\Delta }{=}0. \end{aligned}$$
(32)

From Eq. (4), one can write the following form:

$$\begin{aligned}&G_{\lambda \sigma }=\frac{1}{(f_R-f_QL_m)}\bigg [\bigg (f_T+\frac{1}{2}Rf_Q+1\bigg )T^{(m)}_{\lambda \sigma }\nonumber \\&\quad +\, \bigg \{\frac{R}{2}\bigg (\frac{f}{R}-f_R\bigg )-L_mf_T-\frac{1}{2} \times \nabla _{\pi }\nabla _{\rho }(f_QT^{\pi \rho })\bigg \}g_{\lambda \sigma } \\ \nonumber&\quad -\,\frac{1}{2}\Box (f_QT_{\lambda \sigma }) +\nabla _\lambda \nabla _\sigma f_R+g^{\pi \nu }\nabla _\pi \nabla _{\nu }(T_{\lambda \sigma }f_Q)\\ \nonumber&\quad -\, g_{\lambda \sigma }\Box f_R+2f_QR T_{\lambda \sigma }\bigg ], \end{aligned}$$

which can be transformed to

$$\begin{aligned}&\Omega _{\lambda \sigma }=\frac{1}{\bigg (1+f_T+\frac{5}{2}R f_Q\bigg )}\bigg [(f_R-f_QL_m)G_{\lambda \sigma }\nonumber \\&\quad -\,\frac{1}{2}(f-Rf_R)g_{\lambda \sigma }+L_mf_Tg_{\lambda \sigma } +\frac{1}{2}\nonumber \\&\quad \times \,\nabla _{\pi }\nabla _{\rho }(f_QT^{\pi \rho })\bigg \}g_{\lambda \sigma }\nonumber \\&\quad -\,\frac{1}{2}\Box (f_QT_{\lambda \sigma }) -\nabla _\lambda \nabla _\sigma f_R+g_{\lambda \sigma }\Box f_R\bigg ], \end{aligned}$$
(33)

where \(\Omega _{\lambda \sigma }\) indicates a tensor associated with bulk matter. In a Gaussian normal coordinate system, we have

$$\begin{aligned} \mathrm{d}s^2=\mathrm{d}y^2+\gamma _{ab}\mathrm{d}x^a\mathrm{d}x^b, \end{aligned}$$

in which the boundary surface is at \(y=0\). In this context, the Ricci scalar takes the form

$$\begin{aligned} R=2\partial _yK-\frac{4}{3}K^2-K^*_{ab}K^{*ab}-\tilde{R}, \end{aligned}$$
(34)

where \(K_{ab}\) is the extrinsic curvature at the hypersurface, tilde shows the constant choice of the Ricci scalar evaluated through induced spacetime, while \(K^*_{ab}\) and K are the traceless and trace components of the extrinsic curvature, respectively. The value of the extrinsic curvature can be expressed through \(\gamma _{ab}\) as \(K_{ab}=-1/2\times \partial _y\gamma _{ab}\). The Einstein tensor yields

$$\begin{aligned} G_{yy}= & {} -\frac{1}{2}(K_{\lambda \sigma }K^{\lambda \sigma }+\tilde{R}-K^2),\nonumber \\ G_{y\sigma }= & {} -\nabla _\nu (K^\nu _{~\sigma }-\delta ^\nu _{~\sigma } K),\nonumber \\ \nonumber G_{\lambda \sigma }= & {} \partial _y(K_{\lambda \sigma }-K\gamma _{\lambda \sigma })+\frac{1}{2}\gamma _{\lambda \sigma }(K_{\mu \nu }K^{\mu \nu }+K^2) \\&+\tilde{G}_{\lambda \sigma } -\,3KK_{\lambda \sigma }+2K^\nu _{\lambda }K_{\nu \sigma }. \end{aligned}$$

Now, we split Eq. (33) into two tensorial quantities,

$$\begin{aligned} \Omega _{\lambda \sigma }=Q_{\lambda \sigma }+L_{\lambda \sigma }, \end{aligned}$$
(35)

where

$$\begin{aligned} Q_{\lambda \sigma }&=(f_R-f_QL_m)G_{\lambda \sigma }+L_mf_Tg_{\lambda \sigma }-\frac{1}{2}(f-Rf_R)g_{\lambda \sigma },\end{aligned}$$
(36)
$$\begin{aligned} L_{\lambda \sigma }&=\frac{1}{2}\nabla _\mu \nabla _\nu (f_QT^{\mu \nu })g_{\lambda \sigma }-\frac{1}{2}\Box (f_QT_{\lambda \sigma }) -\nabla _\lambda \nabla _\sigma f_R+g_{\lambda \sigma }\Box f_R. \end{aligned}$$
(37)

The components of Eq. (36) are obtained as follows:

$$\begin{aligned} Q_{yy}&=G_{yy}f_R-f_QL_mG_{yy}+\frac{1}{2}(Rf_R-f),\\ Q_{y\beta }&=f_RG_{y\beta }-f_QL_mG_{y\beta },\\ Q_{\alpha \beta }&=f_RG_{\alpha \beta }-f_QL_mG_{\alpha \beta }+L_m\gamma _{\alpha \beta }f_T-\frac{1}{2}(f-Rf_R)\gamma _{\alpha \beta }, \end{aligned}$$

while Eq. (37) provides us with the following relations:

$$\begin{aligned} \nonumber L_{yy}= & {} -K\partial _y f_R+\tilde{\Box } f_R -\frac{1}{2} \tilde{\Box }(f_QT_{\alpha \beta })+\frac{K}{2}\partial _y(f_QT^{\alpha \beta }),\nonumber \\ \nonumber L_{y\beta }= & {} -\partial _\beta \partial _y f_R-K^\mu _{\beta }\partial _\mu f_R-\frac{1}{2}\tilde{\Box }(f_QT_{y\beta }),\nonumber \\ L_{\alpha \beta }= & {} -\tilde{\nabla }_{\alpha \beta }+K_{\alpha \beta }\partial _y f_R+\frac{1}{2}\gamma _{\alpha \beta } [\tilde{\nabla }_{\mu \nu }(f_QT^{\mu \nu })-K_{\mu \nu }\partial _y(f_QT^{\mu \nu })],\nonumber \\ \nonumber&-\,\frac{1}{2}\tilde{\Box }(f_QT_{\alpha \beta })+\gamma _{\alpha \beta }[\tilde{\Box }f_R+\partial _{yy}f_R-K\partial _yf_R]. \end{aligned}$$

Now, we compute the ya and yy components of Eq. (35), which after some simplifications give rise to

$$\begin{aligned} \partial _y[(K_{\lambda \sigma }-K\gamma _{\lambda \sigma })f_R+\gamma _{\lambda \sigma }f_{QQ}\partial _y Q+\gamma _{\lambda \sigma }f_{RR} \partial _y R]=0. \end{aligned}$$
(38)

Upon integration across the hypersurface, Eq. (38) yields

$$\begin{aligned}{}[(K_{\lambda \sigma }-K\gamma _{\lambda \sigma })f_R+\gamma _{\lambda \sigma }f_{QQ}\partial _y Q+\gamma _{\lambda \sigma }f_{RR} \partial _y R]|_-^+=0. \end{aligned}$$
(39)

The integration of Eq. (34) gives \(R|_-^+=0\), while the trace and traceless components of Eq. (39) give rise to

$$\begin{aligned}&f_{,RR}[\partial _y R|_-^+=0,\quad f_{,RR}K^*_{\lambda \sigma }|_-^+=0,\quad f_{,QQ}[\partial _y Q|_-^+=0,\nonumber \\&\quad K|_-^+=0, \end{aligned}$$
(40)

along with

$$\begin{aligned} R|_-^+=0,\quad Q|_-^+=0,\quad \gamma _{\lambda \sigma }|_-^+=0, \end{aligned}$$
(41)

provided the matching conditions for the f(R, T, Q) theory of gravity in which \(f_{,RR}\ne 0\) and \(f_{,QQ}\ne 0\) should be satisfied. The details of this approach in f(R) gravity have been mentioned in [39, 43, 44]. Equation (32) arises due to the Darmois junction conditions, which indicates that the effective radial pressure on \(\Delta \) is zero. Equations (40) and (41) over \(\Sigma \) are required for the continuity of R and Q invariants even for matter in thin shells.

3.2 Perturbation scheme

In order to discuss the stability of cylindrical celestial objects, we shall explore the perturbed form of field as well as the dynamical equations in this section. A perturbation deals with small variations in a physical system resulting by gravitational effects of other stellar objects. Therefore, in the recent few decades, researchers were very keen to analyze the stability of the cosmic stellar filaments against oscillatory motion induced by perturbations. Here, we use the linear perturbation scheme with a very small perturbation parameter \(\epsilon \) so that one can neglect its second and higher powers. Initially, the celestial system is considered to be in hydrostatic equilibrium, but as time passes, it is subject to oscillatory motion. All the metric functions and fluid parameters can be perturbed [18],

$$\begin{aligned} A(t,r)&=A_o(r)+{\epsilon }\omega (t)a(r),\quad \mu (t,r)=\mu _o(r)+\epsilon {\bar{\mu }}(t,r),\nonumber \\ B(t,r)&=B_o(r)+{\epsilon }\omega (t)b(r),\quad P_r(t,r)=P_{ro}(r)+\epsilon {\bar{P_r}}(t,r),\nonumber \\ C(t,r)&=C_o(r)+{\epsilon }\omega (t)c(r),\quad P_\phi (t,r)=P_{\phi o}(r)+\epsilon {\bar{P_\phi }}(t,r),\nonumber \\ R(t,r)&=R_o(r)+{\epsilon }\omega (t)d(r),\quad P_z(t,r)=P_{zo}(r)+\epsilon {\bar{P_z}}(t,r). \end{aligned}$$
(42)

By using the above perturbation technique along with the junction conditions (32), (40) and (41), Eq. (17) can be executed in terms of a second order partial differential equation,

$$\begin{aligned} \ddot{\omega }-\chi ^2\omega \overset{\Delta }{=}0, \end{aligned}$$
(43)

where

$$\begin{aligned} \nonumber \chi ^2= & {} \bigg [ \bigg (\frac{B_o'}{B_o}+\frac{C_o'}{C_o}\bigg )\bigg (\frac{a}{A_o}\bigg )'+ \bigg (\frac{A_o'}{A_o}+\frac{C_o'}{C_o}\bigg )\bigg (\frac{b}{B_o}\bigg )'\nonumber \\&+\bigg (\frac{B_o'}{B_o}+\frac{A_o'}{A_o}\bigg )\bigg (\frac{c}{C_o}\bigg )'\bigg ] \times \bigg (\frac{b}{B_o}+\frac{c}{C_o}\bigg )^{-1}. \end{aligned}$$

The most general solution of the above equation is given by

$$\begin{aligned} \omega (t)=\mathfrak {c}_1\exp ({\chi }t)+\mathfrak {c}_2\exp (-{\chi }t), \end{aligned}$$
(44)

where \(\mathfrak {c}_1\) and \(\mathfrak {c}_2\) are arbitrary constants. Equation (44) indicates two solutions that are independent from each other. Here, we wish to explore unstable regimes of a collapsing stellar anisotropic system. Due to this, we consider our stellar filament to be in static equilibrium at large past time, i.e., \(\omega (-\infty )=0\); then with the passage of time it enters into the present state and goes forward in the phase of gravitational collapse by decreasing its areal radius. Such a model could be achieved only by taking \(\mathfrak {c}_1=-1\) along with \(\mathfrak {c}_2=0\). This would describe the monotonically decreasing configuration of the solution as time passes.

The most general solution of Eq. (43) includes oscillating and non-oscillating functions that correspond to stable and unstable configurations of stellar anisotropic filament, respectively. The choice \(\mathfrak {c}_1=+1\) is exactly equivalent to the case if one absorbs the sign in a, b, c and d. Our aim is to explore instability regimes of collapsing stellar interiors; therefore, we have to restrict our perturbations a, b, c and d on the boundary surface as positive definite in order to make \(\chi ^2>0\). (This assumption has been made by a number of astrophysicists [18, 35, 45,46,47,48,49,50] to discuss unstable limits of collapsing stellar populations.) The required solution associated with Eq. (43) can be achieved by taking \(\mathfrak {c}_1=-1\) and \(\mathfrak {c}_2=0\) as

$$\begin{aligned} \omega (t)\overset{\Delta }{=}-\exp {({\chi }t)}. \end{aligned}$$
(45)

The perturbed configuration of f(R, T, Q) model is

$$\begin{aligned} f=[\alpha R_o^2+\beta Q_o]+\epsilon 2\alpha \omega (t) d(r)R_o, \end{aligned}$$
(46)

where

$$\begin{aligned} R_o=-\frac{2}{A_o}\left[ \frac{A_o''}{A_o}+\frac{B_o''}{B_o} +\frac{C_o''}{C_o}-\frac{A_o'^2}{A_o}+\frac{B_o'C_o'}{B_oC_o}\right] . \end{aligned}$$

By using above perturbation scheme, the static forms of f(R, T, Q) field equations are

$$\begin{aligned}&G_{00}^{(S)}=\frac{1}{2\alpha R_o+\beta \mu _o}\nonumber \\&\quad \times \,\bigg [\mu _o\chi _{1o}+\mu _o'\chi _{3o}+P_{ro}\chi _{4o} +P_{zo}\chi _{5o}+P_{\phi o}\chi _{6o}\nonumber \\&\quad +\frac{\alpha }{2} \bigg (\frac{4R_o''}{A_o^2}+R_o^2-4R_o'\psi _{2o}\bigg )+\frac{\beta }{2A_o^2}\nonumber \\&\quad \times \,\bigg (\mu _o''+P_{ro}''+P_{zo}'\frac{B'_o}{B_o}+P_{\phi o}'\frac{C'_o}{C_o}-5P_{ro}'\frac{A'_o}{A_o}\bigg )-\frac{\beta }{2}Q_o\bigg ],\end{aligned}$$
(47)
$$\begin{aligned}&G_{11}^{(S)}=\frac{1}{2\alpha R_o+\beta \mu _o}\nonumber \\&\quad \times \,\bigg [\mu _o\chi _{8o}+P_{ro}'\chi _{12o}+P_{ro}\chi _{7o} +P_{zo}\chi _{9o}+P_{\phi o}\chi _{10o}\nonumber \\&\quad -\,\frac{\alpha }{2} (R_o^2\!-\!4R_o'\psi _{2o})\!+\!\frac{\beta }{2A_o^2} \bigg (\mu _{o}'\frac{A'_o}{2A_o}\!-\!P_{zo}'\frac{B'_o}{B_o}-P_{\phi o}'\frac{C'_o}{C_o}\bigg )+\frac{\beta }{2}Q_o\bigg ],\nonumber \\ \end{aligned}$$
(48)
$$\begin{aligned}&G_{22}^{(S)}=\frac{1}{2\alpha R_o+\beta \mu _o}\nonumber \\&\quad \times \,\bigg [\frac{1}{2}(\beta Q_o-\alpha R_o^2)+\mu _o\chi _{13o}+P_{ro}\chi _{15o}+P_{zo}\chi _{16o} +P_{zo}'\chi _{17o}\nonumber \\&\quad +\,P_{\phi o}\chi _{14o}+\psi _{3o}+\frac{\beta }{2A_o^2}\nonumber \\&\quad \times \,\bigg (\mu _{o}'\frac{A'_o}{A_o}+5P_{ro}'\frac{A_o'}{A_o}-P_{ro}''+P_{zo}'' -P_{\phi o}'\frac{C'_o}{C_o}\bigg )\bigg ], \end{aligned}$$
(49)

where the superscript (S) indicates the static form of the Einstein tensors. Their expressions are given in the appendix as Eqs. (A2)–(A4). However, the perturbed configurations of these equations are

$$\begin{aligned}&{\bar{G}_{00}}=\frac{1}{2\alpha R_o+\beta \mu _o}\bigg [\omega (\alpha d R_o+\mu _ox_1+P_{ro}x_4\nonumber \\&\quad +\,P_{zo}x_5+P_{\phi o}x_6+\mu _o'x_3)+\bar{\mu }\chi _{1o}\nonumber \\&\quad +\,\dot{\mu }\chi _{2o}+\bar{\mu }'\chi _{3o}+\bar{P_r}\chi _{4o} +\bar{P_z}\chi _{5o}+\bar{P_\phi }\chi _{6o}+\frac{\beta }{2A_o}\nonumber \\&\quad \times \,\bigg ( \ddot{\bar{\mu }}+\bar{\mu }''+\bar{P_r}''-5\bar{P_r}'\frac{A_o'}{A_o}\nonumber \\&\quad +\,\bar{P_\phi }'\frac{C_o'}{C_o}+\bar{P_z}'\frac{B_o'}{B_o}\bigg ) -\frac{\omega \beta }{A_o^2} \nonumber \\&\quad \times \,\bigg (\frac{a\mu _o''}{A_o}+\frac{aP_{ro}''}{A_o} +\frac{bP'_{zo}B'_o}{2B_o^2}+\frac{aP'_{zo}B'_o}{A_oB_o} +\frac{bP'_{zo}}{2B_o}\nonumber \\&\quad +\,P_{\phi o}'\frac{c'}{2C_o}+\frac{cP_{\phi o}C_o'}{2C_oz}+\frac{aP_{\phi o}C_o'}{2A_oC_o}\bigg )+5\omega P_{ro}'\frac{\beta }{2}\bigg (\frac{b}{A_o^3}\bigg )'\nonumber \\&\quad +\,2\alpha \omega \bigg (\frac{d''}{A_o^2}-\frac{2R''_o}{A_o^3}+R'_oy_2+d'\psi _{2o} \bigg )\nonumber \\&\quad -\,2\alpha \dot{\omega }d\psi _{1o}\bigg ]-\frac{2\alpha \omega d +\beta \bar{\mu }}{2\alpha R_o+\beta \mu _o}\overset{\text {eff}}{{\mu }_o}, \end{aligned}$$
(50)
$$\begin{aligned}&{\bar{G}_{11}}=\frac{1}{2\alpha R_o+\beta \mu _o}\bigg [\omega (P_{ro}x_7-\alpha d R_o+\mu _ox_8\nonumber \\&\quad +\, P_{zo}x_9+P_{\phi o}x_{10}+P_{r o}'x_{12} \nonumber \\&\quad -\,2\alpha R'_oy_2)+2\alpha d\frac{\ddot{\omega }}{A_o^2}-\frac{\beta }{2A_o^2}\nonumber \\&\quad \times \,\bigg (\ddot{\bar{P_r}}+\ddot{\bar{\mu }} -\bar{\mu }\frac{A_o'}{A_o}+\bar{P_\phi }'\frac{C_o'}{C_o} +\bar{P_z}'\frac{B_o'}{B_o}\bigg )+\bar{P_r}\nonumber \\&\quad \times \, \chi _{7o}+\bar{\mu }\chi _{8o}+\bar{P_z}\chi _{9o} +\bar{P_\phi }\chi _{10o}+\dot{\bar{P_r}}\chi _{11o}+\bar{P_r}'\chi _{12o}\nonumber \\&\quad +\,\frac{\omega \beta }{2A_o^2} \bigg (\frac{\mu _o'a'}{A_o}+2a\nonumber \\&\quad \times \,\frac{P_{zo}'B_o'}{A_oB_o}-\frac{3\mu _o'a}{A_o^2} +\frac{bP_{zo}'B_o'}{B_o^2}-\frac{b'P_{zo}'}{B_o} +\frac{cP_{\phi o}'C_o'}{C_o^2}\bigg )\bigg ]\nonumber \\&\quad -\,\frac{2\alpha \omega d +\beta \bar{\mu }}{2\alpha R_o+\beta \mu _o}\overset{~~\text {eff}}{{P}_{zo}}, \end{aligned}$$
(51)
$$\begin{aligned}&{\bar{G}_{22}}=\frac{1}{2\alpha R_o+\beta \mu _o}\bigg [\omega (\mu _ox_{13}-\alpha R_od+P_{ro}x_{15}+P_{zo}x_{16}\nonumber \\&\quad +\,P_{zo}'x_{17}+P_{\phi o}x_{19} +y_3) \nonumber \\&\quad +\,\bar{\mu }\chi _{13o}+\dot{\bar{\mu }}\chi _{14o}+\bar{P_r} \chi _{15o}+\bar{P_z}\chi _{16o}\nonumber \\&\quad +\,\bar{P_z}'\chi _{17o} -\bar{{P_z}}\chi _{18o}+\bar{P_\phi }\chi _{19o} \nonumber \\&\quad +\,\frac{\beta }{2A_o^2}\bigg (\bar{\mu }'\frac{A_o'}{A_o}-\ddot{\bar{\mu }} +\bar{P_r}'\frac{A_o'}{A_o}-\bar{P_r}''-\ddot{\bar{P_z}}+\bar{P_z}'' -\bar{P_\phi }'\frac{C_o'}{C_o}\bigg )\nonumber \\&\quad +\,\frac{\omega \beta }{2A_o^2} \bigg \{\frac{\mu _o'a'}{A_o} -3\mu _o'a\frac{A_o'}{A_o^2} +5P_{ro}'\frac{a'}{A_o}\nonumber \\&\quad -\,15P_{ro}'a\frac{A_o'}{A_o^2} +\frac{aP_{ro}''}{A_o}-\frac{aP_{zo}''}{A_o}\nonumber \\&\quad +\,P_{\phi o}'\frac{C_o'}{C_o}\bigg (\frac{c}{C_o}+\frac{2}{A_o}-1\bigg )\bigg \}\bigg ]-\frac{2\alpha \omega d +\beta \bar{\mu }}{2\alpha R_o+\beta \mu _o}\overset{~~\text {eff}}{{P}_{zo}}, \end{aligned}$$
(52)
$$\begin{aligned}&{\bar{G}_{33}}=\frac{1}{2\alpha R_o+\beta \mu _o}\bigg [\omega (\mu _ox_{13}-\alpha R_od+P_{ro}x_{15}\nonumber \\&\quad +\,P_{zo}x_{20}+P_{zo}'x_{23}+P_{\phi o}x_{21}+y_4) \nonumber \\&\quad -\,\frac{\beta }{2A_o^2}\bigg (\frac{\mu _o'a'}{A_o}-\frac{3\mu _o'a}{A_o^2} +\frac{5a'P_{ro}'}{A_o}\nonumber \\&\quad -\, 15P_{ro}'\frac{a}{A_o^2}+\bar{P_r}''+2aP_{ro}''+bP_{zo}'\nonumber \\&\quad \times \, \frac{B_o'}{B_o} +aP_{zo'}\frac{B_o'}{A_oB_o}-\frac{b'P_{zo}'}{B_o}-\frac{2aP_{zo}''}{A_0}\bigg )\nonumber \\&\quad +\, \bar{\mu }\chi _{13o}+\dot{\bar{\mu }}\chi _{14o}+\bar{P_r} \chi _{15o}+\bar{P_z}\chi _{20o}\nonumber \\&\quad +\,\bar{P_\phi }\chi _{21o}+ \dot{\bar{P_\phi }}\chi _{22o}+\bar{P_\phi }'\chi _{23o}\bigg ]-\frac{2\alpha \omega d +\beta \bar{\mu }}{2\alpha R_o+\beta \mu _o}\overset{~~\text {eff}}{{P}_{\phi o}}, \end{aligned}$$
(53)

where the overbar shows the perturbed form of the Einstein tensors and they are written in the appendix as Eqs. (A5)–(A8). In the case of hydrostatic equilibrium, the second dynamical equation has the following form:

$$\begin{aligned}&\frac{1}{A_o}\overset{~~\text {eff}}{{P_{ro}}'}+\frac{A_o'}{A_o^2} \left( \overset{~~\text {eff}}{\mu _{o}}+\overset{~~\text {eff}}{P_{ro}}\right) +\frac{B_o'}{A_oB_o} \left( \overset{~~\text {eff}}{P_{ro}}-\overset{~~\text {eff}}{P_{zo}}\right) \nonumber \\&\quad +\frac{C_o'}{A_oC_o} \left( \overset{~~\text {eff}}{P_{ro}}-\overset{~~\text {eff}}{P_{\phi o}}\right) +Z_{2o}=0, \end{aligned}$$
(54)

while their non-static forms are

$$\begin{aligned}&\overset{~~\text {eff}}{\dot{\bar{\mu }}}+\dot{\omega }\eta =0,\end{aligned}$$
(55)
$$\begin{aligned}&\frac{1}{A_o}\left[ \overset{~~\text {eff}}{\bar{P_{r}}'}+\frac{\omega a}{A_o}\left\{ \overset{~~\text {eff}}{P_{ro}'}+\frac{B_o'}{B_o} \left( \overset{~~\text {eff}}{P_{zo}}-\overset{~~\text {eff}}{P_{ro}}\right) \right. \right. \nonumber \\&\left. \left. +\frac{C_o'}{C_o} \left( \overset{~~\text {eff}}{P_{\phi o}}-\overset{~~\text {eff}}{P_{ro}}\right) + \left( \overset{~~\text {eff}}{\mu _{o}}+\overset{~~\text {eff}}{P_{ro}}\right) \right. \right. \nonumber \\&\nonumber \left. \left. \times \left( \frac{a'}{a}-\frac{2A_o'}{A_o}\right) \right\} +\omega \left\{ \left( \overset{~~\text {eff}}{P_{ro}}-\overset{\text {eff}}{P_{zo}}\right) \left( \frac{b}{B_o}\right) '\right. \right. \nonumber \\&\left. \left. +\left( \overset{~~\text {eff}}{P_{ro}} -\overset{~~\text {eff}}{P_{\phi o}}\right) \left( \frac{c}{C_o}\right) '-\frac{B_o'}{B_o}\right. \right. \nonumber \\&\left. \left. \times \left( \overset{~~\text {eff}}{\bar{P_{z}}}-\overset{~~\text {eff}}{\bar{P_{r}}}\right) -\left( \overset{~~\text {eff}}{\bar{P_{\phi }}}+\overset{~~\text {eff}}{\bar{P_{r}}}\right) \frac{C_o'}{C_o}\right. \right. \nonumber \\&\left. \left. +\left( \overset{~~\text {eff}}{\bar{\mu }}+\overset{~~\text {eff}}{\bar{P_{r}}}\right) \frac{A_o'}{A_o}\right\} \right] +\omega \bar{Z_2}=0, \end{aligned}$$
(56)

where

$$\begin{aligned} \nonumber \eta =\overset{~~\text {eff}}{\mu _o}\left( \frac{a}{A_o}+\frac{b}{B_o}+\frac{c}{C_o}\right) +\overset{~~\text {eff}}{P_{ro}}\left( \frac{a}{A_o}+\frac{b}{B_o}\right) +\frac{c}{C_o}\overset{~~\text {eff}}{P_{\phi o}}+ A_o\bar{Z_1}. \end{aligned}$$

Against a non-static environment, the scalar variables associated with expansion and shear tensors are found as follows:

$$\begin{aligned}&\bar{\Theta }=\frac{\dot{\omega }}{A_o}\left( \frac{a}{A_o}+\frac{b}{B_o}+\frac{c}{C_o}\right) ,\quad \bar{\sigma _s}=\frac{\dot{\omega }}{A_o}\left( \frac{b}{B_o}-\frac{a}{A_o}\right) \\&\bar{\sigma _k}=\frac{\dot{\omega }}{A_o}\left( \frac{c}{C_o}-\frac{a}{A_o}\right) . \end{aligned}$$

3.3 Stability analysis

Here, we want to discuss the stability of cylindrical anisotropic compact objects in terms of the stiffness parameter \(\Gamma _1\). The Harrison–Wheeler equation of state [51] has a great impact in this context; it forms a relationship between the pressure components and energy density given as

$$\begin{aligned} \bar{P_i}=\bar{\mu }\frac{P_{i0}}{\mu _0+P_{i0}}\Gamma _1. \end{aligned}$$
(57)

Then Eq. (55) can be rewritten as follows:

$$\begin{aligned} \overset{~~\text {eff}}{\dot{\bar{\mu }}}=-\dot{\omega }\eta . \end{aligned}$$

The integration of this equation gives

$$\begin{aligned} \overset{~~\text {eff}}{\bar{\mu }}=-\omega \eta . \end{aligned}$$
(58)

Using the above value of \(\bar{\mu }^{~~\text {eff}}\) in Eq. (57), we obtain

$$\begin{aligned}&\overset{~~\text {eff}}{\bar{P}_{r}}=-\Gamma _1 \frac{\overset{~~\text {eff}}{P_{ro}}\eta \omega }{(\overset{~~\text {eff}}{\mu _{o}}+\overset{~~\text {eff}}{P_{ro}})},\quad \overset{~~\text {eff}}{\bar{P}_{z}}=-\Gamma _1 \frac{\overset{~~\text {eff}}{P_{z o}}\eta \omega }{(\overset{~~\text {eff}}{\mu _{o}}+\overset{~~\text {eff}}{P_{zo}})},\nonumber \\&\overset{~~\text {eff}}{\bar{P}_{\phi }}=-\Gamma _1 \frac{\overset{~~\text {eff}}{P_{\phi o}}\eta \omega }{(\overset{~~\text {eff}}{\mu _{o}}+\overset{~~\text {eff}}{P_{\phi o}})}. \end{aligned}$$
(59)

Substituting the values from Eqs. (58) and (59) in Eq. (54), the corresponding modified collapse equation turns out to be

$$\begin{aligned}&\Gamma _1\left[ \frac{\overset{~~\text {eff}}{P_{ro}}}{\overset{~~\text {eff}}{\mu _{o}}+\overset{~~\text {eff}}{P_{ro}}} \bigg \{\eta \left( \frac{B_o'}{B_o}+\frac{C'_o}{C_o}-\frac{A_o'}{A_o}\right) -\eta '\right. \nonumber \\&\left. +\frac{\eta }{(\overset{~~\text {eff}}{\mu _{o}}+\overset{~~\text {eff}}{P_{ro}})} (\overset{~~\text {eff}}{\mu _{o}}'+\overset{~~\text {eff}}{P_{ro}}')\bigg \} -\eta \right. \nonumber \\&\quad \times \,\left. \frac{\overset{~~\text {eff}}{P_{ro}}'}{\overset{~~\text {eff}}{\mu _{o}}+\overset{~~\text {eff}}{P_{ro}}}+\eta \frac{B_o'}{B_o} \frac{\overset{~~\text {eff}}{P_{zo}}}{\overset{~~\text {eff}}{\mu _{o}}+\overset{~~\text {eff}}{P_{zo}}}+\eta \frac{C_o'}{C_o} \frac{\overset{~~\text {eff}}{P_{\phi o}}}{\overset{~~\text {eff}}{\mu _{o}}+\overset{~~\text {eff}}{P_{\phi o}}}\right] =-\frac{a}{A_o}\overset{~~\text {eff}}{P_{ro}}'\nonumber \\ \nonumber&\quad +\,\overset{~~\text {eff}}{P_{ro}} \left\{ \frac{aB_o'}{A_oB_o}-\frac{C_o'}{C_o}+\frac{a'}{a}-\frac{2A'_o}{A_o} -\left( \frac{b}{B_o}\right) '-\left( \frac{c}{C_o}\right) '\right\} \nonumber \\&\quad +\,\overset{~~\text {eff}}{P_{zo}} \left\{ \left( \frac{b}{B_o}\right) '-\frac{aB_o'}{A_oB_o}\right\} +\overset{~~\text {eff}}{P_{\phi o}}\nonumber \\&\quad \times \, \left\{ \left( \frac{c}{C_o}\right) '-\frac{C_o'}{C_o}\right\} +\overset{~~\text {eff}}{\mu _{o}}\left( \frac{a'}{a}-\frac{2A'_o}{A_o}\right) +\frac{\eta A'_o}{A_o}+A_o\bar{Z_2}. \end{aligned}$$
(60)

In a given equation, the terms including the adiabatic index \(\Gamma _1\) would generate pressure and counter gravitational effects, while the remaining terms work as the generator of the gravity force. The effects, produced by principal stresses and f(R, T, Q) gravity terms intervened by the fluid have great relevance in the analysis of gravity forces.

3.3.1 N approximations

Here, we compute the instability for the cylindrical interior system in the N limit with the theory of gravity induced by the \(\alpha R^2+\beta Q\) model. In the N regime, we shall consider a flat background metric, which leads to weak field approximations. Therefore, we take

$$\begin{aligned} A_0=1, \quad B_0=1. \end{aligned}$$

Since we are dealing with the compact configurations of cosmic stellar filament, we may assume that the energy density of the matter content is much greater than the pressure components. Due to this, we shall consider the following constraint in our calculation in the N limit:

$$\begin{aligned} \mu _0\gg P_{i0}. \end{aligned}$$

It was demonstrated by Chandrasekhar [17] and Herrera et al. [18] that all the terms coming in the stability conditions should be positive definite. Therefore, to attain the instability regions of the cylindrical stellar system, we consider each term in the respective collapse equation to be positive. The collapse equation (60) takes the form

$$\begin{aligned} \left[ \overset{~~\text {eff}}{\mu _{o}}\left( a+b+\frac{c}{C_o}\right) +\bar{Z_1}\right] \Gamma _1=\overset{~~\text {eff}}{\mu _{o}}(a'/a)+\Pi +\bar{Z_2}, \end{aligned}$$

where

$$\begin{aligned} \Pi= & {} b'\left( \overset{~~\text {eff}}{P_{zo}}-\overset{~~\text {eff}}{P_{ro}}\right) +\left( \overset{~~\text {eff}}{P_{\phi o}}+\overset{~~\text {eff}}{P_{ro}}\right) \frac{C'_o}{C_o}\nonumber \\&-\, \left( \frac{c}{C_o}\right) ' \left( \overset{~~\text {eff}}{P_{\phi o}}+\overset{~~\text {eff}}{P_{ro}}\right) -a\overset{~~\text {eff}}{P_{ro}}' \end{aligned}$$
(61)

includes anisotropic effects for the onset of instability regimes in cylindrical compact objects.

Now, we recall the work of Chandrasekhar [17], who checked the collapsing behavior of a perfect spherical star with the help of the numerical value of \(\Gamma _1\). He found three possibilities as regards the N limits of the star. These are:

  1. 1.

    The effects of the star weight will be stronger than pressure, once the system satisfies \(\Gamma _1<4/3\) condition. This would eventually lead the body to enter into collapse state.

  2. 2.

    The initial compression would lead the system towards hydrostatic equilibrium if \(\Gamma _1=4/3\).

  3. 3.

    Further, the limit \(\Gamma _1>4/3\) indicates that the influence of pressure on the stellar dynamics is much greater than the star’s weight, thereby increasing the resulting outward force. Then the system will move towards equilibrium and is said to be dynamically stable.

Keeping in mind the same analysis for f(R, T, Q) theory of gravity, the evolving cylindrical anisotropic stellar object will be in the phase of hydrostatic equilibrium whenever it satisfies

$$\begin{aligned} \Gamma _1=\frac{|\overset{~~\text {eff}}{\mu _{o}}(a'/a)+\Pi +\bar{Z_2}|}{|\overset{~~\text {eff}}{\mu _{o}}\left( a+b+\frac{c}{C_o}\right) +\bar{Z_1}|}. \end{aligned}$$
(62)

If the effects of \(|\overset{~~\text {eff}}{\mu _{o}}(a'/a)+\Pi +\bar{Z_2}|\) and \(|\overset{~~\text {eff}}{\mu _{o}}\left( a+b+\frac{c}{C_o}\right) +\bar{Z_1}|\) are equal, then

$$\begin{aligned} \Gamma _1=1 \end{aligned}$$
(63)

will give us the condition of hydrostatic equilibrium for the cylindrically symmetric anisotropic interiors. However, if the role of \(|\overset{~~\text {eff}}{\mu _{o}}(a'/a)+\Pi +\bar{Z_2}|\) is less significant than \(|\overset{~~\text {eff}}{\mu _{o}}\big (a+b+\frac{c}{C_o}\big )+\bar{Z_1}|\), then the relation

$$\begin{aligned} \Gamma _1<\frac{|\overset{~~\text {eff}}{\mu _{o}}(a'/a)+\Pi +\bar{Z_2}|}{|\overset{~~\text {eff}}{\mu _{o}}\left( a+b+\frac{c}{C_o}\right) +\bar{Z_1}|} \end{aligned}$$
(64)

shows that the given system is in an unstable region and the range of the adiabatic index would be (0, 1). If the modified gravity forces generated by \(|\overset{~~\text {eff}}{\mu _{o}}(a'/a)+\Pi +\bar{Z_2}|\) are higher than that of \(|\overset{~~\text {eff}}{\mu _{o}}\big (a+b+\frac{c}{C_o}\big )+\bar{Z_1}|\), then this will make the system enter the stable window. This means that the forces of anti-gravity and principal stresses produce the stability constraint in the N region:

$$\begin{aligned} \Gamma _1>1. \end{aligned}$$

This state is said to be dynamically stable.

3.3.2 pN approximations

In order to attain the pN instability constraints, we consider \(A_o(r)=1-\phi , B_o(r)=1+\phi \) with effects up to \(O(\phi )\), where \(\phi (r)=\frac{m_0}{r}\). In this context, the collapse equation (60) yields the following value of \(\Gamma _1\):

$$\begin{aligned} \Gamma _1=\frac{F_\mathrm{pN}}{E_\mathrm{pN}}, \end{aligned}$$
(65)

where

$$\begin{aligned}&F_\mathrm{pN}=\frac{\overset{~~\text {eff}}{P_{ro}}}{\overset{~~\text {eff}}{\mu _{o}}+\overset{~~\text {eff}}{P_{ro}}}\\&\quad \times \,\left[ -\eta '_\mathrm{pN}+\eta _\mathrm{pN}\left\{ \frac{C'_o}{C_o}+\frac{\overset{~~\text {eff}}{\mu _{o}'}}{\overset{~~\text {eff}}{\mu _{o}}+\overset{~~\text {eff}}{P_{ro}}}+\phi '(1-\phi )\right. \right. \\&\quad \left. \left. \times \,\frac{\overset{~~\text {eff}}{P_{zo}}}{\overset{~~\text {eff}}{\mu _{o}}+\overset{~~\text {eff}}{P_{zo}}}+2\phi ' +\frac{C'_o}{C_o} \frac{\overset{~~\text {eff}}{P_{\phi o}}}{\overset{~~\text {eff}}{\mu _{o}}+\overset{~~\text {eff}}{P_{\phi o}}}\right\} \right] ,\\&E_\mathrm{pN}=-a(1+\phi )\overset{~~\text {eff}}{P_{ro}}'+S_1 (\overset{~~\text {eff}}{P_{ro}}-\overset{~~\text {eff}}{P_{zo}})\nonumber \\&\quad +\,S_2(\overset{~~\text {eff}}{P_{ro}}-\overset{~~\text {eff}}{P_{\phi o}})+S_3(\overset{~~\text {eff}}{\mu _{o}}+\overset{~~\text {eff}}{P_{ro}}),\nonumber \\&\quad -\,\phi '(1+\phi )\eta _\mathrm{pN}+(1-\phi )\bar{Z_2}. \end{aligned}$$

The anisotropic cosmic filament will enter into the window of stable configurations, once the modified gravity forces generated by \(F_\mathrm{pN}\) are greater than that of \(E_\mathrm{pN}\). In that case, the stability of the relativistic system is ensured by the following pN limit:

$$\begin{aligned} \Gamma _1>\frac{F_\mathrm{pN}}{E_\mathrm{pN}}. \end{aligned}$$
(66)

However, if during evolution the system attains the state at which \(F_\mathrm{pN}=E_\mathrm{pN}\), then the system will cease to be in the regime of equilibrium. At that time, the cylindrical system will no longer be in the evolutionary phases. One can deal with such a situation by considering Eq. (65). The constraint for instability can be entertained by the anisotropic cylindrical compact system, if the impact of \(F_\mathrm{pN}\) is less than \(E_\mathrm{pN}\). This would give

$$\begin{aligned} \Gamma _1<\frac{F_\mathrm{pN}}{E_\mathrm{pN}}. \end{aligned}$$
(67)

This pN instability limit depends upon the contribution of principal stresses and counter gravity terms related with \(\Gamma _1\) and f(R, T, Q) gravity. This also indicates the significance of hydrostatic equilibrium factors in the study of dynamical unstable regimes of our system.

4 Concluding remarks

In the framework of modified gravity, the stability problem of massive objects has appeared to be a main concern in relativistic astrophysics. In this paper, we have analyzed the instability ranges of a self-gravitating cylindrical collapsing model in the f(R, T, Q) gravity structure. We have investigated the field equations for cylindrical symmetric spacetime for an anisotropic and non-dissipative matter distribution. In this aspect, the dynamical equations are developed by using the contraction of the Bianchi identities. The perturbed profile of the field, the dynamical equations and the kinematical quantities are evaluated by imposing the perturbation scheme on the matter and geometric variables.

Initially, we have assumed that our cylindrical system is in hydrostatic equilibrium. However, as time passes, it goes into the oscillating phase. Therefore, the resulting equations are applied to construction of the collapse equation, which is further analyzed in the N and pN limits. Against this background, the adiabatic index assisted by the equation of state has been used to quantify the stiffness of the matter composition. Also, we have considered a feasible model of f(R, T, Q) theory and examined its impact in the dynamical evolution of locally anisotropic celestial system. It is noticed that additional curvature terms appear because of the modification in the gravity model, which are the major cause of obstacles in evolving celestial objects. Consequently, the evolving cosmic filament systems are more stable due to their non-attractive behavior.

It is noted that, for the stability of isotropic spherical relativistic bodies, the particular numerical value of the stiffness parameter, i.e., \(\frac{4}{3}\), was calculated by Chandrasekhar [17]. Since then, many astrophysicists have tried to examine the instability regimes for various celestial geometries. We have observed the critical role of the adiabatic index in the description of unstable/stable regimes. We also examined how \(\Gamma _1\) depends upon the static configuration of geometry and matter as well as on the additional terms which appear due to matter–curvature coupling. It is noted that the system will remain unstable whenever it sticks to the range as specified in Eqs. (64) and (67) for the N and pN limits, respectively. When the system is unable to remain in the above-mentioned ranges, it will enter the stable or equilibrium phase. It should be noted that in the absence of non-minimal coupling of matter and geometry, these results boil down to f(R, T) outcomes. However, in the case of vacuum, one can get the result of f(R) gravity.