1 Introduction

In the recent years, papers aimed at analyzing the chaotic structures of oil prices and financial series have increased in parallel with advancements in time series modeling techniques and the significant improvements in computer technologies. In the literature focusing on the financial markets, although many macroeconomic variables are assumed to exhibit irregular behaviors, these are accepted as stochastic, the variables are linearized by economists in addition to focusing on the utilization of linear techniques which make the inference relatively easier. However, chaos theory which explains irregular behaviors of the world suggests a more realistic approach to modeling complex patterns of macroeconomic variables.

Chaotic and/or nonlinear behavior of macroeconomic variables depends on many factors such as economic recessions (1970s’ first and second oil crises, the 2008 financial crisis), wars, socioeconomic and political fluctuations, geopolitical events, external debt problems, asymmetric reactions to the foreign exchange market developments, alterations in banking activities, market structures that deviate from perfect markets including oligopolistic behavior in the refining and distribution of oil, lags in the production in response to the changing world demand in addition to the expectations of the financial investors.

With respect to the above-mentioned discussion, the volatility of oil prices, stock returns and the expectations of investors are very significant subjects because of the sensitivity of the investment decisions to economic shocks. In particular, the price movements of oil as a strategic commodity lead to significant fluctuations in both the financial and capital markets. Hamilton (1996) has a primary paper on this field, examining the nonlinear structures of oil prices by testing them in terms of regime switching by Markov switching methods. Barone-Adesi et al. (1998) recommended a semi-parametric method to analyze the behavior of oil prices. Adrangi et al. (2001) found the existence of low-dimensional chaotic structures. He (2011), Komijani et al. (2014), Lahmiri (2017) and Bildirici and Sonustun (2018) are other investigators who explored the existence of chaos in crude oil markets. On the other hand, Pindyck (1991) found the volatilities of oil price tend to increase the uncertainty in future equity prices. Wang et al. (2013) examined the short-run and long-run connections between oil price and stock return, in addition to the other variables for USA, Germany, Japan, Chinese Taiwan and Mainland. Lee and Chang (2011) investigated the nexus between gold and oil prices, stock returns and other financial variables for Japan.

The expectations of investors are one of the other important research areas in the economics and finance literature since the concept of the animal spirits was put forth by Keynes (1936). Sharpe (1964) developed the capital asset pricing method (CAPM) which focuses on a single index of returns linking the individual security to the return of a common index. In addition, the CAPM, developed by Sharpe (1964) and Lintner (1965), relates the return/index relation to the investors’ expectations framework. Brenner and Galai (1989) further enhanced the idea to include the derivatives in order to manage the financial risk. Brenner and Galai (1989) and Whaley (1993) are further studies generalizing the notion to the derivative volatility. In this respect, the volatility index (VIX) Futures were developed in 1993 by using the data of Chicago Board Options Exchange and it was modified in 2003 to capture investor sentiments. As proven by Whaley (2000), VIX is a promising economic indicator that is evaluated by many investors and policy makers representing the investors’ sentiment based on the expectations of the behavior of the market participants and therefore it is taken as an important indicator of the future stock market risk.

This paper aims at the investigation of chaos on the selected financial variables. One important contribution of the paper is questioning whether there was any chaotic behavior in the selected variables by using Lyapunov exponent tests and Kolmogorov entropy and whether there exists any co-movement between the oil prices, the stock returns of the Istanbul Stock Exchange and the expectations of investors. For this purpose, the TAR-TR-GARCH and TAR-TR-TGARCH copula methods are developed and utilized to capture new information regarding the behavior of the relations between the analyzed variables. The methodology which will be investigated in Sect. 3 is capable of determining the chaotic behavior, the change in the dependence structure between oil price volatilities and the validity of the nonlinear relation between the expectations of the investors and the stock returns. Further, the generalization of the copula methodology to TAR-TR-GARCH models is fruitful in terms of capturing asymmetric copula parameters in addition to the investigation of tail dependence coefficients. The method also allows the researcher to investigate whether the size of the tail dependence parameters becomes, for instance, larger after the change in volatility of the analyzed variables.

The selection of augmenting the nonlinear volatility models with chaos theory is based on the following. Chaotic behavior means high sensitivity to initial conditions (Viana and Barbosa 2005). Small differences in initial conditions give important outcomes in chaotic behavior, since very small differentiation in initial conditions (Wernecke et al. 2017) leads to enormous differences between expected and realized values in the long term. If chaotic behavior has a nonlinear dynamic structure, the chaotic structure has uncommon complex absorbents and they show sudden structural breaks in their orbits. TAR-TR-GARCH and TAR-TR-TGARCH copula models will be used to analyze conditional heteroscedasticity and excess kurtosis and to trace co-movement among the variables. The central benefit of the copulas lies in separating the dependency structure from the marginal distributions of the variables without constructing any assumptions about the marginal distribution (Boubaker and Sghaier 2016a, b). The employment of copula-dependent models became common practice in financial literature to handle the asymmetric dependency structures between random variables. The key feature of these methods is the possibility of including the problem of classifying the joint distribution into two components such as marginal densities and dependence structure (Filho et al. 2012). These methods contain determination of the functional form of the marginal distributions and secondly, the identification of the adequate copula function that characterizes the dependence between the variables. On the other hand, some papers employed the BEKK GARCH and DCC-GARCH techniques. The DCC-GARCH method permits time-varying conditional correlation. But these structures do not allow reproduction of the nonlinear dependence occurring between the variables and, moreover, emergence of the evidence on tail dependence.

In this condition, it is aimed to determine the chaotic behavior among the analyzed variables via three different methods: Lyapunov exponent, BDS and Kolmogorov entropy, and then the co-movement between the selected variables is tested via TAR-TR-GARCH and TAR-TR-TGARCH copula methods obtained by combining TAR-TR-GARCH and TAR-TR-TGARCH methods, developed by Bildirici et al. (2017), with the copula method. As a result, it will be possible to analyze the presence of complex dynamics and, moreover, to capture the asymmetric effects of negative and positive shocks and to capture different states below and above a certain threshold by running these methods. The TAR-TR-GARCH and TAR-TR-TGARCH copula models allow one to determine both a conditional mean due to regimes given by threshold parameters and a conditional variance. The contribution of this paper is the simultaneous implementation of chaotic and copula methods on the selected variables.

The instructional development of the Istanbul Stock Exchange is given in the second section. The literature is given in the third section. Data and econometric approach are identified in the fourth section. The empirical results are given in the fifth section. The conclusions and policy implications are presented in the last section.

2 Institutional specifications of the Istanbul Stock Exchange

In Turkey, financial markets were strictly regulated in 1980s. This process covered the deregulation of interest rates, the establishment of financial markets including the Istanbul Stock Exchange (ISE), and the liberalization of the exchange rate regime. Today, ISE is evaluated in the context of sector and sub-sector indices, ISE National All Shares Index, ISE New Economy Market Index, ISE Investment Trusts Index, ISE Second National Market Index, ISE-30, ISE-50 and ISE-100. Since the ISE-100 Index includes both the ISE-50 and ISE-30 Indexes it is evaluated as a main economic indicator of financial markets in a number of papers. The ISE-100 Index is a market capitalization-weighted index that indicates at least 80% of the total market capitalization and the traded value.

The Istanbul Stock Exchange developed rapidly with reference to its market capitalization. For example, compared to the index consisting of 80 stocks in 1986, the annual volume of trade surpassed $181.9 billion in 2000 (Odabaşı et al. 2004). In 2002, the average daily trading value was recorded as 287 stocks with the value of US$753 million. The Istanbul Stock Exchange applies the multiple price continuous auction method with no market makers (Bildik and Yilmaz 2008). The ISE-100 Index is one of the best performing indices. Index returns in US$ were calculated as 21% compared to the emerging markets and MSCI developed markets indices’ with yearly returns of 16% and 10% in 2010, respectively.

ISE with a market capitalization valued at US$ 308 billion ranks fourteenth among the emerging markets. The total amount of funds elevated by the ISE from its establishment in 1986 to the end of 2010 was US$48.6  billion (ISE 2010). In 2014, the total traded volume of Istanbul markets was recorded as 7.9 trillion Turkish lira (TL) which was increased by 13.7 percent from 9 trillion TL of the previous year (BIST 2014). In 2016, the total trading volume exceeded 13 trillion TL, with a greater than 9% increase in 2016 compared to 2015. In these years, risk perception directed at emerging markets was different than the previous year’s given the fact that the increasing trade volumes and especially the foreign investors’ share gained an important value. In 2014, 517 bonds were issued, along with 66.4 billion TL of shared capital. In contrast, 621 and 766 bonds were issued, along with 79.2 and 90.8 billion TL of capital in 2015 and 2016, respectively (BIST 2016).

Though the Istanbul Stock Exchange is growing rapidly, the market capitalization of ISE is much smaller than that of the developed markets. With this effect, the Istanbul Stock Exchange is exposed to speculative activities, manipulations and government interventions much more than the developed markets. Moreover, in light of the differentiated investor perceptions, the ISE is exposed to deal with different macroeconomic variables distinctly from any other developed market.

3 Literature

The financial markets literature survey on the volatility of oil prices could be considered as accepting the mainstream consensus that, similar to the stock returns, oil prices follow a Gaussian normal distribution and additionally, the price behavior follows the ‘random walk’ hypothesis (RWH) developed by Bachelier (1900). A second finding is that the general acceptance of the efficient market hypothesis, suggested by Fama (1970), could not be disregarded. However, these approaches are widely criticized by some papers in the financial literature including Alvarez-Ramirez and Rodriguez (2008), He and Chen (2010), He et al. ((2007, 2009) and He and Zheng (2008). Further, a few papers showed that the returns in financial markets and oil price follow different behaviors. These features can be given in four subtitles: (1) fat tails (Plerou et al. 2001); (2) fractals/multifractals (Panas and Ninni 2000; Tabak and Cajueiro 2007; He and Chen 2010; He et al. 2007, 2009; He and Zheng 2008); (3) chaos (Adrangi et al. 2001); and lastly, (4) nonlinearity (Hamilton 2003; Zhang 2008; Lardic and Mignon 2006; Cologni and Manera 2009).

The papers that analyze the chaotic behavior used generally the Lyapunov exponents and Kolmogorov entropy (Kolmogorov 1959). Adrangi et al. (2001), He (2011), Komijani et al. (2014), Lahmiri (2017) and Bildirici and Sonustun (2018) are the investigators that employ these methods to determine chaotic behavior. Nevertheless, He (2011) used two additional methods and examined the presence of chaotic behavior in Brent and WTI crude oil price by the phase space reconstruction technique (PSRT) and fractal integral methodologies, and He (2011) determined the existence of chaos with Lyapunov exponents and Kolmogorov entropy.

Jones and Kaul (1996) and Sadorsky (1999) found that increases in oil prices could affect the stock returns. Jones and Kaul (1996) examined the responses of stock markets of Canada, UK, Japan and the USA to oil price shocks. Sadorsky (1999) analyzed the relation between stock returns in the USA and the volatility of oil price and found that the volatility of oil prices has asymmetric effects: Positive volatility of oil prices has a greater impact on stock returns and economic activity than the negative ones. Ciner’s findings (Ciner 2001) revealed that the oil shocks can affect stock returns in a nonlinear pattern in the USA. Papapetrou (2001) also pointed at the existence of a relationship between the oil price oscillations and the stock markets in Greece. Hammoudeh and Aleisa (2004) determined the evidence of a bidirectional relationship between Saudi stock returns and oil price changes. Hammoudeh and Choi (2006) observed the long-run relation among the Gulf Cooperation Council (GCC) stock markets, the US oil market, the S&P 500 Index and the US Treasury bill rate. Maghyereh and Al-Kandari (2007) showed that, in the long run, oil price influences the stock prices in a nonlinear manner in GCC countries. Arouri and Fouquau (2009) investigated the short-run nexus between oil prices and GCC stock return by using nonparametric methods and showed there are some marks of nonlinearities in the relations between oil and stock market returns in Qatar, Oman and UAE in addition to the relation being asymmetric. Mohanty et al. (2011) presented a linear model by adding a dummy variable and found that the decreases in oil prices have a negative impact in the stock market returns of all countries though the rises affect the stock market returns positively in Saudi Arabia and UAE. Park and Ratti (2008) and Ciner (2013) are other investigators who analyzed the impacts of the increments in oil price on the stock return. Bjørnland (2009) and Wang et al. (2013) determined a positive effect of the increments in crude oil price on the stock return for oil exporting countries. Awartani and Maghyereh (2013) employed the DCC-GARCH method and showed the correlation between stock market returns and the volatility of oil price diverges over time. Bouri (2015a, b) showed an imperative connection between oil price and stock returns for Lebanese markets and determined the tendency to increase in the crisis period and to decrease in the post-crisis period. Dutta et al. (2017) tested if oil volatility index impacts volatility of stock markets for Middle East and African (MEA) countries. They determined the oil market uncertainty has substantial effects on the unpredictability of stock markets and the market participants’ anticipation is the crucial factor for explaining the returns and volatilities in these markets.

In order to analyze the direction of the investor expectations, Whaley (2000) examined the nexus between stock market returns and changes in VIX. A significant finding of Whaley (2000) is that VIX is a barometer of investors’ fear and excitement; hence, high levels of VIX are a significant indicator of high levels of market turmoil. In this respect, Whaley (2009) noted the association of high levels of VIX and increased investor anxiety regarding a potential drop in the stock markets. DeLisle et al. (2011) displayed the sensitivity to VIX innovations and the negative connectedness with the stock returns associated with expectations regarding increases in the volatility. Some of the VIX literature, including Dowling and Muthuswamy (2005), Ederington and Guan (2010) and Giot (2005), underlined the existence of asymmetric impacts. Liu et al. (2013) found that the VIX affected oil price uncertainty by using the oil price volatility index (OVX). Zheng (2014) determined there is a negative nexus between stock market sentiment and returns in the commodity market. Shaikh and Padhi (2015) found that VIX is a gauge for evaluating investors’ fear of market decline and anticipation of increases in stock market volatility. Smales (2017) showed sentimentality has a greater impact on returns during crisis periods. Dutta et al. (2017) investigated whether the oil price volatility index (OVX) impacts the realized volatility of Middle East and African (MEA) countries’ stock markets. They found the oil market uncertainty has substantial effects on the realized volatility of MEA countries’ stock markets and the market participants’ anticipation is crucial factor for explaining the returns and volatilities in these markets.

The papers relating the stock market returns and changes in VIX do not generally give information about the evidence of chaotic behavior, and if omitted, these outcomes could result in erroneous portfolio management and policy recommendations.

4 Data descriptions and econometric methods

4.1 Data descriptions

This paper aims at the analysis of nonlinear and chaotic behavior in the oil prices, investor sentiments and the stock returns of ISE. The ISE-100 Index is renamed recently as the Borsa Istanbul 100 Index, and the series is denoted as BIST. The expectations of investors are measured with the VIX. The data cover the period from January 02, 1990, to June 06, 2017, and were taken from Bloomberg. All variables were converted by taking the first differences of the logarithm of the prices. Daily oil price is calculated as opt = 100(ln(oil pricet) – ln(oil pricet-1)), and VIX is measured as vxt = 100(ln(vixt) – ln(vixt-1)). Further, daily BIST100 stock returns are calculated as rt = 100(ln(BISTt) – ln(BISTt-1)). It is useful to scrutinize a graphical illustration of the data before the analysis to see the general outlook. Figures 1, 2 and 3 present VIX, crude oil price and BIST100 Index for the analyzed period, respectively.

Fig. 1
figure 1

Expectations of investors (VIX)

Full size image
Fig. 2
figure 2

Oil price

Full size image
Fig. 3
figure 3

BIST100 Index

Full size image

4.2 Econometric methods

4.2.1 TAR-TR-TGARCH model and copula measures

The difference of this model from GARCH(1,1) is that the impact of positive and negative shocks is differentiated by employing an indicator variable. It has a value of unity if the previous shock is negative and zero otherwise. The TAR-TR-GARCH model allows threshold-type regime specifications in both the conditional mean and the conditional volatility processes (Bildirici et al. 2017) in Eq. (1).

$$y_{t} = \left( {\omega_{10} + \sum\limits_{i = 1}^{r} {\omega_{1i} y_{t - i} } } \right)\,I(s_{t} \le c) + \left( {\omega_{20} + \sum\limits_{i = 1}^{r} {\omega_{2i} y_{t - i} } } \right)I(s_{t} > c) + \varepsilon_{t}$$
(1)

where the conditional volatility follows a two-regime TR-GARCH(p,q) process given in Eq. (2),

$$\sigma_{t}^{2} = \left( {\alpha_{1,0} + \sum\limits_{i = 1}^{q} {\alpha_{1,i\,} \varepsilon_{1,t - i}^{2} } + \sum\limits_{j = 1}^{p} {\beta_{1,j} \sigma_{1,t - j}^{2} } } \right)\,I(s_{t} \le c) + \left( {\alpha_{2,0} + \sum\limits_{i = 1}^{q} {\alpha_{2,\,i} \varepsilon_{2,\,t - i}^{2} } + \sum\limits_{j = 1}^{p} {\beta_{2,j} \sigma_{2,t - j}^{2} } } \right)I(s_{t} > c) + \varepsilon_{t}$$
(2)

In Eqs. (1) and (2), I(.) is the indicator function with the transition variable st = yt-d being selected among the lag length d that optimizes the explanatory power of the model as d = {1, 2,…, p}. The model allows TAR-style nonlinearity in both the conditional mean and conditional variance processes (Bildirici et al. 2017).

The indicator function in the conditional variance process in which the negative and positive innovations are defined is given in Eq. (3),

$$\begin{aligned} I_{cv} = 1\quad {\text{if}}\quad \varepsilon_{t - 1} < 0, \hfill \\ I_{cv} = 0{\kern 1pt} \quad {\text{if}}\quad \varepsilon_{t - 1} \ge 0 \hfill \\ \end{aligned}$$
(3)

The TAR-TR-TGARCH model allows a threshold autoregressive-type regime switching in both the conditional mean and conditional variance processes. The residuals follow a two-regime TR-TGARCH(p, q) process,

$$\begin{aligned} \sigma_{t}^{2} = \left( {\alpha_{1,0} + \sum\nolimits_{i = 1}^{p} {\beta_{1,i} \sigma_{1,t - i}^{2} + \gamma_{1} \varepsilon_{1,t - 1}^{2} } I_{cv} + \sum\nolimits_{j = 1}^{q} {\alpha_{1,j} \varepsilon_{1,t - j}^{2} } } \right)\,I_{cm} (s_{t} \le c) \hfill \\ + \left( {\alpha_{2,0} + \sum\nolimits_{i = 1}^{p} {\beta_{2,i} \sigma_{2,t - i}^{2} + \gamma_{2} \varepsilon_{2,t - 1}^{2} } I_{cv} + \sum\nolimits_{j = 1}^{q} {\alpha_{2,j} \varepsilon_{2,t - j}^{2} } } \right)I_{cm} (s_{t} > c) \hfill \\ \end{aligned}$$
(4)

In Eq. (4), the indicator function in the conditional mean process Icm(st, c) allows threshold-type regime specifications that require the estimation of threshold c and the optimum transition variable st = yt-d among a set of d = {1, 2,…, p} through Hansen (2000a, b) Wald tests. The conventional TGARCH model of Zakoian employs σt in conditional variance instead of σ 2 t which is accepted in this study.

The TGARCH model of Zakoian (1994) allows negative and positive innovations defined with an indicator function. It is possible to replace Eq. (3) with TGARCH processes defined for each regime, since the transition between regimes for the model in Eq. (3) is governed with st = yt-d and the threshold c. By allowing the residual terms as the transition variable, st = ɛt-1, and if the threshold is taken as c = 0, the model is reduced to a model that switches between regimes for negative and positive innovations.

The test statistic is calculated for the candidate of threshold parameter c in Eq. (5),

$$W_{n} \left( c \right) = \left( {R\theta \left( c \right)} \right)^{\prime } \left[ {\left( {RM_{n} \left( c \right)} \right)^{ - 1} V_{n} \left( c \right)\left( {M_{n} \left( c \right)} \right)^{ - 1} R^{\prime}} \right]^{ - 1} \left( {R\theta \left( c \right)} \right)$$
(5)

where \(\theta = \left[ {\begin{array}{*{20}c} {\omega_{1} } & {\omega_{2} } \\ \end{array} } \right]\), \(R = \left[ {\begin{array}{*{20}c} I & { - I} \\ \end{array} } \right]\), and \(M_{n} \left( c \right) = \sum y_{t} \left( c \right)y_{t} \left( c \right)^{\prime } ,V_{n} \left( c \right) = \sum y_{t} \left( c \right)y_{t} \left( c \right)^{\prime } \varepsilon_{t}^{2} .\)

And Wn = sup Wn(c)c∊R.


The test statistic to evaluate the null hypothesis of no-threshold-type nonlinearity is calculated.

In the threshold effect testing procedure, the critical values are generated using the bootstrap methodology given in Hansen (1996, 2000a, b). For the estimation purposes, the models are estimated with maximum likelihood. The log-likelihood function under conditional normality can be given as Eq. (6):

$$L\left( \varTheta \right) = - T\log \left( {2\pi } \right) - \frac{1}{2}\sum\limits_{t = 1}^{T} {\left( {\log |H_{t} \left( \varTheta \right)| - \varepsilon_{t} \left( \varTheta \right)H_{t}^{ - 1} \left( \varTheta \right)\varepsilon_{t}^{'} \left( \varTheta \right)} \right)}$$
(6)

where T is the number of observations in the sample; Θ is the parameter to be estimated; ɛt = (ɛ1t, ɛ2t) is a vector of residuals or error terms; and Ht is defined as \(H_{t} = \text{cov} \left( {\varepsilon_{t} |\varOmega_{t - 1} } \right)\). The log-likelihood function L(Θ) is maximized subject to the constraint that the conditional variances are positive.

Similar to the previous models, by substituting st = ɛt-1 and taking the threshold c = 0, the TAR-TR-TGARCH models reduce to models that allow regime switches for negative and positive innovations. Though the threshold c and the transition variable st = ɛt-1 are predefined, the Wald test procedure should be applied to evaluate the null hypothesis of no-threshold-type nonlinearity.

The models presented above are to be generalized to copulas. The copula-based TAR-TR-TGARCH method allows the researcher to study the interdependence, the conditional tail dependence and the volatilities of the oil, stock return and expectation of investors. A symmetrized Joe-Clayton (SJC) copula function was preferred and is given in Eq. (7).

$$F^{\text{sjc}} (v_{1} ,v_{2} \left| {\tau^{v} } \right.,\tau^{L} ) = 0.5(F^{\text{jc}} (v_{1} ,v_{2} \left| {\tau^{v} } \right.,\tau^{L} ) + F^{jc} (1 - v_{1} ,1 - v_{2} \left| {\tau^{v} } \right.,\tau^{L} ) + \,v_{1} + \,v_{2} \, - 1).$$
(7)

where τv, τL are the measures of dependence of the upper and lower tails in Eq. (7) (Boubaker and Sghaier 2016a). Fjc is the Joe-Clayton copula given in Eq. (8).

$$F^{\text{jc}} (v_{1} ,v_{2} \left| {\tau^{v} } \right.,\tau^{L} ) = 1 - \left( {1 - \left\{ {\left[ {1 - (1 - v_{1} )^{\kappa } } \right]^{ - \upsilon } + \left[ {1 - (1 - v_{2} )^{\kappa } } \right]^{ - \upsilon } - 1} \right\}^{ - 1/\upsilon } } \right)^{{^{ - 1/\kappa } }}$$
(8)
$${\text{with }}\kappa = 1/\log_{2} (2 - \tau^{v} ),\quad \upsilon = \, - 1/\log_{2} (\tau^{L} )\quad {\text{and}}\,\,\tau^{v} ,\, \tau^{L} \in (0,1).$$

The SJC copula gives the lower and the upper tail dependences. In condition of τv = τL, the dependence is symmetric; if not, it is asymmetric (Boubaker and Sghaier 2016b).

5 Econometric results

Econometric results were obtained by following five stages presented below:

  1. (1)

    In the first stage, Tsay and Hsieh’s tests and some 467 descriptive statistics were applied.

  2. (2)

    In the second stage, the BDS (Brock et al. 1987) test was applied. Although the BDS test measures the evidence of existence of chaotic behavior, it is not sufficient to determine the presence of chaotic behavior (Barnett et al. 1995, 1997; Barnett and Hinich 1992).

  3. (3)

    In the third stage, the Lyapunov exponent and Kolmogorov entropy were presented. Komijani et al. (2014), Lahmiri (2017) and Bildirici and Sonustun (2018) used the largest Lyapunov exponent to determine the volatility of oil prices. The Lyapunov exponent offers a more convenient means to determine chaotic behavior and specifies a system’s level of chaos.

  4. (4)

    TAR-TR-GARCH and TAR-TR-TGARCH models were estimated. For TAR-TR-GARCH and TAR-TR-TGARCH models, it was accepted that there are 1 or 2 threshold parameters and therefore 2 or 3 regime models are estimated depending on upper assumptions. The optimum lag length d is selected depending on the optimization, and d is taken to range between 1 and 5 based on the Akaike information criterion (AIC).

  5. (5)

    And lastly, TAR-TR-GARCH and TAR-TR-TGARCH copula methods were applied to determine the co-movement of the variables. The linear correlation and Gaussian copulas cannot produce a sufficiently high level of dependency experienced in a crisis period. The TAR-TR-GARCH and TAR-TR-TGARCH copula functions were applied to define different stages of dependence, based on regimes introduced by the threshold principle.

5.1 Descriptive results

The descriptive statistics in addition to ARCH-LM, Tsay’s nonlinearity test and Hsieh’s third-order moment tests are reported in Table 1. The results in Table 1 powerfully advocate that the ARCH-type heteroscedasticity and the threshold-type nonlinearity cannot be rejected at conventional significance levels for the series analyzed.

Table 1 Descriptive statistics
Full size table

The results in Table 1 determined that the means of variables are small, but their standard deviations are much higher. Moreover, kurtosis statistics showed that the variables are not normally distributed. The JB statistics is statistically significant at the 1% significance level by showing the robust evidence the null hypothesis of the normal distribution was rejected. Tsay’s tests determine that the linear structure was misspecified for most of the variables. Hsieh’s coefficients are very high. The obtained results show the evidence of nonlinearities.

5.2 BDS test

The BDS test can be accepted as the test of linearity against possible nonlinearity. The BDS test is shown to be more statistically powerful than many other linearity and nonlinearity tests (for details see Brock et al. 1991; Barnett et al. 1997).

BDS test results are reported in Table 2. Accordingly, the null hypothesis of linearity was rejected for the selected variables. And the results are in favor of chaotic behavior in the series investigated.

Table 2 BDS independence test results
Full size table

5.3 Lyapunov exponent test

For the estimation of the Lyapunov exponents (Le), two methods, those of Kantz (1944) and Rosenstein et al. (1993), were employed. The usage of two different methods is based on the fact that, though both are highly respected, the two methods could result in differentiated results. Further, the usage of the two methods is based on precaution in addition to obtaining validation for chaos in the series if detected. Both methods are proven to have good performance in detecting chaotic processes in the existence of noise. The main parameter of embedded dimension is set by having three initial states; Table 3 presents the results by the two methods for only one dimension. The value of the Lyapunov exponent determined by the Kantz and by the Rosenstein et al. methods determined different results in terms of the presence of chaotic dynamics in the oil prices, expectation of investors and the stock returns.

Table 3 Test results of Lyapunov exponent
Full size table

A positive value of Le determines a chaotic process, and therefore the predictability of the path followed by such series is assumed to be fairly low. If the positive value of Le is very close to zero, the existence of the chaotic behavior is assumed to be fragile. A negative value of Le is a strong sign of no chaotic behavior in the long run. The results determined by the Kantz methods exhibited unexpected signs as the dimensions are altered from 1 to 3 for the selected variables. However, this does not hold for the Rosenstein, Collins and De Luca method.

5.4 Kolmogorov entropy

Kolmogorov entropy can be understood as the degrees of falsification of market information in the price structure (He 2011). Kolmogorov entropy (KE) is obtained in two ways. The Lyapunov exponent is one of the methods, and the second one is the correlation integral (Zhao et al. 2009). The first method requires obtaining all positive Lyapunov exponents. For the oil price and the stock returns, the obtained small and positive values of the entropies determined that the information provided by the market will be used to understand the market dynamics. If the variable is non-complex and is completely predictable, KE will approach zero. When the data are random, the value is large; the lower value of KE shows that the series follows a predictable structure.

The results are reported in Table 4. The Kolmogorov entropy for oil price is 0.147. According to this result, the timescale of a rational and effective forecast for that system must be within ~68 days. This result is different to the He (2011) who found that the system should be within 36 days by using the Kolmogorov entropy.

Table 4 Kolmogorov entropy results
Full size table

The volatility series, VIX, is non-chaotic as indicated by a negative largest Lyapunov exponent. The null hypothesis of the existence of chaos for oil price and stock return is rejected. The results show that the vx variable exhibits a long memory in volatility and that the volatility of the variable is not accepted as chaotic.

Even though there are differences between the structures of variables, the variables do not possess a chaotic behavior in high levels. The results of this paper for oil price are dissimilar to the results of some papers which determined the presence of chaos in the oil market. Panas and Ninni (2000), Adrangi et al. (2001), Lahmiri (2017) and Bildirici and Sonustun (2018) showed the presence of chaotic behavior for oil prices. The BDS, Lyapunov and Kolmogorov entropy results determine that the variables are nonlinear. The general investigation of the results shows that VIX is generated by a complex nonlinear and stochastic process.

5.5 The TAR-TR-TGARCH results

As noted in the previous sections, the TAR-TR-GARCH and TAR-TR-TGARCH models allow threshold-type nonlinearity in the conditional mean and the conditional variance processes simultaneously in Table 5. For the optimum lag length d, it is allowed to vary between 1 and 5 depending on the explanatory power of the estimated models. The difference of this model from GARCH(1,1) is that the impacts of positive and negative shocks are differentiated by employing an indicator variable. It has a value of unity if the previous shock is negative, and zero otherwise. This test states the asymmetry in volatility. The estimation results are reported in Table 5. The intercept and the AR(1) terms in each regime are statistically significant at the 1% significance level. The ARCH parameters in regimes 1 and 2 are estimated to be between 0.14 and 0.381 which are statistically significant. The GARCH parameters are estimated as 0.609 and 0.861 which are stable in both regimes. All ARCH and GARCH parameters are statistically significant at 1% significance level. The stability condition is achieved in both regimes since the sum of the ARCH and GARCH parameter estimates is less than 1. The AIC statistics are calculated to be between −2.88 and −4.27.

Table 5 TAR-TR-GARCH copula and TAR-TR-TGARCH copula
Full size table

The threshold estimations are statistically significant, which points to the determination of the existence of asymmetry. The threshold estimate is statistically higher for the first regime compared to the second regime that holds once the threshold variable passes the threshold coefficient of 2.48 for oil, 2.01 for stock return and 1.993 for expectations of investors.

The ARCH-LM tests suggest that no ARCH effect exists in the residuals. As an economic perspective, the modeling of the series with the TAR-TR-GARCH and TAR-TR-TGARCH models should be taken as evidence of the business cycles in the economies. Further, considering the fact that different dynamics under different regimes are relevant characteristics of the business cycles, modeling series with a single-regime traditional GARCH structure is insufficient since the direction of volatility and its forecastability become highly relevant for policy purposes.

The estimated models are extended to copula models to obtain the copula-based TAR-TR-GARCH and TAR-TR-TGARCH models to be used for the measurement of the bivariate density distributions. The models effectively provide important information regarding the conditional tail dependence which could give significant measures to analyze how the stock return and expectations of investors behave once the oil returns are subject to extreme rises or downfalls. The results display the signs and significance of the estimated parameters for both the TAR-TR-GARCH and TAR-TR-TGARCH models, i.e., the marginal density distributions are consistent with the ones expected. Furthermore, the evidence suggests the importance of the estimation of joint distributions which overcome the misspecification problems experienced in the literature. According to the results of this paper, the generalization of the copula-based distributional aspects is useful in the characterization of the co-movements between the evaluated series. The estimation results also reveal that the series have significant asymmetric effects on the volatilities. Additionally, the results determine the existence of threshold effects on the stock returns which are significantly higher than the effects of oil prices on the volatility. Thus, the evidence suggests that the volatility of stock returns may be more sensitive than the oil price volatility. Besides, the significance of the parameters determines that the normality assumption cannot be suitable for making either economic or financial decisions and that the modeling of joint distributional aspects provides augmentation in terms of modeling and forecasting the analyzed financial time series.

6 Conclusions

This paper aimed at the investigation of the existence of chaotic behavior in the stock returns, oil prices and expectations of investors by utilizing chaos-based methodologies including the Lyapunov exponents and Kolmogorov entropy. Nevertheless, a further contribution of this paper was the evaluation of the dependence between the analyzed series for the period covering January 02, 1990–June 06, 2017, with the suggested TAR-TR-GARCH and TAR-TR-TGARCH copula tests. Additionally, the TAR-TR-GARCH and TAR-TR-TGARCH models were used to compare the bad and good regimes. These models displayed significant explanatory power compared to the single-regime variants; hence, they were chosen as the appropriate models in the detection of the interrelations between the oil prices, stock returns and the investor expectations for distinct regimes. Lyapunov exponents and Kolmogorov entropy determined that not only did the oil price and the stock return series exhibit chaotic behavior and nonlinear structure, but the results also revealed the existence of such strong nonlinear effects that failed to hold for the nonlinear structure of the investor expectations. The TAR-TR-GARCH and TAR-TR-TGARCH copula models also favored significant evidence of upper and lower tail dependence among the three series analyzed for the periods falling under the bad and good regimes depending on the estimated thresholds. Overall evaluation suggested significant evidence of dependence structures which further supported the existence of the persistence and contagion effects. The sensitivity of stock returns and expectations of investors to the volatility of oil prices bounced following the bad regime period, and the copula parameters and tail dependence coefficients rose to higher levels during the periods falling under regime 1 compared to those under regime 2. An important result of the empirical evidence regarding the significance of the threshold effect was the existence of the asymmetric dependence structures resulting from the impacts of the oil price, expectations of investors and the volatility of the investigated stock market.

The empirical findings led to differentiated policy suggestions. The policy makers must shield the economy from large fluctuations in oil prices since the oil prices, stock returns and expectations of investors have important nonlinear associations resulting from asymmetry and threshold effects in addition to the persistence effects. Given the implication of the chaotic structure determined for the series analyzed, the policy makers must evaluate the fluctuations and must apply policies with great care since they could have significant effects on the economy and especially on the financial markets. From the political perspective, the nonlinear and the asymmetric characteristics of the oil prices, stock returns and expectations of investors should play a crucial role in the determination of the selection of the policies. This result is especially important for oil prices, since the selected policy or policies directed at stabilizing the volatility may have an inverse outcome and lead to destabilizing effects on production and on financial markets. The fluctuations in oil prices could easily be transferred to stock returns and expectation of investors in the long run. According to the results, policymakers must analyze carefully the volatility of oil prices, especially with regard to the expectations of investors and the stock returns.