Skip to main content
Log in

Forecasting commodity futures returns with stepwise regressions: Do commodity-specific factors help?

  • S.I.: Recent Developments in Financial Modeling and Risk Management
  • Published:
Annals of Operations Research Aims and scope Submit manuscript


The aim of this paper is to assess whether three well-known commodity-specific variables (basis, hedging pressure, and momentum) may improve the predictive power for commodity futures returns of models otherwise based on macroeconomic factors. We compute recursive, out-of-sample forecasts for the monthly returns of fifteen commodity futures, when estimation is based on a stepwise model selection approach under a probability-weighted regime-switching regression that identifies different volatility regimes. We systematically compare these forecasts with those produced by a simple AR(1) model that we use as a benchmark and we find that the inclusion of commodity-specific factors does not improve the forecasting power. We perform a back-testing exercise of a mean–variance investment strategy that exploits any predictability of the conditional risk premium of commodities, stocks, and bond returns, also consider transaction costs caused by portfolio rebalancing. The risk-adjusted performance of this strategy does not allow us to conclude that any forecasting approach outperforms the others. However, there is evidence that investment strategies based on commodity-specific predictors outperform the remaining strategies in the high-volatility state.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2

Similar content being viewed by others


  1. Jensen et al. (2000) provide evidence on the role of commodity futures in mean–variance portfolios. They find that in periods of restrictive monetary policy, commodity futures carry an important weight and yield a considerable performance enhancement. However, since their paper, it has become common to classify financial market regimes on the basis of the level of volatility.

  2. Yamashita et al. (2007) compare the stepwise AIC selection method with other stepwise methods for variable selection and show that this practical criterion leads to the same results as partial F tests.

  3. As a robustness check, instead of running two separate regressions according to a classification of the regime based on the state of the VIX, we also estimate Markov-switching predictive regressions. The results are discussed in detail in Sects. 4.3 and 5.2.

  4. As recently shown by Aslan et al. (2018), with reference to commodity returns, it may be possible to group different commodity returns series on the basis of commonalities in the estimated linear autoregressive and non-linear threshold autoregressive features to further reduce the dimension of the cross-section.

  5. More precisely, the procedure is as follows: at any time, t, we sort the commodities according to their past 12‐month performances and create an equally‐weighted portfolio that is long on the first 5 commodities in the ranking and short on the last 5 commodities in the ranking.

  6. The fact that the number and nature of the principal components included in the “optimized” predictive regressions is highly sensitive to whether the data are drawn from a low- versus a high-volatility regime provides indirect confirmation of the presence of regime switching dynamics in the data.

  7. While the upper portion of the Table relies only on whether the state probability of a low regime exceed 0.5 or not, the bottom parts of the Table also rely on the predictions from the estimated two-state MS model for the VIX. Although one may argue that this way of proceeding is more elegant and consistent with the framework of our paper, note that at this point we find ourselves jointly assessing the forecasting power of the predictive regressions that include or not commodity-specific factors and the forecasting accuracy of a simple MS model for the VIX. The latter model, as simple and compelling as it may appear, does not represent the main object of our analysis.

  8. We have also experimented with 5-year rolling estimation windows, obtaining qualitatively similar results.

  9. In panel C, the average allocations implied by the benchmark are even more biased towards long positions in government bonds, now exceeding 100%. The long positions in commodities are modest and now concentrated in silver, Brent crude oil, and gasoline; gold is instead massively shorted, which represents the most visible difference versus the allocations in panels A and B.

  10. Also in this case, the effect can be noted only when the predictions are computed using a forward stepwise algorithm that starts out with a null model without any predictability, and progressively expands the set of predictors if and when these lower the AIC of the resulting model.

  11. In Table 8 and also as a way to check the robustness of our results, we have extended the exercise to include more values of the risk aversion coefficient \( \gamma \), also exceeding 1.

  12. We have also performed this robustness check for the case without transaction costs and it gave insights qualitatively similar to those reported in the main text.

  13. We also performed the exercises accounting for transaction costs, similarly to Sect. 5.1. The results, which are not reported for the sake of brevity, are comparable to those discussed for the case of no transaction costs.


  • Aslan, S., Yozgatligil, C., & Iyigun, C. (2018). Temporal clustering of time series via threshold autoregressive models: Application to commodity prices. Annals of Operations Research, 260, 51–77.

    Article  Google Scholar 

  • Bakshi, G., Gao, X., & Rossi, A. G. (2019). Understanding the sources of risk underlying the cross section of commodity returns. Management Science, 65(2), 619–641.

    Article  Google Scholar 

  • Bessembinder, H., & Chan, K. (1992). Time-varying risk premia and forecastable returns in futures markets. Journal of Financial Economics, 32, 169–193.

    Article  Google Scholar 

  • Bossaerts, P., & Hillion, P. (1999). Implementing statistical criteria to select return forecasting models: What do we learn? Review of Financial Studies, 12, 405–428.

    Article  Google Scholar 

  • Brennan, M. J. (1958). The supply of storage. American Economic Review, 48, 50–72.

    Google Scholar 

  • Chong, J., & Miffre, J. (2010). Conditional correlation and volatility in commodity futures and traditional asset markets. Journal of Alternative Investments, 12, 61–75.

    Article  Google Scholar 

  • Dal Pra, G., Guidolin, M., Pedio, M., & Vasile, F. (2018). Regime shifts in excess stock return predictability: An out-of-sample portfolio analysis. Journal of Portfolio Management, 40, 10–24.

    Article  Google Scholar 

  • Daskalaki, C., Kostakis, A., & Skiadopoulos, G. (2014). Are there common factors in individual commodity futures returns? Journal of Banking & Finance, 40, 346–363.

    Article  Google Scholar 

  • Daskalaki, C., & Skiadopoulos, G. (2011). Should investors include commodities in their portfolios after all? New evidence. Journal of Banking & Finance, 35, 2606–2626.

    Article  Google Scholar 

  • de Roon, F. A., Nijman, T. E., & Veld, C. (2000). Hedging pressure effects in futures markets. Journal of Finance, 55, 1437–1456.

    Article  Google Scholar 

  • DeMiguel, V., Garlappi, L., & Uppal, R. (2007). Optimal versus naive diversification: How inefficient is the 1/N portfolio strategy? Review of Financial studies, 22, 1915–1953.

    Article  Google Scholar 

  • Erb, C. B., & Harvey, C. R. (2006). The tactical and strategic value of commodity futures. Financial Analysts Journal, 62, 69–97.

    Article  Google Scholar 

  • Fuertes, A. M., Miffre, J., & Rallis, G. (2010). Tactical allocation in commodity futures markets. Journal of Banking & Finance, 34, 2530–2540.

    Article  Google Scholar 

  • Gargano, A., & Timmermann, A. (2014). Forecasting commodity price indexes using macroeconomic and financial predictors. International Journal of Forecasting, 30, 825–843.

    Article  Google Scholar 

  • Giampietro, M., Guidolin, M., & Pedio, M. (2018). Estimating stochastic discount factor models with hidden regimes: Applications to commodity pricing. European Journal of Operational Research, 265, 685–702.

    Article  Google Scholar 

  • Gorton, G. B., Hayashi, F., & Rouwenhorst, K. G. (2012). The fundamentals of commodity futures returns. Review of Finance, 17, 35–105.

    Article  Google Scholar 

  • Guidolin, M., & Timmermann, A. (2007). Asset allocation under multivariate regime switching. Journal of Economic Dynamics and Control, 31(11), 3503–3544.

    Article  Google Scholar 

  • Hamilton, J. D., & Wu, J. C. (2015). Effects of index-fund investing on commodity futures prices. International Economic Review, 56, 187–205.

    Article  Google Scholar 

  • Henriksen, T. E. S., Pichler, A., Westgaard, S., & Frydenberg, S. (2019). Can commodities dominate stock and bond portfolios? Annals of Operations Research, 282(1–2), 155–177.

    Article  Google Scholar 

  • Hicks, J. R. (1939). Value and capital. London: Oxford Clarendon Press.

    Google Scholar 

  • Hirshleifer, D. (1988). Residual risk, trading costs, and commodity futures risk premia. Review of Financial Studies, 1, 173–193.

    Article  Google Scholar 

  • Hong, H., & Yogo, M. (2012). What does futures market interest tell us about the macroeconomy and asset prices? Journal of Financial Economics, 105, 473–490.

    Article  Google Scholar 

  • Irwin, S. H., & Sanders, D. R. (2011). Index funds, financialization, and commodity futures markets. Applied Economic Perspectives and Policy, 33, 1–31.

    Article  Google Scholar 

  • Jensen, G., Johnson, R., & Mercer, J. (2000). Efficient use of commodity futures in diversified portfolios. Journal of Futures Markets, 20, 489–506.

    Article  Google Scholar 

  • Kaldor, N. (1939). Speculation and economic stability. Review of Economic Studies, 7, 1–27.

    Article  Google Scholar 

  • Keynes, J. M. (1930). A Treatise on Money. London: Macmillan Press.

    Google Scholar 

  • Lean, H., H., Nguyen, D., K., & Uddin, G. (2018). On the Role of Commodity Futures in Portfolio Diversification, Working paper, Indiana University.

  • Ludvigson, S. C., & Ng, S. (2009). Macro factors in bond risk premia. Review of Financial Studies, 22, 5027–5067.

    Article  Google Scholar 

  • Narayan, P. K., Ahmed, H. A., & Narayan, S. (2015). Do momentum-based trading strategies work in the commodity futures markets? Journal of Futures Markets, 35, 868–891.

    Article  Google Scholar 

  • Rapach, D. E., & Wohar, M. E. (2006). In-sample vs. out-of-sample tests of stock return predictability in the context of data mining. Journal of Empirical Finance, 13, 231–247.

    Article  Google Scholar 

  • Rapach, D., E., & Zhou, G. (2013). Forecasting stock returns. In Handbook of Economic Forecasting Vol. 2, Part A, pp. 328–383.

  • Sharma, M. J., & Yu, S. J. (2015). Stepwise regression data envelopment analysis for variable reduction. Applied Mathematics and Computation, 253, 126–134.

    Article  Google Scholar 

  • Shen, Q., Szakmary, A. C., & Sharma, S. C. (2007). An examination of momentum strategies in commodity futures markets. Journal of Futures Markets, 27, 227–256.

    Article  Google Scholar 

  • Stock, J. H., & Watson, M. W. (2002). Forecasting using principal components from a large number of predictors. Journal of the American Statistical Association, 97, 1167–1179.

    Article  Google Scholar 

  • Stock, J. H., & Watson, M. W. (2006). Forecasting with many predictors. In G. Elliott, C. Granger, & A. Timmermann (Eds.), Handbook of economic forecasting (Vol. 1, pp. 515–554). Amsterdam: Elsevier.

    Chapter  Google Scholar 

  • Stoll, H. R. (1979). Commodity futures and spot price determination and hedging in capital market equilibrium. Journal of Financial and Quantitative Analysis, 14, 873–894.

    Article  Google Scholar 

  • Tang, K., & Xiong, W. (2012). Index investment and the financialization of commodities. Financial Analysts Journal, 68, 54–74.

    Article  Google Scholar 

  • Welch, I., & Goyal, A. (2008). A comprehensive look at the empirical performance of equity premium prediction. Review of Financial Studies, 21, 1455–1508.

    Article  Google Scholar 

  • Yamashita, T., Yamashita, K., & Kamimura, R. (2007). A stepwise AIC method for variable selection in linear regression. Communications in Statistics-Theory and Methods, 36(13), 2395–2403.

    Article  Google Scholar 

  • You, L., & Daigler, R. T. (2013). A Markowitz optimization of commodity futures portfolios. Journal of Futures Markets, 33, 343–368.

    Article  Google Scholar 

  • Zheng, H., Kulkarni, S. R., & Poor, H. V. (2013). A sequential predictor retraining algorithm and its application to market prediction. Annals of Operations Research, 208, 209–225.

    Article  Google Scholar 

Download references


We would like to thank the editors of the special issue on “Recent Developments in Financial Modeling and Risk Management” and three anonymous referees for insightful comments and encouragement to improve this paper.

Author information

Authors and Affiliations


Corresponding author

Correspondence to Manuela Pedio.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.



The first group of macroeconomic variables includes output and income series, such as personal income, retail sales, and real consumption, taken from The Conference Board’s Indicator (TCB), and total industrial production, extrapolated from the Global Insights Basic Economics Database (GIBED). The second group contains labour market series. The series of average weekly hours of production or non-support workers on private non-farm payrolls, total employees in civilian labour force, and unemployment average duration in weeks are taken from the database GIBED, while the source of average weekly initial claims for unemployment insurance is TCB. The third group contains housing series. The series of total new private housing units are taken from GIBED, while the manufacturing and trade inventories are taken from TCB. The fourth group includes consumption, orders, and inventories series, such as manufacturers’ new orders series and National Association Of Purchasing Managers (NAPM) new orders index, National Association Of Purchasing Managers (NAPM) vendor deliveries index, and National Association Of Purchasing Managers (NAPM) inventories index. The fifth group includes money and credit variables. The series of monetary base, S&P 500, commercial and industrial loans, and average effective exchange rates of this group are taken from GIBED. To this series, we add the historical news-based policy index and the economic policy uncertainty index. The historical news-based policy index is a proxy of the economic policy uncertainty for the US, based on three types of underlying components: the newspaper coverage of policy-related economic uncertainty, the number of federal tax code provisions set to expire in future years, and the disagreement among economic forecasters. The economic policy uncertainty index represents the US movements in policy-related economic uncertainty. This index indicates that when the uncertainty increases, stock prices rise and the level of investment, employment, and country’s outcome go down.

The sixth group includes stock market series. They are S&P 500 Index and S&P 500 industrials series. We also include dividend yield (DY), earning price ratio (EP), and dividend pay-out ratio (DP) of S&P 500 index. The dividends used here are the 12-month moving sums of dividends paid on the S&P 500 Index. Earning price ratio is obtained as the log of earnings minus the log of stock prices where the earnings are the 12-month moving sums on the S&P 500 Index. The dividend pay-out ratio is computed as the log of dividends minus the log of earnings. We also consider stock return volatility (SVOL), which is the monthly sum of the squared daily stock returns on the S&P 500 index, and book-to-market ratio (BM), that is the ratio of the book value to market value for the Dow Jones Industrial Average. We also take into account the net equity expansion (NTIS), which is the ratio of the 12-month moving sums of net issues by New York Stock Exchange (NYSE) listed stocks to the total market capitalization of NYSE stocks.

The seventh group includes bond and exchange rate series. We additionally consider Treasury Bill Rate (TBL), which relates to the interest rate on a three-month Treasury bill, the long-term government bond yield (LTY), the long-term return government bond (LTR) returns, the term spread (TMS), the default yield spread (DFY), calculated as the difference between Moody’s BAA- and AAA-rated bond yields, and the default return spread (DFR), calculated as the difference between long-term corporate and government bond returns.

The eighth group includes prices series. They represent producer price index series, National Association of Purchasing Managers (NAPM) commodity price index, consumer of price index series, personal consumption expenditures series, and average hourly earnings of production and nonsupervisory employees series. To these, we add the crude oil price (COP), computed using the logarithmic changes in the nominal price of West Texas Intermediate crude oil, provided by the Federal Reserve Bank of St. Louis.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Guidolin, M., Pedio, M. Forecasting commodity futures returns with stepwise regressions: Do commodity-specific factors help?. Ann Oper Res 299, 1317–1356 (2021).

Download citation

  • Published:

  • Issue Date:

  • DOI: