Advertisement

Empirical Economics

, Volume 56, Issue 3, pp 797–830 | Cite as

Lag length selection and p-hacking in Granger causality testing: prevalence and performance of meta-regression models

  • Stephan B. BrunsEmail author
  • David I. Stern
Article

Abstract

The academic system incentivizes p-hacking, where researchers select estimates and statistics with statistically significant p-values for publication. We analyze the complete process of Granger causality testing including p-hacking using Monte Carlo simulations. If the degrees of freedom of the underlying vector autoregressive model are small to moderate, information criteria tend to overfit the lag length and overfitted vector autoregressive models tend to result in false-positive findings of Granger causality. Researchers may p-hack Granger causality tests by estimating multiple vector autoregressive models with different lag lengths and then selecting only those models that reject the null of Granger non-causality for presentation in the final publication. We show that overfitted lag lengths and the corresponding false-positive findings of Granger causality can frequently occur in research designs that are prevalent in empirical macroeconomics. We demonstrate that meta-regression models can control for spuriously significant Granger causality tests due to overfitted lag lengths. Finally, we find evidence that false-positive findings of Granger causality may be prevalent in the large literature that tests for Granger causality between energy use and economic output, while we do not find evidence for a genuine relation between these variables as tested in the literature.

Keywords

Granger causality p-hacking Publication bias Information criteria Meta-analysis Vector autoregression 

JEL Classification

C12 C18 C32 Q43 

Notes

Acknowledgements

We thank Alessio Moneta, participants at the Meta-Analysis in Economic Research Network Colloquium 2013 in Greenwich, participants at the Empirical Workshop on Energy 2014 in Kassel, participants at the IWH-CIREQ Macroeconometric Workshop 2015 in Halle, and anonymous reviewers for helpful comments. All remaining errors are ours.

Supplementary material

181_2018_1446_MOESM1_ESM.zip (27 kb)
The data and code used in this paper (1. Code, 2. Data, 3. Detailed readme files) are collected in the electronic supplementary material of this article. (ZIP 27 kb)

References

  1. Adom PK (2011) Electricity consumption-economic growth nexus: the Ghanaian case. Int J Energy Econ Policy 1(1):18–31Google Scholar
  2. Akaike H (1974) A new look at the statistical model identification. IEEE Trans Autom Control 19(6):716–723Google Scholar
  3. Alam M, Begum I, Buysse J, Rahman S, Van Huylenbroeck G (2011) Dynamic modeling of causal relationship between energy consumption, CO2 emissions and economic growth in India. Renew Sustain Energy Rev 15(6):3243–3251Google Scholar
  4. Bowden N, Payne J (2009) The causal relationship between US energy consumption and real output: a disaggregated analysis. J Policy Model 31(2):180–188Google Scholar
  5. Brodeur A, Le M, Sangnier M, Zylberberg Y (2016) Star wars: the empirics strike back. Am Econ J: Appl Econ 8(1):1–32Google Scholar
  6. Bruns SB (2017) Meta-regression models and observational research. Oxf Bull Econ Stat 79(5):637–653Google Scholar
  7. Bruns SB, Ioannidis JPA (2016) p-curve and p-hacking in observational research. PLoS ONE 11:e0149144Google Scholar
  8. Bruns SB, Gross C, Stern DI (2014) Is there really Granger causality between energy use and output? Energy J 35(4):101–134Google Scholar
  9. Ciarreta A, Otaduy J, Zarraga A (2009) Causal relationship between electricity consumption and GDP in Portugal: a multivariate approach. Empir Econ Lett 8(7):693–701Google Scholar
  10. Cumming G (2014) The new statistics: why and how. Psychol Sci 25(1):7–29Google Scholar
  11. DiCiccio TJ, Efron B (1996) Bootstrap confidence intervals. Stat Sci 11(3):189–212Google Scholar
  12. Efron B (1987) Better bootstrap confidence intervals. J Am Stat Assoc 82(397):171–185Google Scholar
  13. Esso LJ (2010) Threshold cointegration and causality relationship between energy use and growth in seven African Countries. Energy Econ 32(6):1383–1391Google Scholar
  14. Glaeser EL (2011) Researcher incentives and empirical methods. In: Schotter A, Caplin A (eds) The foundations of positive and normative economics: a hand book. Oxford University Press, OxfordGoogle Scholar
  15. Granger CWJ (1969) Investigating causal relations by econometric models and cross-spectral methods. Econometrica 37(3):424–438Google Scholar
  16. Granger CWJ (1988) Some recent developments in a concept of causality. J Econom 39:199–211Google Scholar
  17. Hacker RS, Hatemi-J A (2008) Optimal lag-length choice in stable and unstable VAR models under situations of homoscedasticity and ARCH. J Appl Stat 35(6):601–615Google Scholar
  18. Hurvich CM, Tsai C-L (1989) Regression and time series model selection in small samples. Biometrika 76:297–307Google Scholar
  19. Ioannidis JPA (2005) Why most published research findings are false. PLoS Med 2(8):e124Google Scholar
  20. Ioannidis JPA, Doucouliagos C (2013) What’s to know about credibility of empirical economics? J Econ Surv 27(5):997–1004Google Scholar
  21. Ioannidis JPA, Stanley TD, Doucouliagos H (2017) The power of bias in economics research. Econ J 127(605):F236–F265Google Scholar
  22. Kim JH, Ji P (2015) Significance testing in empirical finance: a critical review and assessment. J Emp Fin 34:1–14Google Scholar
  23. Leamer EE (1983) Let’s take the con out of econometrics. Am Econ Rev 73(1):31–43Google Scholar
  24. Lee C (2006) The Causality relationship between energy consumption and GDP in G-11 countries revisited. Energy Policy 34(9):1086–1093Google Scholar
  25. Lotfalipour M, Falahi M, Ashena M (2010) Economic growth, co2 emissions, and fossil fuels consumption in Iran. Energy 35(12):5115–5120Google Scholar
  26. Lütkepohl H (1982) Non-causality due to omitted variables. J Econom 19:367–378Google Scholar
  27. Lütkepohl H (1985) Comparison of criteria for estimating the order of a vector autoregressive process. J Time Ser Anal 6(1):35–52Google Scholar
  28. Lütkepohl H (2007) New introduction to multiple time series analysis. Springer, BerlinGoogle Scholar
  29. MacKinnon JG (2002) Bootstrap inference in econometrics. Can J Econ 35(4):615–644Google Scholar
  30. McCloskey D, Ziliak S (1996) The standard error of regressions. J Econ Lit 34:97–114Google Scholar
  31. Mehrara M (2007) Energy consumption and economic growth: the case of oil exporting countries. Energy Policy 35(5):2939–2945Google Scholar
  32. Menyah K, Wolde-Rufael Y (2010a) CO2 emissions, nuclear energy, renewable energy and economic growth in the US. Energy Policy 38(6):2911–2915Google Scholar
  33. Menyah K, Wolde-Rufael Y (2010b) Energy consumption, pollutant emissions and economic growth in South Africa. Energy Econ 32(6):1374–1382Google Scholar
  34. Nelson CR, Plosser CR (1982) Trends and random walks in macroeconomic time series: some evidence and implications. J Monet Econ 10(2):139–162Google Scholar
  35. Nickelsburg G (1985) Small-sample properties of dimensionality statistics for fitting VAR models to aggregate economic data: a Monte Carlo study. J Econom 28(2):183–192Google Scholar
  36. Ozcicek O, Mcmillin W (1999) Lag length selection in vector autoregressive models: symmetric and asymmetric lags. Appl Econ 31(4):517–524Google Scholar
  37. Payne JE (2009) On the dynamics of energy consumption and output in the US. Appl Energy 86(4):575–577Google Scholar
  38. Payne JE, Taylor JP (2010) Nuclear energy consumption and economic growth in the US: an empirical note. Energy Sources Part B: Econ Plan Policy 5(3):301–307Google Scholar
  39. Sari R, Soytas U (2009) Are global warming and economic growth compatible? evidence from five OPEC countries. Appl Energy 86(10):1887–1893Google Scholar
  40. Schuemie MJ, Ryan PB, Dumouchel W, Suchard MA, Madigan D (2014) Interpreting observational studies: why empirical calibration is needed to correct p-values. Stat Med 33(2):209–218Google Scholar
  41. Schwarz G (1978) Estimating the dimension of a model. Ann Stat 6(2):461–464Google Scholar
  42. Simonsohn U, Nelson LD, Simmons JP (2014) P-curve: a key to the file-drawer. J Exp Psychol Gen 143(2):534–547Google Scholar
  43. Soytas U, Sari R (2009) Energy consumption, economic growth, and carbon emissions: challenges faced by an EU candidate member. Ecol Econ 68(6):1667–1675Google Scholar
  44. Soytas U, Sari R, Ewing B (2007) Energy consumption, income, and carbon emissions in the United States. Ecol Econ 62(3–4):482–489Google Scholar
  45. Stanley TD (2008) Meta-regression methods for detecting and estimating empirical effects in the presence of publication selection. Oxf Bull Econ Stat 70(1):103–127Google Scholar
  46. Stanley TD, Jarrell SB (1989) Meta-regression analysis: a quantitative method of literature surveys. J Econ Surv 3(2):161–170Google Scholar
  47. Stern DI (1993) Energy use and economic growth in the USA: a multivariate approach. Energy Econ 15:137–150Google Scholar
  48. Toda HY, Yamamoto T (1995) Statistical inference in vector autoregressions with possibly integrated processes. J Econom 66(1):225–250Google Scholar
  49. Vaona A (2012) Granger Non-causality between (non)renewable energy consumption and output in Italy since 1861: the (ir)relevance of structural breaks. Energy Policy 45:226–236Google Scholar
  50. Vivalt E (2017) The trajectory of specification searching and publication bias across methods and disciplines. Working paperGoogle Scholar
  51. Wasserstein RL, Lazar NA (2016) The ASA’s statement on p-values: context, process, and purpose. Am Stat 70(2):129–133Google Scholar
  52. Wolde-Rufael Y (2009) Energy consumption and economic growth: the experience of African countries revisited. Energy Economics 31(2):217–224Google Scholar
  53. Wolde-Rufael Y (2010a) Bounds test approach to cointegration and causality between nuclear energy consumption and economic growth in India. Energy Policy 38(1):52–58Google Scholar
  54. Wolde-Rufael Y (2010b) Coal consumption and economic growth revisited. Appl Energy 87(1):160–167Google Scholar
  55. Wolde-Rufael Y, Menyah K (2010) Nuclear energy consumption and economic growth in nine developed countries. Energy Econ 32(3):550–556Google Scholar
  56. Zachariadis T (2007) Exploring the relationship between energy use and economic growth with bivariate models: new evidence from G-7 countries. Energy Econ 29(6):1233–1253Google Scholar
  57. Zapata HO, Rambaldi AN (1997) Monte Carlo evidence on cointegration and causation. Oxf Bull Econ Stat 59(2):285–298Google Scholar
  58. Zhang X-P, Cheng X-M (2009) Energy consumption, carbon emissions, and economic growth in China. Ecol Econ 68(10):2706–2712Google Scholar
  59. Ziramba E (2009) Disaggregate energy consumption and industrial production in South Africa. Energy Policy 37(6):2214–2220Google Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Department of EconomicsUniversity of GöttingenGöttingenGermany
  2. 2.Crawford School of Public PolicyThe Australian National UniversityActonAustralia

Personalised recommendations