Skip to main content

On the Directional Accuracy of United States Housing Starts Forecasts: Evidence from Survey Data

Abstract

I use data from both the Survey of Professional Forecasters and the Livingston Survey to study the directional accuracy of United States housing starts forecasts. Using elements of relative operating characteristic (ROC) analysis, I find that forecasts contain information with respect to subsequent changes in housing starts. Estimates for both surveys are significant at all forecast horizons and robust across time and across forecasters. Implications for the usage of housing starts forecasts from survey data are discussed.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Notes

  1. 1.

    I refer to forecasters as semi-professional if they do not consider forecasting a major part of their main job. See “Data” for a description of the surveys’ panelists.

  2. 2.

    Technically, the presence of noise is also a type of signal that the operator observes. For ease of understanding, I will subsequently use the term nonsignal as a synonym for noise. See also “Directional Forecast Accuracy using ROC Analysis” for the definition used in this paper.

  3. 3.

    An exceptionally high current year forecast from 1986:Q2 stands out. It remains unclear whether the specific individual forecast was actually on that level or if it is due to a type error in the data sheet.

  4. 4.

    Test results for differences in the AUC statistics of the three forecast types based on Eq. 9 are not presented here. The AUC statistics of individual forecasts are smaller than those for mean and median forecasts at all forecast horizons. However, in all cases (except for the current-quarter-ahead forecasts, where the difference is significant at the 5% level) the differences are not statistically significant. The same result holds for the differences in mean and median forecasts. Hence, while the AUC statistics for individual forecasts have the smallest values at all forecast horizons, the differences are not sufficiently large enough to prefer a particular type of forecast when working with the data. All results are available from the author upon request.

  5. 5.

    Test results for differences in the AUC statistics of the three forecast types are not presented because none of the results are statistically significant. All results are available from the author upon request.

  6. 6.

    Rolling window estimates are robust to perturbations of the length of the estimation window. Estimates for fifteen and twenty year rolling windows are not presented but available from the author upon request.

  7. 7.

    Results of the forecaster subgroups AUC comparison are not presented here because 34 out of 36 differences are largely insignificant. Hence, when using forecasts from the LS there is relatively little discrepancy in using aggregate data or subgroups because the directional forecast accuracy is relatively homogeneous across the subgroups of forecasters. The results are available from the author upon request.

  8. 8.

    In general, the results of the Efficiency index look similar to those of the Youden Index with a relatively higher volatility. They are not presented here but available from the author upon request.

References

  1. Adams, Z., & Füss, R. (2010). Macroeconomic determinants of international housing markets. Journal of Housing Economics, 19(1), 38–50.

    Article  Google Scholar 

  2. Agnello, L., & Schuknecht, L. (2011). Booms and busts in housing markets: Determinants and implications. Journal of Housing Economics, 20(3), 171–190.

    Article  Google Scholar 

  3. Ash, J.C.K., Smyth, D.J., & Heravi, S.M. (1998). Are OECD forecasts rational and useful?: a directional analysis. International Journal of Forecasting, 14 (3), 381–391.

    Article  Google Scholar 

  4. Baghestani, H. (2009). A comparison of US housing starts forecasts. Economics Bulletin, 29(4), 2525–2530.

    Google Scholar 

  5. Baker, S.G., & Kramer, B.S. (2007). Peirce, Youden, and receiver operating characteristic curves. The American Statistician, 61(4), 343–346.

    Article  Google Scholar 

  6. Ball, M., Meen, G., & Nygaard, C. (2010). Housing supply price elasticities revisited: Evidence from international, national, local and company data. Journal of Housing Economics, 19(4), 255–268.

    Article  Google Scholar 

  7. Bamber, D. (1975). The area above the ordinal dominance graph and the area below the receiver operating characteristic graph. Journal of Mathematical Psychology, 12(4), 387–415.

    Article  Google Scholar 

  8. Berge, T.J., & Jordà, Ò. (2011). Evaluating the classification of economic activity into recessions and expansions. American Economic Journal: Macroeconomics, 3(2), 246–277.

    Google Scholar 

  9. Bork, L., & Møller, S.V. (2015). Forecasting house prices in the 50 states using dynamic model averaging and dynamic model selection. International Journal of Forecasting, 31(1), 63–78.

    Article  Google Scholar 

  10. Bracke, P. (2013). How long do housing cycles last? A duration analysis for 19 OECD countries. Journal of Housing Economics, 22(3), 213–230.

    Article  Google Scholar 

  11. Burnside, C., Eichenbaum, M., & Rebelo, S. (2011). Understanding booms and busts in housing markets. Working Paper 16734, National Bureau of Economic Research.

  12. Caldera, A., & Johansson, Å. (2013). The price responsiveness of housing supply in OECD countries. Journal of Housing Economics, 22(3), 231–249.

    Article  Google Scholar 

  13. Campbell, J.Y., & Cocco, J.F. (2007). How do house prices affect consumption? Evidence from micro data. Journal of Monetary Economics, 54(3), 591–621.

    Article  Google Scholar 

  14. Case, B., Goetzmann, W.N., & Rouwenhorst, K.G. (2000). Global real estate markets - Cycles and fundamentals. Working Paper 7566, National Bureau of Economic Research.

  15. Croce, R.M., & Haurin, D.R. (2009). Predicting turning points in the housing market. Journal of Housing Economics, 18(4), 281–293.

    Article  Google Scholar 

  16. Croushore, D. (2011). Frontiers of real-time data analysis. Journal of Economic Literature, 49(1), 72–100.

    Article  Google Scholar 

  17. Croushore, D.D. (1993). Introducing: the Survey of Professional Forecasters. Business Review - Federal Reserve Bank of Philadelphia Nov / Dec 1993.

  18. Croushore, D.D. (1997). The Livingston Survey: still useful after all these years. Business Review - Federal Reserve Bank of Philadelphia Mar / Apr 1997.

  19. Cumby, R.E., & Modest, D.M. (1987). Testing for market timing ability: a framework for forecast evaluation. Journal of Financial Economics, 19(1), 169–189.

    Article  Google Scholar 

  20. De Bandt, O., Barhoumi, K., & Bruneau, C. (2010). The international transmission of house price shocks. In De bandt, O., Knetsch, T., Peñalosa, J., & Zollino, F. (Eds.) Housing Markets in Europe - A Macroeconomic Perspective (pp. 129–158): Springer.

  21. DiPasquale, D. (1999). Why don’t we know more about housing supply? The Journal of Real Estate Finance and Economics, 18(1), 9–23.

    Article  Google Scholar 

  22. Fawcett, T. (2004). ROC graphs: Notes and practical considerations for researchers. Machine Learning, 31(1), 1–38.

    Google Scholar 

  23. Fawcett, T. (2006). An introduction to ROC analysis. Pattern Recognition Letters, 27(8), 861–874.

    Article  Google Scholar 

  24. Federal Reserve Bank of Philadelphia. (2015). Survey of Professional Forecasters: Documentation. Philadelphia, USA, Last updated: December 16, 2015.

  25. Fritsche, J., Döpke, U., & Pierdzioch, C. (2017). Predicting recessions with boosted regression trees. International Journal of Forecasting forthcoming.

  26. Garcês, P.M., & Pires, C.P. (2011). New housing supply: what do we know and how can we learn more. Working Paper 2011/18, Centro de estudos e formação avançada em gestão e economica (CEFAGE).

  27. Ghent, A.C., & Owyang, M.T. (2010). Is housing the business cycle? Evidence from us cities. Journal of Urban Economics, 67(3), 336–351.

    Article  Google Scholar 

  28. Ghysels, E., Plazzi, A., Torous, W.N., & Valkanov, R. (2013). Forecasting real estate prices. In Elliot, G., & Timmermann, A. (Eds.) Handbook of Economic Forcasting: Vol. II (pp. 509–580): Elsevier.

  29. Goodhart, C., & Hofmann, B. (2008). House prices, money, credit, and the macroeconomy. Oxford Review of Economic Policy, 24(1), 180–205.

    Article  Google Scholar 

  30. Goodman, J. (1994). Using attitude data to forecast housing activity. Journal of Real Estate Research, 9(4), 445–454.

    Google Scholar 

  31. Greer, M. (2003). Directional accuracy tests of long-term interest rate forecasts. International Journal of Forecasting, 19(2), 291–298.

    Article  Google Scholar 

  32. Greiner, M., Pfeiffer, D., & Smith, R. (2000). Principles and practical application of the receiver-operating characteristic analysis for diagnostic tests. Preventive Veterinary Medicine, 45(1), 23–41.

    Article  Google Scholar 

  33. Gupta, R., Kabundi, A., & Miller, S. (2011a). Using large data sets to forecast house prices: A case study of twenty US states. Journal of Housing Research, 20(2), 161–190.

    Google Scholar 

  34. Gupta, R., Kabundi, A., & Miller, S.M. (2011b). Forecasting the US real house price index: Structural and non-structural models with and without fundamentals. Economic Modelling, 28(4), 2013– 2021.

    Article  Google Scholar 

  35. Hanley, J.A., & McNeil, B.J. (1982). The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology, 143(1), 29–36.

    Article  Google Scholar 

  36. Hanley, J.A., & McNeil, B.J. (1983). A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology, 148 (3), 839–843.

    Article  Google Scholar 

  37. Henriksson, R.D., & Merton, R.C. (1981). On market timing and investment performance. II. Statistical procedures for evaluating forecasting skills. Journal of Business, 54(4), 513–533.

    Article  Google Scholar 

  38. Jordà, Ò. (2014). Assessing the historical role of credit: Business cycles, financial crises and the legacy of Charles S. Peirce. International Journal of Forecasting, 30 (3), 729–740.

    Article  Google Scholar 

  39. Jordà, Ò., & Taylor, A.M. (2011). Performance evaluation of zero net-investment strategies. Working Paper 17150, National Bureau of Economic Research.

  40. Jordà, Ò., & Taylor, A.M. (2012). The carry trade and fundamentals: Nothing to fear but feer itself. Journal of International Economics, 88(1), 74–90.

    Article  Google Scholar 

  41. Khandani, A.E., Kim, A.J., & Lo, A.W. (2010). Consumer credit-risk models via machine-learning algorithms. Journal of Banking & Finance, 34(11), 2767–2787.

    Article  Google Scholar 

  42. Kolb, R., & Stekler, H.O. (1996). How well do analysts forecast interest rates? Journal of Forecasting, 15(5), 385–394.

    Article  Google Scholar 

  43. Kydland, F.E., Rupert, P., & Sustek, R. (2012). Housing dynamics over the business cycle. Working Paper 18432, National Bureau of Economic Research.

  44. Lahiri, K., & Wang, J.G. (2013). Evaluating probability forecasts for GDP declines using alternative methodologies. International Journal of Forecasting, 29(1), 175–190.

    Article  Google Scholar 

  45. Leamer, E.E. (2007). Housing is the business cycle. Working Paper 13428, National Bureau of Economic Research.

  46. Leung, C. (2004). Macroeconomics and housing: a review of the literature. Journal of Housing Economics, 13(4), 249–267.

    Article  Google Scholar 

  47. Liu, W., & Moench, E. (2016). What predicts US recessions? International Journal of Forecasting, 32(4), 1138–1150.

    Article  Google Scholar 

  48. Loh, C.P., & Mason, P. (2014). Housing and non-housing CPI components: The national case versus a local example. Journal of Housing Research, 23(2), 163–176.

    Google Scholar 

  49. Lusted, L.B. (1960). Logical analysis in Roentgen diagnosis: Memorial Fund lecture 1. Radiology, 74(2), 178–193.

    Article  Google Scholar 

  50. Majnik, M., & Bosnić, Z. (2013). ROC analysis of classifiers in machine learning: A survey. Intelligent Data Analysis, 17(3), 531–558.

    Article  Google Scholar 

  51. Mason, I. (1982). A model for assessment of weather forecasts. Australian Meteorological Magazine, 30(4), 291–303.

    Google Scholar 

  52. Mason, S.J., & Graham, N.E. (2002). Areas beneath the relative operating characteristics (ROC) and relative operating levels (ROL) curves: Statistical significance and interpretation. Quarterly Journal of the Royal Meteorological Society, 128(584), 2145–2166.

    Article  Google Scholar 

  53. Merton, R.C. (1981). On market timing and investment performance. I. An equilibrium theory of value for market forecasts. Journal of Business, 54(3), 363–406.

    Article  Google Scholar 

  54. Miles, W. (2009). Housing investment and the US economy: How have the relationships changed? Journal of Real Estate Research, 31(3), 329–349.

    Google Scholar 

  55. Mishkin, F.S. (2007). Housing and the monetary transmission mechanism. Working Paper 13518, National Bureau of Economic Research.

  56. Nneji, O., Brooks, C., & Ward, C.W. (2013). House price dynamics and their reaction to macroeconomic changes. Economic Modelling, 32, 172–178.

    Article  Google Scholar 

  57. Paciorek, A. (2013). Supply constraints and housing market dynamics. Journal of Urban Economics, 77, 11–26.

    Article  Google Scholar 

  58. Peirce, C.S. (1884). The numerical measure of the success of predictions. Science, 4(93), 453–454.

    Article  Google Scholar 

  59. Pepe, M.S. (2003). The statistical evaluation of medical tests for classification and prediction. London: Oxford University Press.

    Google Scholar 

  60. Pesaran, M.H., & Timmermann, A. (1992). A simple nonparametric test of predictive performance. Journal of Business & Economic Statistics, 10(4), 461–465.

    Google Scholar 

  61. Peterson, W.W., & Birdsall, T.G. (1953). The theory of signal detectability: part I. The general theory. Technical Report 13, University of Michigan.

  62. Pierdzioch, C. (2015). A note on the directional accuracy of inflation-rate forecasts. Applied Economics Letters, 22(13), 1073–1077.

    Article  Google Scholar 

  63. Pierdzioch, C. (2016). Using ROC techniques to measure the effectiveness of foreign exchange market interventions. Applied Economics Letters, 23(6), 389–393.

    Google Scholar 

  64. Pierdzioch, C., & Rülke, J.C. (2015). On the directional accuracy of forecasts of emerging market exchange rates. International Review of Economics & Finance, 38, 369–376.

    Article  Google Scholar 

  65. Pierdzioch, C., Rülke, J.C., & Stadtmann, G. (2012). Housing starts in canada, Japan, and the United States: Do forecasters herd? The Journal of Real Estate Finance and Economics, 45(3), 754–773.

    Article  Google Scholar 

  66. Pierdzioch, C., Rülke, J.C., & Stadtmann, G. (2013a). Forecasting US housing starts under asymmetric loss. Applied Financial Economics, 23(6), 505–513.

    Article  Google Scholar 

  67. Pierdzioch, C., Rülke, J.C., & Stadtmann, G. (2013b). House price forecasts, forecaster herding, and the recent crisis. International Journal of Financial Studies, 1(1), 16–29.

    Article  Google Scholar 

  68. Pierdzioch, C., Rülke, J.C., & Stadtmann, G. (2013c). On the internal consistency of the term structure of forecasts of housing starts. Applied Economics Letters, 20(9), 847–851.

    Article  Google Scholar 

  69. Puri, A.K., & Van Lierop, J. (1988). Forecasting housing starts. International Journal of Forecasting, 4(1), 125–134.

    Article  Google Scholar 

  70. R Core Team. (2017). R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria, https://www.R-project.org/.

  71. Rosenthal, S.S. (1999). Housing supply: The other half of the market. A note from the editor. The Journal of Real Estate Finance and Economics, 18(1), 5–7.

    Article  Google Scholar 

  72. Schnader, M.H., & Stekler, H. (1990). Evaluating predictions of change. Journal of Business, 63(1), 99–107.

    Article  Google Scholar 

  73. Shiller, R.J. (2014). Speculative asset prices. American Economic Review, 104 (6), 1486–1517.

    Article  Google Scholar 

  74. Sklarz, M.A., Miller, N.G., & Gersch, W. (1987). Forecasting using long-order autoregressive processes: an example using housing starts. Real Estate Economics, 15 (4), 374–388.

    Article  Google Scholar 

  75. Stein, R.M. (2005). The relationship between default prediction and lending profits: Integrating ROC analysis and loan pricing. Journal of Banking & Finance, 29 (5), 1213–1236.

    Article  Google Scholar 

  76. Stekler, H.O. (1994). Are economic forecasts valuable? Journal of Forecasting, 13(6), 495–505.

    Article  Google Scholar 

  77. Stekler, H.O., & Schnader, M. (1991). Evaluating predictions of change: an application to inflation forecasts. Applied Financial Economics, 1(3), 135–137.

    Article  Google Scholar 

  78. Swets, J.A. (1961). Detection theory and psychophysics: a review. Psychometrika, 26(1), 49–63.

    Article  Google Scholar 

  79. Swets, J.A. (1973). The relative operating characteristic in psychology. Science, 182(4116), 990–1000.

    Article  Google Scholar 

  80. Taylor, J.B. (2007). Housing and monetary policy. Working Paper 13682, National Bureau of Economic Research.

  81. Vargas-Silva, C. (2008). Monetary policy and the US housing market: a VAR analysis imposing sign restrictions. Journal of Macroeconomics, 30(3), 977–990.

    Article  Google Scholar 

  82. Youden, W.J. (1950). Index for rating diagnostic tests. Cancer, 3(1), 32–35.

    Article  Google Scholar 

  83. Yunus, N. (2015). Trends and convergence in global housing markets. Journal of International Financial Markets, Institutions and Money, 36, 100–112.

    Article  Google Scholar 

Download references

Acknowledgements

I thank an anonymous reviewer for very helpful comments and suggestions. The Chair of Monetary Economics at the Helmut Schmidt University has provided invaluable feedback.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Tim Meyer.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Meyer, T. On the Directional Accuracy of United States Housing Starts Forecasts: Evidence from Survey Data. J Real Estate Finan Econ 58, 457–488 (2019). https://doi.org/10.1007/s11146-017-9637-9

Download citation

Keywords

  • Housing starts
  • Forecasts
  • Survey data
  • Directional accuracy
  • ROC techniques