Introduction

Housing costs such as rental or mortgage costs are one of the most onerous expenses faced by households in the United Kingdom. Both private renting and home ownership has become a growing societal concern. According to the English Housing Survey 2018, only 38% of 25–34-year-olds own a house down from 55% in 2007. The average age of first-time buyers has also increased from 31 to 34. There has also been an increase in homelessness, with a weak supply of houses being coined as one of the many reasons. One of the perennial factors for these societal ills is long-term house price inflation, which, in standard economic theory, is determined by the forces of demand and supply. Any factor ranging from demography, stamp duty taxes, mortgage credit expansion, monetary or fiscal policy is likely to influence house price volatility. According to the present value approach, all these factors can be reflected in the price to rent ratio as it measures the current and future expectations of agents in the economy.

The price to rent ratio or rental yield can be used to assess property markets in different countries or subregions. Both housing and rental markets are spatially polarised (Hochstenbach and Arundel 2020; Clark and Lomax 2020) which further motivates the present value approach specific to a region. The geographical dimension to property value (a flat in London may cost more than a similar flat in Liverpool) offers an alternative dimension as to why the price to rent ratio differs. Most analyses on rental yields have focussed essentially on England geographically, as it dominates the overall UK figures. Recently, there has been an increase in good quality data at individual property levels in different counties, enabling more reliable estimations.

The Scottish evidence is currently lacking, although there is evidence of price and rent differentials. Private rent sector statistics published by the Scottish governmentFootnote 1 show an increase in rents for 15 out of 18 counties between 2017 and 2018. The growth in rent from 2010 to 2018 in Scotland (21.6%) has outpaced UK CPI (18.7%). Unlike England and Wales, one of the most interesting facts is the spatial heterogeneity in terms of rental growth. For instance, Glasgow (one of the most populous areas) witnesses the highest rental growth compared to Aberdeen (third most dense city), which posts one of the slowest rental growth.

An important factor that motivates this study is devolution. The Scottish Experience is interesting because of devolved housing policy (McKee et al. 2017). Over time, since the early 2000s, Scottish housing has developed its own landscape. Much of the research on Scottish housing has geared towards policy and how devolution may have affected ownership. As an example, housing policy in Scotland is supportive of low-income families with benefits and tax credit, and there are considerable investments in social housing. Friendly policies such as Help to Buy has increased the supply of new builds (Gibb 2015), which has also led to increased house prices. The critical reforms to the PRS (Private Renting sector) have also meant that there has been a surge in private renting.

Since devolution, the focus on housing has been mostly on policy to reduce homelessness through social housing. Home ownership is affected by a progressive stamp duty taxation system and council tax bands. On the supply side, grant funding has geared towards increasing social housing and low-cost home ownership. The financial crisis of 2007–09 led to a fall in the volume of transactions of houses. The growth in the post-crash is sluggish, with affordability to rent and owning becoming a serious issue.Footnote 2

This paper answers the question of what moves house prices in Scotland by collating two datasets and then deriving time series for discount rates and expected rent growth for Scottish cities using a state-space approach. This paper does not formally investigate why expected returns and expected rent growth differ across the regions, and as such does not make any causal inference. In real estate economics, current house prices should represent a discounted sum of utility from owning the house, which is often measured by the rental value. Any changes in house prices can be traced to either the discount rate, expected future rent or a combination of both. One of the predictions of this model is that it can reconcile both market and behavioural explanations to market movements. Empirically, the present value can be estimated using sign restricted Vector Autoregressions (VAR) models based on the log-linearisation approach of Campbell and Shiller (1988). Most of the literature in North America and Europe suggest that the link between the house prices and rents may have been reduced because of future expectations of capital gains.

In this study, the dynamics of rent growth and expected returns are explained in Scotland through a state-space present value approach, which is used in stock prices by Binsbergen and Koijen (2010), Ma and Wohar (2014), Rambaccussing and Power (2018a, b). Using this approach, discount rates and expected rent growth are latent (unmeasurable contemporaneously), time-varying and are assumed to follow an autoregressive process of order one (AR (1)). It links them to the price–rent ratio through the dynamic present value equation. The model is optimised to provide an implicit time series of the discount rate and expected future rent for each Scottish city and region under investigation.

Section 2 of this paper provides some of the recent trends in rents and house prices in the United Kingdom. Section 3 introduces the data and the state-space method to estimate expected returns and expected rent growth rate. Section 4 presents the results of the state-space model and explains the differences. Finally, Sect. 5 discusses these findings.

Background literature

The price-rent ratio is perhaps one of the better candidates for forecasting of house prices (Ghysels et al. 2013). Many have investigated the predictability and movement of house prices in US and OECD countries using decompositions of this ratio. Campbell et al. (2009) decompose the expected returns component into the housing premium and the interest rate. The housing premium component is found to be unforecastable in US metropolitan areas. However, the covariation among the rent growth, interest rate and housing premium appear to account for the volatility of house prices. Similarly, Hiebert and Sydow (2011) show that fundamentals seem to move the market in Europe. The variation in expected returns (future premia) plays a more important role than the variation in rent growth. Changes in the price–rent ratio are negatively correlated with expected future returns, which is the primary driver of house price volatility. In a similar vein, Engsted and Pedersen (2015) show that the rent–price ratio forecasts better the housing returns and rent growth when measured in real terms. They explain the differentials through money illusion.

The fundamental present value approach is also used in the identification of housing bubbles (Engsted and Nielsen 2012; Kivedal 2013; Liu et al. 2014). Coskun et al. (2020) show the absence of bubbles but not overvaluation. They trace house price increases to market conditions and investment psychology. Bangura and Lee (2020) investigate bubbles in different regions in Greater Sydney using a plethora of approaches, including testing for a cointegrating relationship between price and rent. Evidence of a bubble fuelled by speculative behaviour was noted in Western Sydney. Martinez-Garcia and Grossman (2020) show that real house prices in the US can display nonlinear and mildly excessive behaviour, which can be linked to financial market development.

The state-space approach has been used in the real estate literature. Plogmann et al. (2020) analyse the price–rent ratio of agricultural land across various states in Germany. The findings are that the price–rent ratio is lower than real estate and show more significant variation across geographical states. The decomposition of the price–rent ratio shows that land price volatility is driven by changing expectations of returns. The rent premium, however, can be negative. Kim and Lim (2014) decompose the price–rent ratio using the Campbell–Shiller formula, and show that the principal variation in house prices in Ireland is the expected returns, which account for nearly 500% of the total variation. The real interest rate and the expected rent growth lowers the volatility of the ratio.

The present value approach has also been used in studies in the United Kingdom, but with the principal focus being England, which inherits a wealth of data in terms of house sales across postcodes. Differences in the price–rent ratio in England are explored by Clark and Lomax (2020), who show that rental yields differ across property types (flats and number of bedrooms) and geographical areas, with better and healthier areas commanding a lower rent–price ratio. Bracke (2015) uses rental and sale transactions in Central London in the period 2006–2012 to show that the rent–price ratios are different across properties (bigger properties having higher price–rent ratios) and neighbourhood. Bracke et al. (2018) estimate the discount rate for prime Central London houses using historical datasets on leases to show that discount rates have been on the decline. Howard and Liebersohn (2020) found that expected rental income plays a key role in driving house prices. They also found regional inequalities to be drivers of higher average rents and hence current house prices.

With respect to the state-space approach in the real estate literature, Kishor and Morley (2015) have applied the model to US metropolitan cities. Schulz and Werwatz (2004) also use the state-space present value model in a hybrid model of hedonic pricing, with the objective of using the Kalman filter to extract the common price component in the housing market in Berlin. Other papers which have used different dimensions of the state-space approach to housing markets include Riddel (1999), Gabrieli et al. (2018), Hui and Yue (2006) and Teng et al. (2013).

Trends in house prices and rents

House prices differ across the regions in the United Kingdom. Figure 1 plots the time series of changes in average house prices across the different countries of the United Kingdom.

Fig. 1
figure 1

(Source: UK House Price Index, Her Majesty’s Land Registry)

Plot of house prices growth in the different geographical regions in the United Kingdom for the period January 2000 until December 2018. The vertical axis represents the monthly change for the average of all property types, with the horizontal axis representing the months. The data can be retrieved from https://landregistry.data.gov.uk/app/ukhpi (accessed June 2020)

Full size image

England drives the trend in UK average house price changes, sharing a correlation of 0.979 with the national average at the 1% significance level. Scottish house price changes appear to be the most volatile, with a correlation of 0.54 at the 1% significance level. The monthly rate of growth for the UK is at 0.29%, while that of Scotland is only 0.13%. However, Scotland has the highest growth volatility across the different countries.

Figure 2 plots the percentage change in the rental index. The rental index in levels shows an upward trend for all the countries, but with a steeper slope for England. The correlation in the upward trend is slightly lower than in the case of the house values; England and Scotland have a correlation of 0.85 at the 1% significance level. There is a distinct divergence at the beginning of January 2015, where increases in rents are slower and become marginally negative in 2016. As a result of this decline, the volatility of Scottish rental changes is the highest across the UK (0.70). Rental growth is higher in England, averaging a 2.05% increase each month. As depicted in the graph, the correlation of the changes is rather weak across the different regions; for instance, Scotland and England have a correlation of 0.08.

Fig. 2
figure 2

(Source: Index of Private Housing Rental Prices (IPHRP), Office for National Statistics)

Plot of rental change (%) in the different regions of the UK (The time series for Great Britain has been reported instead of the UK due to the short Northern Irish time series) for the period January 2012 until December 2018. The y-axis represents the percentage change. The data can be retrieved from https://www.ons.gov.uk/file?uri=/economy/inflationandpriceindices/bulletins/indexofprivatehousingrentalprices/march2020/7d0bb861.xls (accessed June 2020)

Full size image

Figure 3a and b plots the histogram of annual quoted rent and house prices, respectively. There is essentially enough spread in both rental and house prices across cities to model the price–rent ratio individually in these cities. The distributions for each city differ in terms of mass and spread. In terms of rent, the collective distribution has three peaks with more spread for Edinburgh at the upper end. Dundee has lowest rent followed closely by Glasgow. Dundee also has the lowest house price, followed by Glasgow. Comparatively, Aberdeen and Edinburgh have a much wider spread in house prices. There is enough in the mix of distributions and individual distributions to hypothesise different patterns of growth (movement) in house price and rent across regions in Scotland.

Fig. 3
figure 3

(Source: Citylets Market Rental Reports)

a Histogram of average house prices (£) in Scottish cities for the period 2008–2018. https://www.ros.gov.uk/data-and-statistics/house-price-statistics (Accessed June 2020). (Source: Registers of Scotland). b Histogram of average house prices in Scottish cities. https://www.citylets.co.uk/research/datahub/ (Accessed in May 2020)

Full size image

The three major cities having a buoyant housing market are Glasgow, Edinburgh and Aberdeen. Historically, houses in Aberdeen and Edinburgh have higher prices due to the oil industry and tourism in these cities, respectively. Glasgow is denser and has an east and west end divide in housing prices.

According to Table 1, growth in rent (25.61%) has exceeded the growth in house price (17.27%) in Scotland. Buy to let schemes have proved popular over this period. In line with the national figures in Scotland, Dundee and Edinburgh have higher rental growth. However, this trend is not uniform across Scotland. For instance, Aberdeen and Glasgow witness the opposite experience where house price growth outpaced rents. The oil crash of late 2014 has left Aberdeen brittle in terms of house prices, where the mean rental income was negative.

Table 1 Summary statistics on house prices and rent
Full size table

Results of the state-space model

Present value

In this section, we expand on the log-linearised present value model before introducing the state-space model. Denoting \({X}_{t}\) and \({P}_{t}\) as the respective rental price and market price of the average property (in each town and region) at time t, the returns from ownership from \(t\) to \(t+1\) (\({r}_{t+1})\), the rent growth from \(t\) to \(t+1\) (\(\Delta {x}_{t+1})\) and the logarithm of the price–rent (\({px}_{t}\)) is defined:

$${r}_{t+1}=\mathrm{ln}\left(\frac{{P}_{t+1}+{X}_{t+1}}{{P}_{t}}\right),$$
(1)
$${px}_{t}=\mathrm{ln}\left(\frac{{P}_{t}}{{X}_{t}}\right),$$
(2)
$$\Delta {x}_{t+1}=\mathrm{ln}\left(\frac{{X}_{t+1}}{{X}_{t}}\right).$$
(3)

(1)–(3) is measured directly from the data. However, expected rent growth (\({g}_{t+1})\) and expected returns (\({\mu }_{t+1})\) from household sales are unobserved at time t. \({\mu }_{t+1}\) and \({g}_{t+1}\) can be modelled by imposing an Autoregressive Moving Average (ARMA) structure onto the model. The sample size (which will be discussed in a later section) provides a valid justification to provide a stable autoregressive specification for expected returns and expected rent growth:

$${\mu }_{t+1}-{\phi }_{\mu ,0}={\phi }_{\mu ,1}\left({\mu }_{t}-{\phi }_{\mu ,0}\right)+{\varepsilon }_{\mu ,t+1} ,$$
(4)
$${g}_{t+1}-{\phi }_{g,0}={\phi }_{g,1}\left({g}_{t}-{\phi }_{g,0}\right)+{\varepsilon }_{g, t+1.}$$
(5)

Equations (4) and (5) are demeaned autoregressive equations for expected returns and expected rent growth. \({\phi }_{\mu ,1}\) and \({\phi }_{g,1}\) are assumed to be stable and are less than the absolute value of one. \({\varepsilon }_{\mu ,t+1}\) and \({\varepsilon }_{g,t+1}\) are normally identically distributed. The dynamic present value based on Campbell and Shiller (1988) can be written as:

$${px}_{t}=\kappa +\rho {px}_{t+1}+\Delta {x}_{t+1}-{r}_{t+1}.$$
(6)

After substituting (4) and (5) into Eq. (6), we can write the observed price to rent ratio in terms of the unobserved expected returns and expected rent growth:

$${px}_{t}\simeq {A}_{1}+{A}_{2}\widehat{{\mu }_{t}}+{A}_{3}\widehat{{g}_{t,}}$$
(7)

where \({A}_{1}=\frac{\kappa }{1-\rho }+\frac{{\phi }_{g,0}-{\phi }_{\mu ,0}}{1-\rho }\), \({A}_{2}=\frac{1}{1-\rho {\phi }_{\mu ,1}}\), \({A}_{3}=\frac{1}{1-\rho {\phi }_{g,1}}\). \(\widehat{{\mu }_{t}}\) and \(\widehat{{g}_{t}}\) are demeaned variables. An identification assumption is that realized rent growth and expected rent growth are close to each other:

$$\Delta {x}_{t+1}={g}_{t}+{\upvarepsilon }_{\mathrm{x},\mathrm{t}+1},$$
(8)

where \({\upvarepsilon }_{\mathrm{x},\mathrm{t}+1}\) and \({g}_{t}\) are assumed to be orthogonal to each other.

To implement the Kalman filter to derive the unobserved expected returns and expected rent growth, the state-space equation is written as:

Measurement equations:

$$\Delta {x}_{t+1}={g}_{t}+{\upvarepsilon }_{\mathrm{x},\mathrm{t}+1},$$
$${px}_{t}\simeq {A}_{1}+{A}_{2}\widehat{{\mu }_{t}}+{A}_{3}\widehat{{g}_{t}}.$$

(4) and (5) give the state equations.

The unknown parameters (intercepts and autoregressive coefficients, standard errors and correlations) can be estimated from the data. The vector of parameters to be estimated is:

$$\Phi =\left({\phi }_{\mu ,0}, {\phi }_{g,0}, {\phi }_{\mu ,1}, {\phi }_{g,1}, {\sigma }_{\mu }, {\sigma }_{g}, {\sigma }_{x}, {\rho }_{g\mu },{\rho }_{x\mu }, {\rho }_{xg}\right)$$

Some practical aspects of the state-space procedure include the identification of the model. For the model to be identified, we must fix one parameter at an arbitrary value. The correlations are potential candidates. We consider setting either \({\rho }_{g\mu },{\rho }_{x\mu }\) or \({\rho }_{xg}\) equal to zero. In the estimation, \({\rho }_{xg}\) is set to zero. The model is cast in matrix form, and after applying the Kalman Filter the likelihood is derived and optimised.

Construction of dataset

The construction of the dataset on the price to rent ratio is constructed by matching two datasets. Data on rent were collected from the quarterly reports of CityLets,Footnote 3 a search gateway for houses to rent. The recent reports also include monthly rental prices for different properties (based on the number of rooms) and smaller towns. The rents comprise the average rent advertised by landlords for houses that were sublet during the quarter. The exact measure of rent is the ultimate figure when the rent advert leaves the advertising website (when an offer is made).Footnote 4 Quarterly observations of average house prices were collected from the Registers of Scotland, which contains house price data on houses of different sizes across Scotland. When the median of house prices (instead of the mean) was considered, no major differences in correlations across cities in the logarithmsFootnote 5 of the values was noted. The data were collected from the first quarter of 2008 until the second quarter of 2019.

Results and discussion

Estimation of the present value model

The results from the optimisation of the present value are provided in Table 2.

Table 2 Estimation of the present value model
Full size table

There are considerable differences in the intercept term for both rent growth and expected returns. Aberdeen encounters a negative rent growth (− 0.4%), which stands in contrast to a higher intercept in the case of Edinburgh (0.9%), which is higher than the average encountered in Scotland (0.6%). Interestingly the findings show that there is not much disparity in expected returns in the housing market across Scotland—where 6% seems to be the norm.

However, the cities exhibit diversity in the autoregressive parameters in the expected return and growth series. In Scotland, the expected return (0.56) is more persistent than expected rent growth (0.28). Aberdeen, which has witnessed falling rental rates over the years, exhibits high persistence at 0.74. At the other extreme, Glasgow has negative persistence, which implies that shocks to expected rent growth disappear in an oscillatory fashion. Edinburgh is also characterised by a low persistence of expected rent growth. One potential reason which accounts for the latter fact is that expected growth is increasing in the intercept, and shocks do not have an everlasting impact on the mean growth. This would show a consistently growing rental market with positive premiums over time. As a contrast, Dundee and Glasgow have moderate persistence of 0.33 and 0.12, respectively. A low level of persistence, coupled with the high unconditional term, suggests that the rental market remains buoyant and shocks emanating from policies or news would not have a lasting effect.

Expected returns for Aberdeen witness the highest persistence, which can be attributed to the high demand for housing in the city and the build-up prior to the housing crash following the oil shock of 2014. Scotland and the remaining cities have persistence levels ranging from 0.4 to 0.6. The persistence parameters reflect the attractiveness of buying into real estate for capital gain potential. The correlation between rent growth and expected returns differs across regions in Scotland. Edinburgh stands out as a city with a strong negative correlation between expected rent and returns, which implies that the change of the price–rent ratio is more likely to be skewed towards one extreme. The estimated parameters show that there are considerable differences in the correlations.

Patterns in expected rent growth and expected returns

The correlation across the discount rates across the four cities and Scotland is computed and illustrated in Table 3.

Table 3 Correlation in expected returns and rent growth
Full size table

Table 3 displays high correlation patterns between Edinburgh and the Scottish average for both expected returns and expected rent growth. The correlations vary across the different cities. The time series of expected rent growth is positively correlated across the cities, except for Glasgow, the only West Coast City. Geographically, the East coast cities (Edinburgh, Dundee, and Aberdeen) correlate with values ranging from 0.22 to 0.27. These cities are, however, engaged with different levels of economic activity. Unlike the general UK or English case, the commute to neighbouring cities is not excessively long. Individuals may live in a neighbouring city where the rent is more affordable and commute to their job elsewhere.

The positive correlation in expected returns can be traced to the nationwide macroeconomic trends in the United Kingdom. However, the low magnitude across the cities is due to the nature of economic activity levels in these cities. Dundee and Edinburgh share high correlations with the average Scottish level at 0.32 and 0.84, respectively. It is interesting to note the geographical factors underlying the correlations: cities closer to each other in terms of railway links have higher correlations. Notable examples are between Edinburgh and Dundee and between Dundee and Aberdeen.

Decomposition of house price movements

The variance of the price to rent ratio from Eq. (7) can be decomposed into the factors that move house prices: the expected return component, the expected rent growth component, or the covariance between both:

$${\mathrm{var}(px}_{t})\simeq {A}_{2}^{2}\mathrm{var}\left(\widehat{{\mu }_{t}}\right)+{A}_{3}^{2}\mathrm{var}\left(\widehat{{g}_{t}}\right)+{A}_{2}{A}_{3}\mathrm{cov}\left(\widehat{{\mu }_{t}},\widehat{{g}_{t}}\right),$$

where the values of A2 and A3 are as previously determined.

Table 4 shows the variation is not uniform across the cities. Expected return predominantly drives house prices in Glasgow (99.8%). Aberdeen and Dundee show it is mostly a mixture of both, with expected returns being stronger in the former (46.9% as opposed to 33.3%) and expected rent growth (31% as opposed to 48.3%) appears the more interesting factor for the latter. Essentially, expected rent growth is the key driver of Edinburgh and Scotland. With Edinburgh, the covariance is negative, which means it moderates the overall variation in the price–rent ratio for the present value emanating from the higher share of expected rent growth.

Table 4 Sources of movements in residential prices in Scotland
Full size table

The findings from Table 4 depart from findings in the literature studied in other states and countries. Kishor and Morley (2015) suggest that discount rates mostly drive the changes in the house prices of metropolitan cities in the US and metropolitan cities. They found that the variation explained by expected rent growth is no higher than 6.5%, with expected housing return accounting for over 100%. The findings of the model show that the decomposition in Scotland is different, with rental income appearing as the most usual source of variation. The reason for this finding is in the nature of the workforce. Cities such as Edinburgh employ workers on fixed-term contracts, reducing their need to buy a house, whereby pushing the demand in the rental market.

Conclusion

This paper is relevant for policy-making purposes. This research advocates a geographical dimension to help solve the “generation rent” problem. Cities such as Glasgow and, to a lesser extent, Aberdeen should aim towards helping first-time buyers in securing mortgage deposits to improve affordability. In a similar vein, it is worth rethinking mortgage cost practices, as well as policies regarding stamp duties on ownership of over one house. A city like Aberdeen, which witnessed a housing market crash in 2014, has witnessed high persistence in expected returns but weaker persistence in rent growth, suggesting that this may have led to a housing bubble. However, Edinburgh does not incur speculative pressures as the increased house prices are due to rent growth.

The contribution of the state space to the literature is to allow time variation in expected returns and expected rent growth, which conforms to owners’ and investors’ actions. It is unlikely that expected returns are constant in the wake of unconventional monetary policy and changing laws. This paper shows that the case of Scotland differs from findings in other countries such as the United States. Expected returns and rent growth vary, and their variation account for movements in house prices. A finding of interest is the degree of spatial polarisation (Zietz and Sirmans 2004) in Scotland, where each city and council area is distinct from each other.

An important point conveyed in this paper is that the Scottish market possesses features that differ from both the English and the international experience. The contribution of this paper essentially points out the role of city and region-specific features which drive rent and capital gain. It essentially points out the important role played by geography in housing movements, noting that economic conditions and policies affect the country in the same way. This paper does not investigate why expected returns and expected rent growth differ across cities.