Who’s miserable now? Identifying clusters of people with the lowest subjective wellbeing in the UK

Abstract

Policymakers are generally most concerned about improving the lives of the worst-off members of society. Identifying these people can be challenging. We take various measures of subjective wellbeing (SWB) as indicators of the how well people are doing in life and employ Latent Class Analysis to identify those with greatest propensity to be among the worst-off in a nationally representative sample of over 215,000 people in the United Kingdom. Our results have important implications for how best to analyse data on SWB and who to target when looking to improve the lives of those with the lowest SWB (The authors owe a massive debt of gratitude to the Office for National Statistics for their support throughout this research. We are particularly grateful to Dawn Snape and Eleanor Rees for their valuable comments on earlier drafts of this paper, to Salah Mehad for the thorough review of methodology, and to Vahe Nafilyan for advice on clustering analysis. We also thank the anonymous reviewers for the very helpful comments. Thank you all very much.).

1 Introduction

Over the past couple of decades, there has been increasing interest in going beyond standard economic measures of welfare, such as income, to consider wellbeing in a broader sense (Stiglitz et al. 2009). There has been a burgeoning of research using measures of subjective wellbeing (SWB) i.e. reports of how individuals think about their lives and feel as they go about them. This regression-based approach has yielded important insights. Key findings from the literature include a substantive negative association between unemployment and life satisfaction (Knabe et al. 2010), as well as a strong association between health and SWB (Steptoe et al. 2015).

The research to date suffers from three main limitations that directly impact its policy relevance. First, the work typically examines the relationship between a single measure of SWB, typically life satisfaction, and so it ignores the multidimensionality of wellbeing (Diener 1994; Diener et al. 1999; Kahneman and Riis 2005; Dolan and Kudrna 2016). Second, it provides insights into “average Joe’s” SWB (Binder and Coad 2011) and is largely silent on whether the main determinants of SWB differ for those with the lowest SWB, who will arguably matter most to policymakers. Third, it focuses on single determinants of SWB ceteris paribus, thus ignoring policy relevant interactions between determinants.

Against this background, the current work fills an important gap by analysing a cross-sectional dataset of over 215,000 people from the UK’s Annual Population Survey (APS). First, we use four measures of SWB included in the APS which span evaluative, experiential and eudemonic dimensions of wellbeing. Second, we focus on those who report the lowest SWB across all measures. Third, we employ Latent Class Analysis (LCA) to partition the sample into groups (classes) most similar to each other in terms of life circumstances (age, health, socio-economic status, SWB etc.) and explore their representation amongst the worst-off in society. As LCA allows us to identify “clusters” of determinants that are closely linked to reporting low SWB across all four SWB measures, the analysis provides us with valuable insights into who is miserable now.

According to our multidimensional definition of misery, we find that just over 1% of the APS sample are miserable. Our LCA results suggest that the overall sample can be reduced to seven main groups. We find that two of the seven classes have an above-average probability of being the most miserable: (1) unemployed/inactive people over 30 with severe health problems and/or a disability, who live in rental accommodation, are not in a partnership and have up to compulsory level education; and (2) 16–59 year olds, who are in employment and have GCSE or above compulsory level education but are facing some issues with health and disability, are not in a relationship, and who live in rental accommodation or have a mortgage. Policymakers interested in improving the lives of the worst-off in society should pay special attention to these two groups as they account for 96% of the miserable people in our sample.

The rest of this paper is organised as follows. Section 2 further details the background to the study. Section 3 describes the data and presents our methods. Section 4 presents the results, Finally, Sect. 5 discusses the findings in the context of the existing SWB literature.

2 Background

In the current work, we take a person-centred approach to examining the worst-off in society, defining the worst-off according to a novel definition of misery that spans different dimensions of SWB. In so doing, we build on existing literature in terms of what measures, who matters, and in what ways?

2.1 What measures?

Existing research typically focuses on single measures of SWB and SWB is an umbrella term for how people think about their lives and feel as they go about them (Diener et al. 2002). An important distinction in the literature is between evaluative measures, which elicit global reports of happiness or, more commonly, life satisfaction, and experience-based measures, which elicit more granular reports of happiness in the moment (Kahneman and Riis 2005). Some measures of SWB also tap into a “eudemonic” account of wellbeing, which assesses the purpose or meaning people have in their lives or experiences (Ryff 1989; Ryff and Keyes 1995; see Dolan and Kudrna 2016 for how to categorise the measures).

Most large-scale surveys around the world solely elicit evaluations of life satisfaction (e.g. World Values Survey, General Social Survey, German Socio-Economic Panel, and Understanding Society) although some additionally capture reports of SWB in the moment (most notably the American Time Use Survey). As such, these surveys provide only a partial picture of SWB. When the Office for National Statistics (ONS) in the UK began measuring wellbeing in 2011, it decided to elicit reports of evaluative, experiential and eudemonic wellbeing, thus facilitating a more comprehensive assessment of individuals’ SWB (Dolan and Metcalfe 2012). The four questions used are as follows:

• Overall, how satisfied are you with your life nowadays?

• Overall, to what extent do you feel the things you do in your life are worthwhile?

• Overall, how happy did you feel yesterday?

• Overall, how anxious did you feel yesterday?

The responses to all four SWB questions are measured on a 0–10 scale where 0 is “not at all” and 10 is “completely”. The ONS data allow for a multidimensional approach to modelling individuals’ SWB which spans evaluative and experiential measures, as well as both hedonic and eudemonic wellbeing types of wellbeing (Dolan and Metcalfe 2012). However, being restricted to four survey measures, the set does not tap into all of the dimensions of SWB that many would consider relevant, including sad or joyous moments, which would arguably be best assessed using naturalistic monitoring tools, or evaluations such as the meaning of one’s life overall (Stone and Mackie 2013). As a result, the ONS measures can be understood as provided a richer but still incomplete picture of SWB in comparison to that offered by many large-scale surveys.

2.2 Who matters?

Previous research has tended to look for average effects, treating a population as if it was a single representative citizen (O’Donnell and Oswald 2015). Some limited work using quantile regression techniques, however, demonstrates that looking at averages alone provides an incomplete and, at times, misleading picture of the relationship between SWB and determinants of interest at different points in the wellbeing distribution. For example, Binder and Coad (2011) find that education is positively associated with happiness at the bottom end of the wellbeing distribution but negatively so at the top, and Lamu and Olsen (2016) find that both income and health are relatively more important at the lower end of the wellbeing distribution.

Those with the lowest wellbeing will be of more concern to policymakers than the average citizen. Most “common-sense conceptions of justice” are seen to lie somewhere between the utilitarian social welfare function (SWF), which solely emphasises improving overall welfare as much as possible irrespective of its distribution, and the Rawlsian SWF, which focuses only on improving the welfare of the worst-off individual and disregards how efficiently resources might be used elsewhere (Elster 1993; see also Dolan and Tsuchiya 2011 for an empirical investigation of the SWF). The wellbeing of the worst-off is therefore a policy concern on distributional (equity-related) grounds.

Understanding the SWB of the worst-off also matters for efficiency reasons as it provides insights into how best to target resources at the worst-off. As existing research indicates that the determinants of SWB at the lower end of the distribution differ from those at the top, research comparing the determinants of SWB across the distribution can highlight the different strategies aimed at improving wellbeing at different points. Such research may also highlight the potential differential feasibility or costs of improving the wellbeing of the worst-off compared to those who already have high levels of SWB. If, for example, the determinants of wellbeing at the bottom of the distribution are such that policy interventions are likely to have limited potential impact, i.e. that the worst-off are inelastic suppliers of wellbeing, then policymakers need to be aware of this.

The ONS defines low wellbeing on each question according to the underlying distribution of the data. For the three positively framed questions, a score of four or less is deemed as low. For the anxiety question, a score of six or more is defined as having high levels of anxiety and therefore low wellbeing according to this measure. While the research community, policymakers, and individuals might disagree about which the best measure of SWB is, they will surely all agree that someone doing badly on all four questions has low wellbeing. Someone who reports low SWB on all four ways of tapping into it is clearly doing at least as badly as someone who reports low SWB if they were only asked one of those questions, and arguably worse. A person who reports being both dissatisfied with their lives overall and as well as experiencing unhappiness day to day is more miserable than their counterpart who is dissatisfied but relatively happy day to day.

Moreover, measuring SWB is a fuzzy concept: an individual has no objective indication of where to rate themselves on the scale (for a thorough discussion of fuzzy sets, and empirical examples of how much trouble people have even with notions of ‘tallness’ and ‘beauty’, see Norwich and Turksen 1984). Consider, for instance, an individual who is repeatedly asked to evaluate the same stimulus, e.g., SWB on a given day. Empirical evidence shows that 75% of the time she may rate it as a ‘5’, and 25% of time as a ‘4’. Hence, one-quarter of occasions she would fall into the miserable subset, although three-quarters of the time she is above the cut-off point. Defining misery over four questions creates a stricter criterion for identifying the individual as a member of the miserable population: an individual with the same response pattern would only be misclassified as miserable (1/0.25)⁴ = 1/256 of the time (assuming that the inconsistency in reporting is perfectly random and uncorrelated between measures).

A stricter definition of misery means that policymakers can be more confident in identifying the worst-off in society by mitigating noise in who they categorise as the most miserable (be that due to the fuzziness of the concept or simply the mistakes people may make when filling in a form). In this setup, the strategy of defining misery through n + 1 measures weakly dominates defining it through n measures. In other words, the stricter definition yields at least as good an outcome as the more lenient definition; and when individuals may make unintentional mistakes, it yields a superior outcome (for further discussion of strategies in response to such ‘trembling hand’ errors see Selten 1975). On this basis, we posit that identifying the worst-off individuals through all available SWB measures is the best strategy for examining misery.

Our focus on people who report low wellbeing across all four measures acknowledges the multifaceted nature of SWB, homing in on those that are doing badly across evaluative, experiential and eudemonic dimensions, while also mitigating potential noise in the classification of who is miserable. Importantly, as the ONS bases their definitions of low wellbeing for all four measures on the underlying wellbeing distribution it is worth highlighting that this group represents the worst-off in the population—a relative rather than an absolute measure of low SWB. As such a subgroup of the population will always fall into this category regardless of improvements to wellbeing, much like there are always people living in relative poverty.

2.3 In what ways?

The extant literature on SWB focuses on each determinant of wellbeing ceteris paribus and so it ignores “clusters” of determinants that interact with one another in important ways. What predicts the SWB of a middle-aged man, for example, may be different to what predicts the SWB of a retirement-age woman, and the relationship between SWB and being in poor health might vary greatly depending on a person’s level of income. Furthermore, there may be an unobserved factors (such as personality) that affect both the observable characteristics—such as health and socioeconomic status—and SWB. Although some SWB work does present subgroup analysis and interactions between specific factors, the challenge of treating the individual holistically remains largely unaddressed in the SWB literature (Clark et al. 2005).

The current work uses LCA to identify clusters of people within society that report low wellbeing. LCA is increasingly used to deal with the challenges of heterogeneity and endogeneity by allowing the latent (unobserved) characteristic to partition the data into clusters united by combinations of observable characteristics (see e.g., Anand et al. 2011; Brown et al. 2014; Clark et al. 2005; Fernandez-Blanco et al. 2009; Giovanis, 2014). LCA splits respondents into homogenous groups (latent classes) such that individuals in the same latent class will have similar response patterns to the independent variables while individuals across latent classes will have different response patterns to each other. The relationship between being in a certain class and an outcome of interest (such as being among the worst-off in terms of SWB) can then be examined. This approach is consistent with our contention that it is important, for academic and policy purposes, to think about SWB of whole people rather than exploring specific determinants ceteris paribus.

3 Data and methods

3.1 Data

This study uses the pooled¹ 3-year dataset of the ONS’s UK representative APS² covering the period April 2014–March 2016.³ The dataset contains observations for 284,456 adults, 280,003 (over 98%) of whom responded to all four SWB questions.

3.2 Methods

3.2.1 Latent class analysis

In step one, we perform Latent Class Analysis (LCA). This allows us to examine unobserved heterogeneity; that is, to identify groups of participants who represent the greatest similarity on the same set of observed variables within a given group and the greatest dissimilarity between other participants’ groups. LCA is a type of finite mixture model which makes it particularly well-suited for categorically scored data and variables with different scale types (Vermunt and Magidson 2003). For a more detailed comparison of alternative clustering approaches, see Appendix 6.2. LCA has been applied in research concerning a wide range of outcomes such as self-reported consumer taste (Fernandez-Blanco et al. 2009), financial satisfaction across life stages (Brown et al. 2014), and the relationship between self-reported wellbeing and recycling in Britain (Giovanis 2014).

Here we assume that a latent variable of “person’s life circumstances” (the X in Fig. 1) defines the combination (A, B, C, D in Fig. 1) of a person’s socioeconomic status (their employment status, education, etc.), their reference group (age, marital status, etc.) and the way they perceive their life (SWB and self-reported health). For full details of the model specification and the intuition behind the method, see Appendix 6.3. We assume the latent class to be “person’s life circumstances” and use as the manifest variables the major socio-economic and personal characteristics available in the dataset and traditionally used in SWB literature. These are age, sex, health and disability status, SWB, employment status, socio-economic status, housing tenure, marital status, education, and income.

Fig. 1

Generic path diagram for an unconditional latent class model

The LCA model assumes conditional independence. For two independent categorical variables—A (with categories) and B (with categories), the joint probability of observing both responses should be equal to the product of the probabilities of observing each response: Pr( Y₁₁ ∩ Y_jk) = Pr(Y₁₁) ⋅ Pr(Y_jk). We recognise that conditional independence of the observable characteristics is a rather bold assumption for our dataset. From the technical point of view, however, the literature agrees that the tenability of the local independence assumption may also be partially relaxed (see, e.g., Huang and Bandeen-Roche 2004). More importantly, from the conceptual point of view, it fits with our goal of being able to step back from the well-established negative correlations between SWB and bad health or unemployment, for example, and explore if there are groups of these factors that are associated with misery in combination.

In step two, we then examine the proportion of miserable people in each of the classes emerging from the LCA. It is common in LCA to use the model-driven partition to then compare the emergent classification on the variables used in the model (e.g., to verify significant differences between classes) as well as the variables not used in the clustering step (e.g., to explore further differences between classes).⁴ While we favour looking at the intersection of all four measures in our definition of misery, we also examine an alternative threshold of low life satisfaction—as the most commonly used singular measure of SWB (Diener et al. 2018) and the only measure which has been long used in economics (Frey and Stutzer 2002). In the results, we report the observable characteristics of individuals belonging to each class, together with the proportion of those in misery, for these two different thresholds. This allows us to achieve our main goal of identifying observable characteristics of individuals that are more likely to be miserable without making claims on what exactly causes it.

In comparison to regression analysis, LCA is person-centred, looking at groups of factors rather than individual variables, and allows for the ranking of groups against thresholds of interest, e.g., the proportions of miserable people. It is also particularly suitable for reducing multidimensionality in the data (Masyn 2013). For comparison, see Appendix 6.4, which presents binary logistic regressions predicting misery.

3.2.2 Model selection

We start by using all the variables traditionally used in the SWB literature: age, gender, SWB, health and disability status, employment status, socio-economic status, housing tenure, marital status, education, and income. While household income is not reported for most respondents, we retain this variable with the inclusion of the ‘NA’ category. We interpret it in conjunction with the housing tenure which is commonly used as a proxy for income.⁵ In choosing the optimal number of classes, we look to Bayesian information criterion (BIC) statistic and its variations (ABIC, CAIC, Chi-squared and Likelihood Ratio), which is commonly used to balance the gain in log-likelihood through an increase in the number of classes and the loss of degrees of freedom from the greater number of parameters (Lin and Dayton 1997; Lanza and Rhoades 2013). Thus, a lower value of the information criterion suggests a better balance between model fit and parsimony.

A good latent class partition is one for which there is a strong relationship between the item and the latent class variable. Strong item-class relationships must have a particular item response (e.g., item endorsement in the case of binary items, epitomises members in at least one of the K latent classes in the model) and the item must be able to distinguish between members across at least one pair of classes among the K latent classes in the model. The first quality is referred to as latent class homogeneity⁶ and the second quality is referred to as latent class separation⁷ (Collins and Lanza 2009). Gender, sexual orientation, ethnicity and religion did not present a separation into classes above a 70% probability threshold, but we do retain gender as a control variable.

To establish the appropriate class number, we run the clustering algorithm (poLCA package, R version 3.6.3) on multiple random samples from the dataset. The information criteria values indicated that the model with n = 7 classes had the best fit (BIC = 1′144′518, Chi-squared = 6′842′326) (see Table 8 and Fig. 5 and Appendix 6.5 for the full details on fitting the LCA model).

4 Results

4.1 Descriptive statistics

Looking at the percentage of the sample reporting low wellbeing across the different dimensions of SWB included in the APS, we see that 5% of the sample report low life satisfaction, 3.8% report low worthwhileness, 8.9% report low happiness yesterday and 20% of the sample report high anxiety yesterday. That only 1.1% of the sample reports low wellbeing across all four measures highlights the strict nature of our definition and suggests that those identified as miserable according to all four measures in combination are unlikely to represent false positives. Importantly, it does however also underline that there are many people experiencing low SWB that will not be captured using this definition due to their not fulfilling all four criteria. Full descriptive statistics for the entire sample can be found in Table 4 and for the miserable subsample in Table 5.

4.2 LCA results

Table 3 provides conditional item response probabilities by outcome variable for each class., with columns representing the latent classes. The rows indicate the categories of each indicator variable. The table shows the probability of having a given life circumstance conditional on belonging to the different classes. For example, 99% of respondents in class 5 are aged over 60, compared to only 3% of respondents in class 4, and 34% of the sample average. This difference from the sample average can be interpreted as one of the defining characteristics of the class. For ease of interpretation, we highlight in grey the proportions on which the classes differ most from the dataset average. For example, class 7 contains predominantly individuals who are over 60, retired, and homeowners. LCA also allows us to observe the estimated size of each class in relation to the sample (bottom row in Table 1). For example, class 3 makes up 11.2% of the sample, and these individuals are predominately self-employed, in good health, and three-quarters of them have an above-compulsory education.

Table 1 The probability of belonging to a given latent class, on each of observable characteristic

4.3 Applying the misery threshold

According to the definition of misery using all four ONS wellbeing questions and the ONS thresholds (0–4 on Happy, 0–4 on LS, 6–10 on Anxious, 0–4 on Worthwhile), in the three-year dataset 3076 individuals (out of 280,003) or approximately 1.1% fall into this subgroup. Looking at another possible threshold, when we define misery as low LS in isolation, just under 5% of the sample are miserable (Table 2). This comparison highlights the strictness of our preferred definition of misery.

Table 2 The proportion representation of the miserable among the overall sample and the seven classes

Miserable on all 4 SWB measures: all proportions apart from between class 3 and class 4 are significantly different from each other at p < 0.05 level. Proportion difference between class 3 and class 4 is different p < 0.1 significance level. Miserable on Life Satisfaction measure: all proportions are significantly different from each other at p < 0.05 level. Pairwise proportion comparison performed with Holm’s p-value adjustment method, penalising the multiple comparisons. Sample size: n = 215,758.

On average (before any classes are considered), a person in our sample had about 1/100 chance of being miserable. Once the partition into classes is considered, the risk of misery can be roughly organised into 3 tiers which we summarise in Table 3, along with the accompanying combinations of characteristics. Both classes 1 and 2, have an above-average proportion of miserable people. Together they make up 15.4% of the sample, but account for 96% of the total miserable group. Classes 3 and 4, consisting of 20% individuals, have a below average but non-negligible proportion of miserable people. Finally, classes 5–7, comprising over 63% of the sample, contain almost no miserable individuals. Importantly, we do not claim that the combination of characteristics that each class represents are causally related to SWB.

Table 3 The representation of different classes among the miserable

4.4 Comparing misery on all four to low life satisfaction

About 5% of the sample falls under the ‘low’ threshold on life satisfaction measure, while only 1% also fall beyond the threshold on all four measures (see Table 2). When we compare the order rankings of proportions of the miserable using our definition of misery to those produced using low life satisfaction, they are largely consistent. While the two first classes are the most vulnerable to misery across the different definitions, misery on LS dimension ranks class 4 above class 3, with the proportion difference being significant between these two classes. While we are careful to avoid overinterpreting this exploratory analysis, the result is suggestive of a difference in priority ordering of most vulnerable groups depending on the chosen definition of misery.

We also observe the ratio of miserable by the LS definition to miserable on all four SWB measures increasing from class 1 to class 7. For example, class 1 contains three respondents reporting low LS to each one reporting misery on all four SWB scales. This ratio increases to over 6/1 in class 2, 10/1 in class 3, and so on. This suggests that the two definitions of misery are closely aligned for the classes containing the largest proportion of individuals with low SWB. As the proportion of individuals with low SWB falls, the definitions diverge, with the proportion of miserable on all four SWB scales falling at a much higher rate than the proportion of people reporting low LS.

5 Discussion

In this paper, we define misery using the four measures of SWB used by the ONS. We consider someone to be in the most miserable group in society if they report low wellbeing on all four measures. In this way, we partly circumvent the debate about which of the four questions best reflects SWB and address concerns surrounding fuzzy preferences and simply mistaken subjective reports. According to this definition, 1.1% of the total sample are miserable. We examine who is among the worst-off in society by using LCA to identify groups of people united by specific observable characteristics and highlighting those characteristics that differentiate groups more vulnerable to misery from those at lower-than-average risk of being miserable.

The LCA highlights two groups that are at higher-than-average risk of being miserable. By far the most vulnerable are those belonging to class 1. Of the miserable people included in our analysis, class 1 account for 77%. Members of this group tend to be aged 30 + , economically inactive, face disability and health problems, live in rented accommodation, have compulsory or lower levels of education and tend not to be in a partnership. Those in class 2 are also vulnerable to misery, making up 19% of the miserable people in our sample. People in this class share some but not all of the characteristics which define class 1. Members of class 2 also report some health issues and have a higher-than-average risk of disability. They also tend not to be in a partnership. Unlike class 1, this group tends to be employed, is younger, more educated and is just as likely to have a mortgage as to be renting.

Together the members of these two classes make up just over 15% of the sample but they account for 96% of the most miserable members of society. These people, therefore, answer the question of who is miserable now. Their shared characteristics are perhaps unsurprising given some of the existing SWB literature. Many of the same characteristics that matter on average appear to be linked to misery too. Health, marital status and job security, for example, are long-established factors associated with SWB (Dush and Amato 2005; Steptoe et al. 2015; Dawson et al. 2017). The current work builds on existing studies by highlighting the substantive risk of misery facing those who concurrently lack a number of these different protective factors. Health literature is known to use clustering approaches to identify high and low health risk groups by looking at a combination of self-assessed, lifestyle and socio-demographic characteristics and propose tailored interventions (see e.g. Dodd et al. 2010)—and SWB literature can benefit from identifying misery-risk groups too.

Much of the existing literature has examined the determinants of LS. An analysis of the most miserable 5% of the population on LS yields similar results, with classes 1 and 2 remaining the classes which are the most vulnerable to misery. The major difference in the response to our overarching question of who is miserable when we look across the two definitions of misery, therefore, is one of scale rather than composition: Many more people are miserable when we define misery as low life satisfaction, compared to reporting low SWB across all four measures, but class 1 and 2 still account for the vast majority of the miserable in both cases.

We are aware that our approach is not without its limitations. In terms of identifying who are the most miserable, we must rely on the APS survey questions on people’s life circumstances and we must rely on those surveyed in the sample. The APS includes a broad range of questions, but it does not cover all of the dimensions of wellbeing of potential interest, nor all of the determinants of SWB that have been identified in the literature. For example, the APS is lacking indicators on people’s evaluation of the meaning of their lives and how people spend their time, which existing work identifies as an important dimension and predictor of SWB respectively (Stone and Mackie 2013; Laffan 2018). Furthermore, those interested in SWB and misery must do more to get at populations who do not participate in population surveys, such as the homeless and those in institutions such as care homes and prisons, many of whom we might expect to be among the worst-off in society. For example, homeless people, which, depending on the definition, constitute about 0.5% (320,000) of the UK population (Shelter 2018) and we do not capture them in our analysis.⁸

In terms of establishing the factors associated with who is the worst-off, LCA helps us to identify groups of individuals at the highest risks of misery but like most data science tools it requires large volumes of complete observations. This means that once a person fails to answer one of the survey questions (e.g., housing tenure) their entire entry is dropped from the clustering analysis, which can be a problem for the cases where the non-response to certain questions is group-specific (Heffetz and Rabin 2013). This can be particularly challenging if the non-response behaviour is correlated with the variable of interest, i.e., if the miserable tend to avoid answering certain questions about themselves.

We also cannot make causal claims based on our analysis. Like other correlational SWB research, the associations we present are vulnerable to reverse causality and omitted variable bias. As a result, insights from the current work do not suggest how to address people’s misery but rather identify those groups of people that policymakers should pay particular attention to. In particular, our results emphasise the importance of considering how and why individual factors may interplay to make people more or less vulnerable to misery. For example, the misery of those in poor health whilst active in the workforce may be driven by daily concern about job security. In contrast, the misery of those individuals whose poor health prevents them from participating in the workforce may be, in part, caused by the resulting loneliness they experience. Optimal policies to address misery should be informed by evidence on the way combinations of factors influence people’s SWB.

Even if the combination of characteristics that the analysis identifies as being predictive of misery do represent causal impacts on wellbeing, some characteristics will be more susceptible to policy intervention than others: job security compared to marital and disability status, for example. Several of the shared characteristics in both groups with a higher-than-average percentage of miserable—including a relatively high risk of being in poor health and having a disability—suggest that members of this group may be inelastic suppliers of wellbeing and the potential for policy intervention to improve their wellbeing may be limited.

Notwithstanding these limitations, the current work makes significant contributions to our understanding of who’s miserable now. One of the most important yardsticks for judging a society is how well it treats its worst-off. By looking across the four ONS wellbeing questions, we classify just over 1% of the APS sample as being in the most miserable group. By identifying which clusters of people are most vulnerable, we hope to have provided researchers and policymakers with insights which can assist them in more accurately identifying who to target when trying to improve the lives of the worst-off.

Appendices

Appendix

1.1 Descriptive statistics

See Tables 4, 5.

Table 4 Descriptive statistics for the 2014–2016 Annual Population Survey dataset, with full breakdown of the variables as per the ONS coding

Table 5 Descriptive statistics for the Miserable subsample of the 2014–2016 Annual Population Survey dataset

1.2 LCA compared to other classification approaches

Latent Class Analysis is a type of a Finite Mixture Model. The main differences between the LCA and other types of cluster analysis are: (i) model-based (rather than distance-based) grouping of data, (ii) probabilistic (rather than deterministic) assignment of class/group membership. Model-based grouping is well-suited for categorical variables, since there are no “distances” between nominal categories, in contrast to continuous variables. Even though in some earlier literature SWB scales were assumed to be continuous, the general consensus now is that we should treat them as categorical. Naturally, this approach is more dependent on the initial selection of the cut-off points for the categories (Hagenaars and McCutcheon 2002).

Another advantage of a latent class model is that it is a probabilistic model for clustering. Probabilistic assignment allows for ‘fuzzy sets’ where we can measure to what extent we are sure that the individual belongs to a particular group. One may then assign the case to the latent class with the highest a posteriori probability (modal assignment), or leave classification “fuzzy”, i.e. view the case as belonging probabilistically to each latent class to the degree indicated. Because the latent class model is probabilistic it provides additional alternatives for assessing model fit via likelihood statistics, and better captures/retains uncertainty in the classification (Linzer and Lewis 2011).

1.3 Latent class analysis model—specification and intuition

Figure 2 details the components of the LCA model in our specification. The “dependent” variables that are being partitioned into classes comprise the APS measures reflecting the subjective and objective reports of life circumstances. Latent class partition is estimated by fitting the model. At the stage of selecting the variables for the clustering exercise, sex did not show distinct partition between classes which motivated its exclusion from the clustering part of the model. We retained it as a covariate variable in the model, which effectively estimates probabilities of belonging to each class given person’s sex.

Fig. 2

Latent class analysis (LCA) model for the life circumstances typology

Mixture modelling is a widely applied data analysis technique used to identify unobserved heterogeneity in a population. The key function of the finite mixture models is to express the overall distribution of one or more variables as a mixture of (or composite of) a finite number of component distributions, usually simpler and more tractable in form than the overall distribution (Masyn 2013). The central idea is to fit a model in which any confounding between the manifest variables can be explained by a single unobserved "latent" categorical variable.

As an example, consider the dataset (1) in Fig. 3. It lists three variables that have three response levels [Life satisfaction (High, Medium, Low), Happy Yesterday (same as LS), House Ownership (Owner, Mortgage, Rent)], one variable with two response levels [Disability (Yes, No)], and one variable with K response levels [Variable J (Responses 1 to K_j)]. Just for these five manifest variables, and if we assume that K_j = 4, there are 3³ × 2 × 4 = 72 possible response patterns that we might observe in the individuals 1 to N. Latent class analysis enables the researcher to group (or cluster) these responses patterns (and, thus, the individuals with those response patterns) into a smaller number of R latent classes (K <  < 72) such that the response patterns for individuals within each class are more similar than response patterns across classes. For example, response patterns of Person 1 (High, Medium, No, Mortgage,…., Response₄) and Person 3 (High, Medium, No, Mortgage,…, Response₁) might be grouped in the same latent class, different from that which would comprise responses of Person 2 (Medium, High, Yes, Mortgage,…, Response₃) and Person i (Low, High, Yes, Mortgage,…, Response_Kj). Because grouping the observed response patterns is tantamount to grouping individuals, this framing of LCA makes it more person-oriented (Masyn 2013).

We will now explain the intuition behind the LCA-based data partition model. Table (2) in Fig. 3. presents the same example dataset reformatted to highlight the components of the optimisation process. Note how person 1’s response “Life Satisfaction: High” in panel (1) transforms into three variables in table (2): Y₁₁₁ = 1, Y₁₁₂ = 0, Y₁₁₃ = 0. The model takes all the Y_ijk observations and estimates the model parameters in table (3) Fig. 3, using the following log-likelihood function:

$$\mathrm{ln}L= \sum_{i=1}^{N}\mathrm{ln}\sum_{r=1}^{R}{p}_{r}= \prod_{j=1}^{J}\prod_{k=1}^{{K}_{J}}{({\pi }_{jrk})}^{{Y}^{ijk}}$$

The log-likelihood function is solved with respect to p_r and π_jrk using the expectation–maximization (EM) algorithm (Dempster, Laird, and Rubin 1977). Here, p_r (the bottom row of table (3) Fig. 3) denotes the class-specific proportions of the sample. For example, p₁ = 0.15 indicates that 15% of the sample were classified as Class 1. In turn, π_jrk denotes the estimates of outcome probabilities conditional on belonging to class r. For example, values (π₁₁₁ = 0.9, π₁₁₂ = 0.04, π₁₁₃ = 0.01) would suggest that conditional on belonging to Class 1, an individual would rate their LS as high with a 90% probability, while as medium with 4%, and low with 1% probability. Likewise, values (π₁₂₁ = 0.3, π₁₃₁ = 0.1) would indicate that individuals in classes 2 and 3 would rate their LS as high with probabilities 30% and 10% respectively.

Fig. 3

The (1) Illustration of the initial data structure; (2) illustration of the same dataset, as the input for the model; (3) illustration the output of the LCA model

Consistency check using binary logistic regressions

2.1 Covariates of misery

In a robustness check, we use logistic regression with a binary outcome of being/not being miserable to examine the odds associated with a given predictor controlling for all the others. We include the standard determinants from the SWB literature: age, sex, sexual orientation, marital status, health, employment status, socio-economic status, property ownership, religion, and ethnicity.

The logistic regression with misery as outcome variable estimates how an individual’s characteristics relate to their odds of being miserable. We run models with and without sampling weights that correct for response rate based on age, sex, and geographical area. The results are aligned in significance level and effect size, in the unweighted and weighted models for (Table 6, models 1 and 2, respectively). Health, disability status and employment status emerge as the factors most strongly associated with misery once other covariates are controlled for. A person in fair health had 3.45 times greater odds of being miserable than the same person in good or very good health. The odds of misery for a person in bad or very bad health were over 12 times higher than those of a person in good or very good health. Having a disability had a smaller, albeit sizeable effect on misery—the odds of being miserable for a disabled person were 1.82 those of a non-disabled one.

Unemployment is strongly associated with misery: compared to the employed individuals, the unemployed had 2.91 times greater odds of being miserable. The APS questionnaire allows for a distinction between being just economically inactive and inactive with a long-term sickness or disability. Predictably, being both economically inactive and having a long-term sickness and disability was associated with a greater increase in odds of misery than inactivity alone. Economically inactive individuals had 1.56 times greater odds of misery than the employed ones, whereas for those in category ‘inactive (long-term sick/disabled) the odds of misery are 2.47 greater than those of the employed ones. For average marginal effects see Table 7.

While health, disability and employment status are the most important predictors of misery, we observed significant effects in other common life factors. Ranked in terms of vulnerability to misery these are: socio-economic status (having a semi-routine or routine occupation was associated with greater odds of misery, compared to holding a managerial job), education (individuals with an A-level were less likely to be miserable than those with basic or no education; interestingly, education beyond A-level did not appear to matter), housing tenure (house owners were less likely to be miserable), ethnicity and religiosity (non-white British and non-atheist individuals were less likely to be miserable than their white British and religious counterparts), and marital status (couples were less miserable than single people).

We also observed the classic U-shape relationship between SWB and age⁹: the odds of misery of individuals aged 30–39 were 1.54 those of individuals aged 16–29; they increased further for the individuals aged 40–49, whose odds of misery were 1.80 those of the reference group; and around retirement age odds of misery decreased again, not differing significantly from those of the reference group aged 16–29.¹⁰ Notably, in the LCA analysis we identify heterogeneity behind this overall coefficient in the 16–29 age group. Class 4 comprising predominantly respondents of this age, healthy, yet out of employment or in lower SES had larger proportion of respondents classified as miserable compared to the subsequent three classes (5, 6, and 7) comprising more than half of the sample.

Table 6 Covariates of Misery: (1) unweighted, (2) weighted

Table 7 Average marginal effects (average effect of the covariate on the probability (0–100%) of misery) of the unweighted logistic model (Table 6, model 1)

2.2 Fitting the LCA

The LCA model is fitted by Maximum Likelihood (ML) using the EM algorithm with the following steps. (1) start with random initial probabilities (i.e. random split of people into classes on all observable characteristics), (2) maximize the log-likelihood (LL) function (reclassify people based on an improvement criterion), (3) update the probabilities (based on the posterior distribution), (4) repeat (2) and (3) until no further improvement is possible more (LL is at the maximum value).

The analysis used the raw matrix of the variables available in the APS dataset (see Appendix 6.3 for the list of variables). Fitted models with different numbers of classes were compared on the goodness-of-fit statistics reported in Fig. 4 and Table 8. The ABIC, CAIC, Chi-sq and Likelihood Ratio are all used to measure the goodness-of-fit and differ with respect to how additional parameters are penalize. Overall, a lower value of the information criterion suggests a better balance between model fit and parsimony. The second function of this process was to select the variables that allow for maximally distinct classes to emerge—according to Nagin (1999) the rule of thumb for the acceptable group classification is that the average posterior probability of correct group membership assignment is ≥ 0.80. In general, entropy with values approaching 1 indicate clear delineation of classes (Celeux and Soromenho 1996). Hence, we run the clustering algorithm to achieve the optimal goodness-of-fit statistics for the given model specification and then collapsed variable categories that tended to same class into broader categories for this variable (see Fig. 4).

Specifically, to establish the appropriate class number, we took 50 random samples of 60,000 (~ 33% of total sample) and run the clustering algorithm (poLCA) for number of classes (n) from 1 to 10 on each of them. For each run, we have set the number of repetitions (nrep) to 30 and maximum iterations (maxiter) to 4000. A high number of repetitions and iterations allows the model to re-start from new random initial values which is crucial for finding the global rather than local maximum.

Fig. 4

Elbow plot of Information Criteria. n = 60,000 subsample. The lowest value indicated the optimal number of classes

Table 8 Numeric values of the information criteria

Once obtaining the optimal number of classes for the selected model specification, in accordance with the best practices (see e.g., Hagenaars and McCutcheon 2002) we run the model with seven classes multiple times to be reasonably certain that we have found the parameter estimates that produce the global maximum likelihood solution. A well-known drawback of the Expectation–Maximisation (EM) algorithm is that depending upon the initial parameter values chosen in the first iteration, the algorithm may only find a local, rather than the global, maximum of the log-likelihood function (McLachlan and Krishnan 2007). To avoid these local maxima, it is a standard to run poLCA with the same model specification and same number of classes multiple times using different starting values, to locate the estimated model parameters that correspond to the model with the global maximum likelihood. Upon re-running the model 50 times we observe convergence to the same maximum log-likelihood value. Hence, we can be reasonably sure we found the global maximum for the given specification. Additionally, we looked to the smallest estimated class size to verify that it is not close to zero which would indicate non-convergence of the model. In our specification, the smallest class size in the full-sample model was estimated as 7.3% which indicates a successful convergence.

To ensure that the classes we find represent naturally occurring subgroups in the population rather than being a sample-specific statistical artefact, we conduct clustering on multiple random samples (n = 40 k, 50 k, 70 k) from the dataset, to ensure consistent separation into classes of same sizes and characteristics. Given that the same classes appear consistently when conducting the same analysis with multiple subsets of the same sample, the classes are considered reliable (Bauer and Curran 2004; Lenzenweger 2004) (see Fig. 5).

Fig. 5

Elbow plot of Information Criteria. n = 60,000 subsample. The lowest value indicated the optimal number of classes