Introduction

It has been argued that “welfare reform has become a defining feature of contemporary UK government policy” (Beatty and Fothergil 2016). With social housing at the centre of these changes, studies have suggested that the reduction of income felt by the policies has resulted in broad-ranging effects on health, wellbeing and community (Moffatt et al. 2016), including a significant risk of increased household poverty (Crisp et al. 2017).

The changing role of social housing has been driven by measures to promote private rented accommodation and access to home ownership, whilst reducing support for accessible housing for low-income households (Crisp et al. 2017). As a result, the traditional role of housing officers is broadening, with some being required to take on a public health/health promotion aspect to their roles.

One approach to widen the role and responsibilities of the housing officers is represented by a new model for delivering an integrated and enhanced housing service to council tenants in Sheffield, UK. The overall aim of the new role of neighbourhood officers is to deliver community and individual resilience via sustainable tenancies. The service differs from the previous approach, as neighbourhood officers act in a generalist role within a defined neighbourhood to meet the needs of their tenants, rather than having specialist roles in city-wide teams, such as dealing with antisocial behaviour or tenancy management across a wider geographical area. The service uses a tailored approach that takes into account the needs of individual households, rather than trying to adopt a one-size-fits-all approach. The neighbourhood officer (a local authority employee) works with a geographically based caseload of between 180 and 330 households. As a minimum, the service involves undertaking an annual visit, which is designed to discuss wider determinants of tenancy sustainability, including health, employment, education, home skills, neighbourhood issues and community engagement. Neighbourhood officers take a holistic approach, where the focus is on prevention, dealing with low-level issues directly, signposting people to resources within the local community to help prevent problems escalating and, where necessary, referral to other services for more specialist help and support as required. As well as referral to other services, neighbourhood officers still provide general housing management in the neighbourhood, such as dealing with outstanding repairs and rent arrears. Overall, the intervention could be expected to impact on a range of public health outcomes, particularly in relation to the implementation of preventive interventions and referrals to appropriate community and health and social care services, which may, in turn, contribute to reduced health inequalities.

Evaluation of such complexity presents challenges for summarising and presenting evidence concisely. In order to evaluate such an intervention effectively, there is also a need to understand the complex system in which the intervention is being delivered. One approach to attempt to succinctly describe such complexity is through the development of a logic model. A logic model is a summary diagram that maps out a complex intervention and the links with both observed and potential outcomes in order to develop a theory of how the intervention works (Baxter et al. 2010). Logic models typically adopt a left-to-right flow of “if...then” propositions to illustrate the chain of reasoning underpinning how interventions lead to immediate (or short term) outcomes and then to longer term outcomes and impacts”. Logic models seek to uncover the theories of change or logic underpinning pathways from interventions to outcomes (Weiss 1995) in order to ascertain assumptions that underpin links between interventions, and the intended short- and long-term outcomes and broader impacts (Rogers and Weiss 2007). Uncovering the assumptions and processes within a complex social housing intervention requires an understanding of whole systems, which a logic model methodology is well placed to address. The use of logic models have a number of proposed benefits, including defining understandings or theories about how an intervention works, clarity as to which interventions lead to which outcomes, diagrammatic representation of the main influencing factors in intervention delivery and the generation of testable hypotheses (Rogers 2008). The process of developing an evidence-based logic model has the potential to generate insights that would not automatically emerge from a more narrative synthesis of secondary and primary data.

In this study, we extended our previous approach of using systematic review methodologies to develop logic models (Blank et al. 2014, 2016; Baxter et al. 2010) by incorporating extracted secondary data from published studies, along with primary data from interviews and a telephone survey, which were combined and treated as textual (qualitative) data. A process of charting and categorising the various data sources leads to a thematic synthesis (18) of the extracted quantitative and qualitative data, which, in turn, leads to developing individual elements of the model. An essential part of the final logic model is detailing the mechanism(s) of change within the pathway and the moderating and mediating factors that may be associated with or influence outcomes.

Materials and methods

Our approach of building a logic model systematically from primary and secondary evidence is a novel methodology that contrasts with the approach traditionally adopted, whereby logic models are built by discussion and consensus at meetings of stakeholders or expert groups (Baxter 2010). The processes we adopted here build on our previous work of developing logic models as part of a systematic review process (Blank et al. 2014, 2016; Baxter et al. 2010), as we also incorporated primary data collection and analysis into the process. Development of this logic model was, therefore, supported by a range of data generation methods, including a scoping review of the literature, telephone survey with housing tenants, in-depth interviews with tenants and housing staff, and workshops with key stakeholders to help to develop and then validate the model. The processes followed are described in further detail below.

Secondary data collection

A brief scoping review of the literature was undertaken, along with analysis of documents relating to how the intervention was intended to work in theory. A full systematic review was not appropriate, as the aim of the review element of the work was to inform the interview guides and to begin to develop an initial model, not to find every piece of relevant evidence. This decision was also taken to meet the project timescales and due to an understanding that there was not a huge body of evidence available. The scoping review considered recent published UK evidence on the impact of changes to the role of housing officers on the wellbeing of their tenants. In keeping with the scoping review methodology, relevant studies were summarised with a focus on the linkages they describe. No quality appraisal was undertaken, as is typical for a scoping review (Arksey and O’Malley 2005). Further detail relating to the search strategy, data extraction and synthesis methods will be published elsewhere.

Primary data collection

In the first year of the process evaluation, we completed 55 interviews with both tenants (30) and housing staff (25, including 4 managers) between October 2016 and January 2017. This was followed up in the second year (October 2017 to January 2018) by a further 13 interviews with tenants (5 repeat, 8 new) and 12 housing staff (9 repeat, including 2 managers and 3 new). Focus groups were also held throughout the evaluation with housing staff, tenants and staff from key stakeholder agencies. A telephone survey of 1000 tenants was also completed. Tenants who completed the survey in the first year were followed up in the second year where possible. In total, 561 tenants completed both survey points, with the second-year sample being completed by an additional 439 new tenants where first-year participants could not be re-contacted. Further details of the qualitative analysis methodology employed have already been published (Blank et al. 2019).

Developing the model through workshops

Workshop sessions with key stakeholders were delivered at two points during the project in order to refine and validate the model as it developed. Following the development of a theoretic model (from documentation relating to the intervention), we sought input from stakeholders regarding the clarity of presentation of the draft. We carried out a group session with academics, practitioners (including key partner agencies), housing officers and tenant representatives where we presented the initial model and asked for verbal comments regarding the clarity of the model as a tool to interpret what the intervention set out to do. We asked whether any elements were missing, if there was anything that did not seem to make sense or fit participant knowledge or experience and whether they would change anything about the model. Notes were taken by a researcher throughout the session. We also provided participants with a printed copy of the model, along with feedback forms, so that they could provide written comments on these aspects. The process was repeated with the final draft logic model after data from the scoping review, interviews and survey had been used to fully populate the model.

Results

Developing the initial logic model

We began to develop our initial model from analysis of documents relating to the intervention, along with wider literature, which detailed the policy context in which the intervention was set, details of what the intervention was theoretically intended to deliver (approach) and the expected outcomes (both in the short term and longer term) as a result of the intervention. Model factors are referred to in square brackets throughout the following sections.

As outlined in the introduction to this paper, the context for development of the intervention centres around ongoing reform to public services [Public service reform] and welfare provision [Welfare reform], and the knock-on effect on health inequalities in the council housing population versus other sectors of society. As a result, the need to provide cost-effective council housing services [Cost effective provision of council housing services] combined with the concept of expanding the role of housing staff to incorporate a greater public health role [Expanding public health workforce] under the “Making every contact count” [MECC] model (Health Education England 2019) resulted in the concept of Housing+. This context is outlined in the first column of the initial model (Fig. 1).

Fig. 1
figure 1

A priori model

Full size image

The theoretical concept of Housing+ is outlined in column two of the model. [Area based neighbourhood officers] taking a [Preventative approach] incorporating a tenant [Health and wellbeing focus] were expected to meet all tenants annually via a home visit to identify unmet need, refer or signpost tenants to other support agencies to meet these needs, and promote joined up working between referral agencies [Offer of an annual home visit]. It was anticipated that this approach would provide tenants with [One point of contact] for all their interactions with the city council regarding their tenancy. This was expected to result in [Identifying unmet need] within the tenant population, improved [Referral/signposting] to associated services where needed and better [Joined up working] both within the council and with other services. This would be achieved through [Service development], [Redeployment] of staff into newly defined roles (and associated [Staff training]), [Housing office reorganisation], improving [IT including mobile technology], [Publicity] of the new ways of working and external [Evaluation] of the intervention.

At the outset of the evaluation, the anticipated short-term outcomes were change in staff roles as a result of [Staff re-employment], the number of [Housing visits offered/completed], an increase in [Community awareness and engagement] in the intervention, [Improved partnership working] with partner agencies and more appropriate [Referral rates], along with any changes to [Staff morale] and [Tenant satisfaction] with the housing service. It was also hoped that, in the longer term, the service would impact positively on [Tenancy sustainability], [Tenant health] and wellbeing, be more cost-effective [Cost reduction] and also have a positive effect on antisocial behaviour in the neighbourhoods [Reduced ASB]. These are given in the final column Fig. 1.

To develop this “a priori” model into one which more clearly defined the systems and influences that were operating in the new intervention, we incorporated the main themes from our qualitative analysis of interview data to bridge the gap in understanding between the intervention as it was set out on paper and the outcomes which were (or were not) being achieved in reality. This final model (Model 2) is presented in Fig. 2.

Fig. 2
figure 2

Logic model

Full size image

Interview and focus group data

The main themes identified in the primary data defined the mediating and moderating factors that form the central part of the final logic model (Fig. 2). They were defined as follows:

Neighbourhood officers: The neighbourhood officers’ data included a number of themes relating to their change in roles, including: [Loss of specialist knowledge] gained through their [Previous experience] over many years working in the housing service. This resulted in a knock-on effect on their confidence in managing additional responsibilities in the new roles. This [Lack of confidence] initially led to increased [Workload stress], due to the [Additional responsibilities] of the new roles, and (in some cases) [Sickness absence] resulted. The likelihood of this chain of events unfolding was often determined by the perceived level of support they received in the transition period [Managerial support variable].

The neighbourhood officers also discussed their own approach to the new role [Personal approach to visits], as they were [Settling in to Housing+ roles] and concerns over dealing with [Sensitive topics] (e.g. relating to health and wellbeing and also others, such as finances). They talked about variable [Workload], which was exacerbated by paperwork demands [Paperwork extensive] and also depended on the particular demands of their ‘patch’ (neighbourhood) [Variable patch demands], their previous experience of working directly with tenants [Responsive to tenant need/hard to predict] and if their demands to reduce excessive workloads were being addressed through [Anticipated patch changes] by the city council.

The neighbourhood officer experience was also affected by [Differing approaches to setting up Housing+ visits (letters, cold calling and varying number of attempts to contact a tenant)], which could impact on their perception of workload. The also discussed the acceptability of the [Training] they received, including the [Availability] and [Suitability] of training during the transition period (with particular reference to the value of [Peer support], as well as more formal training packages) and issues around the provision of [IT] and mobile systems [Phablets], which resulted in problems with remote access to the council’s housing databases [System access]. These issues contributed to conflict between time in the office and time spent out in their neighbourhood, leading to a feeling of being [“Tied to the office”] and a subsequent reduction in the number of Housing+ visits achieved.

Tenants: Throughout the two years of the evaluation, awareness and knowledge of the service amongst the participants remained mixed [Low service awareness], in which some tenants were unable to recall whether they had received a visit [Visit recall limited]. Awareness of the service ranged from knowing the location of the local housing office to receiving visits from a neighbourhood officer. The number of tenants who reported receiving a Housing+ visit was lower than anticipated in both years of the evaluation. There was some confusion and suspicion over the process of home visits [Nervousness and lack of clarity over visits]. Some tenants were unsure about the new arrangements, which may have led to a reluctance to accept a visit, whilst others were happy to have the neighbourhood officer in their home [Difficult to contact/Happy with visits] and were also happy to contact the neighbourhood officer directly [Willing to contact neighbourhood officer].

Overall, the impact on the tenant population was minimal in the first two years of the intervention [Minimum/no impact on most tenants], suggesting that the Housing+ implementation had not caused significant upheaval or problems for the tenants. Topic areas covered during visits varied but conversations were often dominated by issues with the repairs service and the stress and anxiety tenants experience as a result of this. There were, however, a number of examples of significant positive impact on individual tenants, particularly where unmet needs had been identified and positively managed [Specific examples of significant positive impact → Identifying unmet need], including where health needs had been identified and appropriate referrals made [Health needs and poor health in household → Health service access].

Referral agencies: The results from a focus group with referral agency staff indicated that the main moderating factors with the potential to influence the Housing+ outcomes were the capacity within the referring agencies (i.e. whether they would be able to cope with and administer a possible increase in referrals as a result of unmet need identified through Housing+ visits) and the referral thresholds (whether new referrals made via neighbourhood officers were likely to meet the agencies’ own criteria for referral and whether there were processes in place to manage the referral if the criteria were not met) [Referral capacity/threshold]. Neighbourhood officers were not always able to follow up referrals, which caused tension in their relationships with referral agencies [Referral follow up]. Representatives from some agencies were also concerned about a potential for an [Overlap of responsibilities] between themselves and housing officers in their new, broader roles.

Council services: Interviews with tenants and housing officers also indicated a number of core council services that had a key role in mitigating the potential to achieve the Housing+ aims and outcomes. The main problems identified by the tenants interviewed in terms of the relationship with the council related to ongoing or [Unresolved problems] with [Repairs] to their property. It was felt that the frustrations encountered by some tenants over the repairs service not meeting their expectations (whether this would be considered appropriate or not) had the potential to limit their relationship with their housing officer. This was made difficult by the fact that housing officers could not control the speed at which the repair was undertaken or assessed, leading to frustration on the part of the tenant. [Support from central teams (rents etc.)] provided by the council as well as how efficiently [Property transfers] (requests to move home) were managed also impacted on tenant views of their neighbourhood officer and their opinion on the intervention overall.

Relationships: Throughout the evaluation period, tenants felt that it was important to build a trusting relationship [Building trust] with their neighbourhood officer, both to give the tenant confidence in talking to them and to not to be “passed from pillar to post” across different staff members and departments. In each aspect of data collection, the central role of successful relationships in the success (or otherwise) of the intervention were noted. These included the key relationship between a tenant and their neighbourhood officer, but also relationships between the tenant and other service personnel (both with and external to the council), and also the neighbourhood officers’ relationships with these agencies. For example, a strong relationship between a neighbourhood officer and tenant could lead to a tenant accepting a referral for further support via external agencies, but the success of this referral would depend on whether the tenant would be able to develop a positive relationship with their contact in the new agency and also whether good relationships between the neighbourhood officer and the referring agency contributed to a referral that was timely and appropriate. [Relationship conflict] of any type reduces the chance of these positive pathways being followed.

Discussion

Key findings

The process of defining our final logic model generated insights that would not have emerged from a more narrative synthesis of secondary and primary data. The most important of these was a clear message about the central role of relationships between neighbourhood officers and tenants (and neighbourhood officers and other front-line statutory and voluntary organisations and their staff).

After defining the main themes to be used in the final logic model, discussion in our focus groups and workshop sessions strongly indicated that the relationships between each group of people in the system (tenants, neighbourhood officers, referral agencies and other council services) were central to defining the intervention and were able to explain some of the variation in data and individual views that were seen in the interviews we conducted. As a result, we constructed a model that firmly placed ‘relationships’ (including building trust between individuals and organisations, along with managing relationships conflict) at the centre of the moderating factors for this intervention, as these relationship-based moderating factors seem to affect all the other moderating factors identified during the analysis. As such, our final model attempts to represent the complexity in the system and give some understanding of what is actually happening in terms of delivering the intervention to influence how the inputs lead to the short-term outcomes seen (and the potential for longer-term impact in the future).

Therefore, the key message to take from this model is that the relationships between neighbourhood officers, tenants and other supporting agencies (within and external to the council) are integral to the delivery and success of this intervention. In considering similar interventions, thought needs to be given on how a relationship can be built between a tenant and a neighbourhood officer, as the conflict between supporting a tenant whilst also being potentially responsible for rent collection, arrears management and, ultimately, eviction was frequently noted in our data collection. Neighbourhood officers would benefit from opportunities to build relationships with professionals in organisations to which they refer tenants in order to identify specific opportunities for networking and joint working (formally and informally) to better develop these essential relationships.

Strengths of this approach

This work aimed to identify and summarise the true complexity of an intervention to develop and broaden the role of housing officers. Our logic model sets out the chain of reasoning as to how and if the intervention is likely to lead to the anticipated impacts and clarifies the assumptions that underpin this process in the form of moderating factors. In this way, the logic model is able to summarise a hugely complex system on a single page. To our knowledge, this is the first logic model that considers this type of intervention.

The use of stakeholder input to develop theories is well established (Blamey and Mackenzie (2007). Following this, participants in our workshop sessions provided valuable insight and were able to indicate where gaps in the evidence appeared likely. In building our model from primary and secondary data analysis, we have sought to be systematic and evidence-based, rather than to allow expert/stakeholder opinion to lead the process from the start (as is more typically seen in logic model development). We believe that this approach leads to an initial unbiased model that can be validated with expert opinion. However, it is important to consider potential sources of bias in any study methodology, especially where it has not been widely used. This is why the ultimate involvement of stakeholders and seeking opinion on potential gaps in the evidence (and, therefore, the model) remains a key aspect of the process. In this case, our workshop participants supported our model and made suggestions and revisions only at the level of revising text to facilitate understanding (rather than suggesting additional key linkages or factors that were missing from the model). However, previously in other evaluation projects that have contributed to the development of this method, stakeholder involvement has resulted in additional areas being identified and, therefore, more significant revisions to the model being made. This may potentially be explained by the fact that this model is based primarily on one intervention (although it draws on a wider body of literature), whereas previous models have been broader and have been based on a type or (several types) of intervention and have, therefore, had a broader scope and potential field of inclusion. In previous work, we have considered the use of ‘ghost boxes’ or similar to indicate in the model where suggestions have been made through the consultation that have not been supported by the data collection up to that point. This may also indicate questions that could be addressed by future research in the area to validate the suggestions made by stakeholders.

Limitations and considerations

Any visual representation of data must be able to stand up to scrutiny in order to have an intrinsic value, meaning that the concepts and structure can be readily understood by others in order to facilitate discussion (Dixon-Woods et al. 2001). We are of the belief that the continued development of our logic model approaches is able to stand up to this scrutiny. However, it is still important to recognise that, in identifying a level of complexity, we accept that, in reality, a system is always more complex than the level at which it can be successfully described (Vogel 2012). However, stakeholder workshop feedback on the logic model suggested that it is a good representation of the complexity within the system, along with the key drivers that are acting to affect the success (or otherwise) of the intervention. Our validation workshops demonstrate that this method of presenting data can be understood and interpreted by a wide range of stakeholders, including academics from related fields, health and housing professionals, tenant and community representatives, and staff from other key stakeholder agencies.

As with many interventions evaluations, our logic model takes a snapshot of a particular phase in the development of Housing+. This intervention is still ongoing and developing; and further continued evaluation would be required in order to obtain the best learning from this. Feedback and recommendations from our evaluation have impacted the way that Housing+ continues to develop (for example, questioning whether annual visits are the best way of engaging with all tenants in the population), but, at present, we do not have the funds to continue to develop and refine our model to keep pace with this progress.

Our sample populations for both our telephone survey and interviews (which both contributed data to developing the logic model) may have excluded those who were not available to participate due to work commitments or who had no fixed address or means of contact (telephone). Overall, our sample population was slightly older, more likely to be unemployed and less ethnically mixed than the tenant population overall. As always, the potential for these variances to impact on the results obtained should be noted.

Conclusions

The key message from our evaluation of this tenancy sustainability intervention is that the relationships between neighbourhood officers, tenants and other supporting agencies (within and external to the council) are integral to the delivery and success of this intervention. In developing similar complex interventions, it is, therefore, vital that funders identify and carefully consider the specific relationships that have the potential to impact on the successful delivery of their intervention. Opportunities for identifying which relationships are important should be developed and their importance should be reflected in the objectives and outcomes measures set out for the evaluation. Any potential to encourage or develop key relationships ahead of implementing the evaluation may remove points of tension and allow the intervention to be implemented as successfully as possible.