Social Impact Measurement and Non-profit Organisations: Compliance, Resistance, and Promotion
- First Online:
- Cite this article as:
- Arvidson, M. & Lyon, F. Voluntas (2014) 25: 869. doi:10.1007/s11266-013-9373-6
- 2.3k Downloads
Non-profit organisations are under increasing pressure to demonstrate their social impact. This paper examines the experience and behaviour of non-profit organisations in the UK in relation to a demand for social impact evaluations. External resource providers request organisations to present evidence on how resources are used and what organisations have achieved. While most organisations are willing to comply and accept this control, they can also resist through using their discretion in deciding what to measure, how to measure and what to report. Non-profit organisations can proactively and voluntarily use social impact measurement for learning and promotional purposes, and as a way of exerting control over their environment. The analysis develops the concept of strategic decoupling to explain the differences observed between what organisations are asked to do, what they plan to do and what they are doing in practice.
KeywordsEvaluation Social impact Decoupling Resistance
On attend de plus en plus souvent des organisations à but non lucratif qu’elles démontrent leur impact social. Cet article examine l’expérience et l’attitude d’organisations à but non lucratif au Royaume Uni vis-à-vis de cette exigence d’auto-évaluation. Les fournisseurs de ressources externes demandent des organisations qu’elles apportent des preuves de la façon dont elles utilisent les ressources et des objectifs qu’elles atteignent. Même si la plupart des organisations sont prêtes à se conformer à ces exigences et à accepter ce contrôle, elles peuvent aussi résister par le choix de ce qu’elles mesurent, comment elles le mesurent, et ce qu’elles incluent ou non dans leurs rapports. Les organisations à but non lucratif peuvent activement et délibérément utiliser l’évaluation de leur impact social pour apprendre ou pour promouvoir leur action et comme un moyen de maîtrise de leur environnement. L’analyse développe le concept de découplage stratégique pour expliquer les différences constatées entre ce qu’on attend des organisations, ce qu’elles projettent de faire et ce qu’elles font en pratique.
Nonprofit-Organisationen stehen unter zunehmenden Druck, ihren sozialen Einfluss zu demonstrieren. Dieser Beitrag untersucht die Erfahrungen und Verhaltensweisen von Nonprofit-Organisationen in Großbritannien in Bezug auf eine Nachfrage nach einer Beurteilung sozialer Auswirkungen. Externe Ressourcenanbieter fordern von den Organisationen, Nachweise darüber zu erbringen, wie die Ressourcen verwendet werden und was die Organisationen geleistet haben. Zwar sind die meisten Organisationen bereit, dieser Forderung nachzukommen, und sie akzeptieren diese Art von Kontrolle; doch können sie auch Widerstand leisten, indem sie nach eigenem Ermessen entscheiden, welche Aspekte sie erfassen, wie sie sie erfassen und worüber sie Bericht erstatten. Nonprofit-Organisationen können die Messung des sozialen Einflusses auf eigene Initiative und freiwillig zu Lern- und Werbezwecken nutzen sowie zur Ausübung von Kontrolle auf ihren Wirkungskreis einsetzen. Die Untersuchung entwickelt das Konzept der strategischen Entkopplung, um die beobachteten Unterschiede zwischen den Anforderungen an die Organisationen, ihren Plänen und dem, was sie tatsächlich in der Praxis tun, zu erläutern.
Las organizaciones sin ánimo de lucro sufren la creciente presión de demostrar su impacto social. El presente documento examina la experiencia y el comportamiento de las organizaciones sin ánimo de lucro en el Reino Unido en relación con la exigencia de evaluaciones de su impacto social. Los proveedores de recursos externos solicitan a las organizaciones que presenten pruebas de cómo se utilizan los recursos y qué han logrado las organizaciones. Aunque la mayor parte de las organizaciones están dispuestas a cumplir y aceptar este control, también pueden resistirse utilizando su criterio para decidir qué medir, cómo medir y de qué informar. Las organizaciones sin ánimo de lucro pueden utilizar proactiva y voluntariamente la medición del impacto social con fines de aprendizaje y promocionales, y como forma de ejercer control sobre su entorno. El análisis desarrolla el concepto de separación estratégica para explicar las diferencias observadas entre lo que se pide a las organizaciones que hagan, lo que ellas planean hacer y lo que están haciendo en la práctica.
المنظمات الغير ربحية تتعرض لضغوط متزايدة لإثبات أثرها الاجتماعي. يفحص هذا البحث خبرة وسلوك المنظمات الغير ربحية في المملكة المتحدة بالنسبة إلى الطلب على تقييمات الأثر الاجتماعي. مقدمي الموارد الخارجية يطلبون من المنظمات أن يعرضوا أدلة على الكيفية التي تستخدم بها الموارد و ما حققته المنظمات. في حين أن معظم المنظمات على إستعداد للإمتثال و قبول هذه السيطرة، فإنه يمكنهم أن يقاوموا أيضا˝ من خلال إستخدام حريتهم في التصرف في التوصل إلى قرار فيما يجب قياسه، وكيفية القياس وماذا يقدم تقريرعنه. يمكن للمنظمات الغير الربحية أن تستخدم مسبقا˝ وطواعية قياس الأثر الإجتماعي للتعلم وأغراض ترويجية، وكوسيلة لممارسة السيطرة على بيئتهم وتحليل تطور مفهوم لفك الإرتباط الإستراتيجي لشرح الفروق الملحوظة بين ما يطلب من المنظمات القيام به، ما هم يخططون للقيام به وماذا يفعلون في الواقع العملي.
There is a growing interest amongst non-profit organisations in approaches to measure their social impact. While control can be exercised by outside resource holders, there is also a need to understand the responses of organisations themselves. This paper examines the experience and behaviour of non-profit organisations (charities, social enterprises), in the UK, in relation to evaluation and social impact measurements. Social auditing and social performance measurements can be presented as means of controlling staff and organisations. The control can be exercised formally and directly, through evaluation requirements linked to contracts, and it can be exercised indirectly, through norms where individuals and organisations seek approval and verification (Power 1999). However, there is emerging evidence that organisations are not seeing the practice of evaluation as simply a form of compliance to resource holders (Pritchard et al. 2012). This paper shows how organisations both accept and actively resist control, with many organisations using the evaluation process for promotional purposes.
At present, UK non-profit organisations find themselves under a new auditing regime introduced through new social policies that emphasise social value (Department of Health 2011; HM Government 2012) and philanthropic funding arrangements (Pritchard et al. 2012), where auditing is linked to accountability, legitimacy, and competition for resources. Social auditing and accountability controlled by outside stakeholders (Ebrahim 2003, 2005), and conducted with the purpose of managing and mediating competition in the field of care and social mission, is causing discomfort (Turco 2012). There is also fear that the auditing process, with ensuing performance management, has detrimental effects on organisational culture and staff morale (Christensen and Ebrahim 2006; Hwang and Powell 2009; Townley 2011).
This paper makes a contribution to the understanding of how evaluation and social impact measurement is part of organisational strategy, with a difference between what organisations are asked to do, what they say they are doing, and what they are doing in practice. The concept of decoupling describes how organisations ‘make visible, public commitments to satisfy the demands of their external environment’ while the organisation’s internal operations can follow different trajectories (Turco 2012, p. 386). Decoupling can be seen as a way of managing the dilemma of resisting external monitoring (Levay and Waks 2009) combined with a wish to gain legitimacy from the very same external context. The study shows how organisations aim to reconcile such underlying conflicts without upsetting relations with stakeholders.
Based on interviews with individuals representing social purpose organisations the paper examines how social impact measurement can be used by resource holders to exert control over funded organisations. The paper goes on to argue that there is a need to go beyond conceptualisation of evaluation as a form of control by those providing funding over those providing the impact. An alternative, or additional, view can see evaluation or social impact measurement as a space of resistance and a strategy for organisational promotion.
The paper illustrates how non-profit organisations are being shaped by the influence of logics and norms held by other parties. With an emphasis on the role of agency, it also illustrates how organisations manage to subvert this power. The concept of decoupling can suitably describe some of the actions taken by organisations. The paper explores how organisations respond to requests for evaluation, and how strategic decoupling is developed. The study suggests that organisations who engage in evaluations do so with a dual purpose. On the one hand, they do so to comply with social audit norms and thereby ascertain organisational legitimacy and survival. On the other, they engage in this with in a way that allows them a level of independence as well as influence over how the performance of social purpose organisations is understood and measured. In order to describe the processes surrounding these strategies, the paper explores motivations to comply and the sources of discomfort and tensions experienced by organisations, as a basis for understanding different intentionalities that guide decoupling.
Evaluation as a Control Measure
Pressure on non-profit organisations for more rigorous impact measurement and reporting can come from a range of stakeholders. These include the clients, other social purpose organisations, and so-called patrons (Ebrahim 2003). However, these stakeholders have diverse ideas about what and how activities should be measured (Kendall and Knapp 2000). This, together with the nature of social impact being hard to define (Barman 2007), makes for an interesting arena in which power and control can be both exercised and negotiated.
Patrons or funders are often found to have more influence than other stakeholders, resulting in an emphasis on ‘accountability upwards’ (Ebrahim 2003, p. 314). This pattern of asymmetric power relations is becoming more prevalent across the world as grant making and philanthropy is moving from a previous position of a ‘fund it and forget it’ approach (Leat 2006) to a view that sees funding as an investment that requires understanding of ‘value for money’ (Ostrander 2007). In many cases organisations are under contractual obligations to report achievements to grant or contract providers. Requests for evaluations and social performance reporting are seen as a way for funders to manage the performance of organisations without having to manage the organisations themselves (Paton 2003, p. 14).
There is relatively little research on the behaviour of non-profit organisations in relation to social impact measurement, including the effect of control and disciplining exercised by resource providers. Studies of international nongovernment organisations (NGOs) show that the way these organisations behave in relation to financial and social auditing can be strongly related to the power of outsiders (Ebrahim 2003; O’Dwyer and Unerman 2007). Ebrahim (2002) argues that power asymmetry is an essential component that influences the way NGOs experience new accountability regimes. The meaning of information, suggests Ebrahim, is both symbolic and strategic and although organisations show resistance, their behaviour results in the reinforcement of this asymmetric relationship. In a study of Belgian NGOs exploring variation in compliance with rules of financial transparency, differences in the nature of compliance is explained through resource dependence and coercive isomorphism (Verbruggen et al. 2010).
Discomfort and Resistance
Evaluations as surveillance, with the possibility of disciplining, controlled by outside stakeholders cause discomfort among groups that are at the receiving end of this control. The experience and response to this takes different forms depending, among other things, on the nature of this discomfort and what room for manoeuvre the situation offers. Transparency based on measurable indicators and audits made accessible to outside stakeholders can support the appreciation of some aspects of work. But it also threatens the autonomy of staff and organisations, and this causes anxiety and discomfort (Levay and Waks 2009; Lipsky 1980). Discomfort also comes from the way evaluations are framed and the competing priorities and conflicting values they may present to organisations and their staff. O’Neill describes how an encompassing audit culture, aimed at restoring trust in institutions, both ‘distort[s] the proper aim of professional practice’ and ‘damage[s] professional pride and integrity’ (2002, p. 50). Similarly Hwang and Powell (2009, p. 268) report on loss in core organisational values that come as a result of ‘the integration of professional ideals’, some of which are related with evaluation practices. This illustrates discomfort caused by tensions within the organisation as well as in relation to external actors caused by the introduction of evaluation and social performance measurement.
Although there is asymmetry in allocation of power based on resources, this paper examines the responses of organisations to demands for evaluations and explores how those receiving funds can also exert power in the form of resistance to controls proposed by others (Clegg 1989). Evaluation practices can be used to actively shape the organisational environment and challenge authorities (Asdal 2011). Key to understanding organisations’ behaviour in the context of non-profit organisations is the discretion, or room for manoeuvre, offered at various points in the social impact evaluation process. One may assume that there is greater opportunity to make use of discretion with the purpose of subverting power when there are contested discourses of what is deemed ‘social’. This paper understands social impact as a social construction: there is no clear definition of what is meant by ‘social’ and consequently there is discretion involved when social impact is assessed (Barman 2007; Lyon and Sepulveda 2009; Hall 2012). This provides room for organisations to forward particular interpretations of the concept through their evaluations.
While there is evidence that funders and other resource holders exert control through directives related to evaluation, others highlight that organisations may hide information, or control the flow and content of information (Burger and Owens 2010). Burger and Owens (2010) suggest that this is a ‘deflection strategy’ used by organisations to cope with control imposed by stakeholders that hold unrealistic expectations, and to cope with a competitive climate. Based on a study of five social enterprises in the UK, Nicholls argues that evaluations and audits are used as means to ‘enhance social mission rather than merely to respond to regulation’ (2009, p. 756) and suggests that ‘emergent reporting practices constitute a spectrum of disclosure logics that social entrepreneurs exploit strategically to support their various mission objectives with key stakeholders’ (p. 756). Here, impact measurement is part of the process of negotiation between stakeholders and can be seen as a symbolic mediator. In another study, Nicholls shows how a ‘flexible reporting format can be used strategically in various ways by companies according to their particular objectives and resource limitations’ (2010, p. 407). Discretion, embedded in such a flexible format, can be seen as a key element of social entrepreneurial activity, with organisations providing different information for different audiences as they reach their objectives through impression management (Osma and Guillamon-Saoring 2011; Teasdale 2010).
The research reviewed above suggests that, faced with competing demands, non-profit organisations seek different ways of avoiding and coping with potential conflicts that may arise both internally within the organisation, and in relations with external stakeholders. There is a need for analysis that unpicks organisational strategies used in situations of power asymmetries and competing demands. The concept of decoupling is used to explore this space.
Decoupling conveys an understanding that rational plans seldom correlate with what actually goes on in organisations (Weick 1976). Rather than seeing organisations as sets of controlled activities we appreciate that they encompass highly complex relations and policies that are not easily coordinated (Meyer and Rowan 1977). The conceptualision of decoupling has shifted over time, moving from simply pointing out a discrepancy between plans and action, to asking questions about the source and motivation of decoupling and what different types of decoupling processes we may encounter. We therefore introduce a perspective on organisational behaviour that considers organisational actions and strategies in relation to powerful audiences and internal organisational dynamics (Pache and Santos 2010; Sandholtz 2012). Decoupling is for example described as an action that aims to show compliance with norms of transparency to an outside audience, while at the same time also preventing full insight and in order to protect professional autonomy (Levay and Waks 2009). Words such as concealment (Hirsch and Bermiss 2009), resistance (Levay and Waks 2009), and symbolic compliance (Sandholtz 2012) suggest a defensive motivation for a decoupling of system from action. A slightly different take on the process is presented by Hirsch and Bermiss (2009) who use ‘strategic decoupling’, which implies more proactive agency in a decoupling process where actors use decoupling to ‘creatively navigate’ change. Such an engaged way of decoupling is also expressed in Pache and Santo’s (2010, p. 463) ‘manipulation’ where, for example, the organisation tries to ‘influence the definition of norms’ or to ‘control the source of pressure’ they may find themselves subject to.
The act of decoupling directs focus to the nature of relations between the organisation and outside stakeholders, where typically external actors seek control (Levay and Waks 2009; Sandholtz 2012). In a recent study by Turco (2012), the author shows how decoupling reflects conflicts that come from outside as well as within the organisation and has a two-fold purpose. Firstly, as a way of complying with values that were not easily reconciled (such as values of the commercial market on the one hand and the norms of providing care on the other). Secondly, decoupling was used to contain conflicts between groups of staff that represent different sides in the conflict caused by competing demands.
Using decoupling as a conceptual point of departure supports a call for institutionalist approaches to better recognise agency in organisations (Lawrence et al. 2009; Tracey 2011). Decoupling summarises a range of responses that are different in nature and that emerge over time, and studies of organisational behaviour increasingly seek ways of moving beyond an analysis that simply views decoupling ‘in binary terms as being either present or absent’ (Sandholtz 2012, p. 659). Pache and Santos (2010) discuss a ‘repertoire of responses’ (p. 462) and use the nature of demands and internal representation as the two key points of departure when explaining different types of organisational responses to conflicting demands.
This paper therefore explores organisational responses, including decoupling by firstly looking at the nature of the relationship between non-profit organisations and external stakeholders, with associated power asymmetries based on resources, and information asymmetries (Kendall and Knapp 2000). Resource holders demanding social impact evaluations are often remote from the beneficiary groups, and any information about the fate and views of beneficiaries is mediated by the organisation.
Secondly, we emphasise the nature of the demand. In the context presented here the ‘demand’ is represented by the practice of social impact evaluation. The study explores how demand is characterised by degrees of uncertainties that allow non-profit organisations to use ‘creative techniques of strategic decoupling’ (Hirsch and Bermiss 2009, p. 265), to subvert control exercised by resource holders. In the analysis we refer to the concepts of control, discomfort, resistance and discretion. These concepts express the nature of both relations and the demand put on organisations, and they describe the basis on which their responses are formed.
The data comes from qualitative interviews that addressed research questions concerning how organisations experience and act on pressure to conduct social impact evaluation, and how they conduct such evaluations including the reporting of results. Data was gathered from individuals that play key roles in making the decision of how and what to measure, such as chief executives, marketing and finance managers, or senior service delivery managers. The cases are all non-governmental, non-profit organisations, defined as those with restrictions on distributing assets to individuals, having social aims and governance systems independent of the state. All cases had charitable legal status that requires them to state their public benefit each year.
A theoretically informed non-probability purposive sampling process was used to produce a cross section of different sectors and sizes of organisations with a multiple case study design (Yin 2003). Details of the case studies are provided below. A sampling frame of organisations carrying out social impact reporting was built up following responses to a request put out by support organisations and on-line networks to their members. Starting with a sampling frame of 80 organisations in one of the nine regions of England, structured interviews were carried out with 32 third sector organisations. Within this group more detailed information was collected through open-ended interviews from 14 detailed case studies. All respondents received detailed explanations of the purpose of the research and how the results would be kept anonymous.
The interviews were carried out face to face, by the authors with additional interaction by telephone and email where necessary. An interview protocol was used which explored the motivations and challenges of using their social impact measurement and reporting approaches, with the opportunity for detailed probing on specific issues. The data collection also involved analysis of their evaluation reports where available, and material in the public domain (websites, articles, etc.). Through cross-case comparison (Yin 2003), common issues were drawn out and attention given to any evidence that might refute emerging conclusions.
To ensure analytical rigor, a further ten organisations or individuals providing training in social impact measurement to organisations were also interviewed. This allowed an element of triangulation as they were involved in many of the evaluations completed by the case studies. Data were also collected on the issues raised by the attendees of two training courses on Social Impact Measurement, one covering Social Return On Investment and one covering Social Accounting and Audit, each of which had 12 participants. Further triangulation was gained from reporting the early findings back to respondents and other organisations and seeking their views on whether it was an accurate reflection of their activities.
The details of the case studies
Social impact reporting methods
Homeless support wanting to expand but requiring funding and planning permission
Tackling eating disorders funded by the health sector and donations
Employment support for the disabled and with large contracts from the Department of Work and Pensions
Funding organization that provides small grants after pooling individual donations and legacies
Supporting homeless through providing housing and recovery program, income generated from recycling business
SCBA and setting up own
Training in cooking to improve health, with funding from a single grant which ends soon
Training, employment and housing for ex-offenders with funding primarily from public sector contracts
Support for victims of crime
Housing for people with severe and enduring mental health problems funded through public sector grants and contracts
Creating partnerships between schools and businesses for work experience
Arranging yearly arts festival; creative Partnership with schools; various arts projects
Supporting people with learning difficulties
Support service for families and children at risk, counselling and working with children with behavioural problems
Supporting voluntary sector, placing volunteers, lobbying
Context: Social Impact Measurement and New Welfare Policies in the UK
In the context of non-profit organisations, measurements, and evaluations have taken on different roles over time (Barman 2007), and it is therefore important to recognize the context in which the case study organisations operate. The impetus to focus on social impact measurement among non-profit organisations in the UK has recently increased through the introduction of new government policies (Department of Health 2011; HM Government 2012) that place greater emphasis on competition for contracts, user choice and increased efficiency. Further impetus has come from philanthropic funders and grant makers wanting to demonstrate their own impact and use performance measures in their decision making related to the allocation of funding. These resource holders also promote evaluation and social impact measurement as a way of encouraging learning and self reflection within organisations.
As a consequence an array of new evaluation frameworks have emerged. Many of those are adapted from models of evaluation and accounting used in private and public organisational settings. The Social Return on Investment (SROI) method, introduced through a government sponsored project, is one such evaluation framework, which explicitly aims to mirror the private sector model of assessing return on investment with the reporting of a ratio of investment to financial values of social benefit (Nicholls et al. 2008). Other approaches include Social Accounting and Audit (SAA), Balanced Scorecards, Triple Bottom Line accounting and Blended Value (Emerson 2003). In this study location, the Soft Outcome Universal Learning (SOUL) method was widely promoted, with three of the case studies using this approach that maps out the ‘distance travelled’ by clients following support. While some question whether frameworks based on the private sector experience are entirely suitable, others argue that unless organisations use frameworks and language that are recognised by the wider system of decision-making in our mixed economy of welfare, the value and achievements of social purpose organisations will not be acknowledged (Gibbon and Dey 2011). There are also tensions around methods used in evaluations where for example some favour statistics and ‘objective’ indicators over ‘subjective’ case studies (Hall 2012). Debates furthermore focus on the use of market based language to express the achievements of non-profit organisations (Gibbon and Dey 2011), what role evaluations should play in the commissioning of contracts (Jones and Liddle 2011), the standardisation of social impact reporting (Principles into Practice 2012).
The findings provide important insights into how the studied organisations experience a situation of increasing demands for social impact evaluations. They reveal compliance, rejection, and organisations come to use evaluations as a strategy to promote their work. The nature of relationships with outside stakeholders, and the nature of social impact evaluation inform the way social purpose organisations engage in strategic decoupling.
Power and Pressure From Outside
The organisations measuring impact suggested a range of motivations for doing so. There was an overwhelming reference to outside stakeholders imposing demands. Pressure from agencies and public sector commissioners was cited as most important (stated by 19 of the 32 interviewees). Such pressure comes through requirements of social impact data in applications for grants as well as requirements that organisations collect such evidence once they have received funding. For organisations delivering public service contracts evaluations and some forms of social impact measurement are seen as necessary elements in meeting the requirements stipulated as directives by the resource holders.
Organisations reported that they had recently noticed a marked change in how public sector commissioners, and grant making bodies talk about social impact and evaluations. An established organisation working with ex-offenders noted that ‘There has been a big push from funders such as the Big Lottery’ and the CEO noted that ‘it’s a challenge to deal with the new term ‘social impact’ that authorities are talking about now. And we know that in the near future the local authorities will require us to use a particular evaluation tool. Before, they have specified what information they want but now they also specify what tools we should use’ (case 12).
Motivations to conduct some level of social impact measurement reveal explicit directives from resource holders, but this pressure also has a more implicit source. Organisations find themselves in an environment that is based on new types of contracts and relationships, characterised by competition. Detailed evidence of social impact is seen as a necessity and there are norms emerging related to expectations of funders and commissioners. This is clearly expressed by a large charity delivering government funded support for the unemployed, who found that evidence of social impact was needed because ‘we are up against national and international organisations when bidding for work so if we are close to their price but we can show all these added benefits… We see ourselves as a charity that runs as a business… it is not enough to have a good mission… we need rock hard evidence so this forces us to be more businesslike’ (case 7). The interviewee continued by explaining that before they could be ‘a bit airy fairy’ when discussing achievements, as opposed to ‘now you need more than a picture of a smiley face’.
Other cases expressed similar views and reported that social impact measurement is essential in justifying the cost of their services to resource providers and in gaining credibility as a professionally run organisation in an environment where for-profit and not-for-profit organisations are competing for the same contracts. Organisations clearly describe themselves as being at the receiving end of a request posed by individual funding bodies as well as by a general climate that favours evidence of achievements.
We can detect different sources of discomfort in the way our interviewees experience the pressure for them to conduct impact evaluation. Organisations expressed discomfort with being controlled by outside stakeholders who demand information on how the organisation is operating and what it is achieving. But there was also discomfort and resistance that caused tensions within the organisation. While senior staff tended to support social impact measurement, comprehensive assessment exercises caused protests and internal tensions.
Staff would resist being managed in this way, and one manager described how ‘it was like pulling teeth’ (case 6) and the regional manager of another organisations stated that ‘staff are different here… they have to be treated with cotton gloves……many of them think that ‘I have done like this for 15 years, there is no need to change!’ It is because of a feeling of ownership, that they have initiated activities, and they don’t want to change on anyone else’s initiative’ (case 13). This senior manager continued to explain that staff resist as a matter of principle: they do not want to be supervised. The manager also believed that staff disagree with accepting funders’ requests and that behind the drive towards evaluations lies a drive to make money. He had noticed a clear change though, as the organisation adapted to the new environment and began to engage with devising their own tools for measuring impact. Staff at all levels became familiar with the new procedures: ‘They are just insecure and feel uncomfortable about change. The challenge is internal organisational culture. Two to three years ago there was definitely some resistance… but ideas have percolated and staff are beginning to get what this is all about. Still some resist as a matter of principle’ (case 13).
As the measurement procedures become embedded within the organisations, staff began to express further discomfort about the impact on organisational relations. In one case, the organisation was meant to ask clients to fill in a lengthy questionnaire (case 11). Since staff felt it inappropriate to pressure clients, often with poor literacy skills, to deal with such a task, they managed to negotiate funders to agree with an alternative way of assessing their work. Several others described how they feared the evaluation tool would disturb relations with clients (case 29), were intrusive (case 14), used inappropriate language (case 18), and requested data that they could not obtain for ethical reasons (case 12).
Impact Evaluation for Promotion
However, analysis of the cases shows that eleven of the fourteen cases were also using the impact measurement and evaluation process for purposes other than reporting to funders and learning within the organisation. Information based on social impact measurement was used to gain the attention of future possible funders and also used to attract donations from new groups (individuals, philanthropists). A small foundation giving grants to other small third sector organisations had invested in a ‘Social Return on Investment’ evaluation, i.e., a type of cost-benefit analysis of their services, and used the report for attracting donations: ‘We are here and we are good, so how can we show it? Philanthropy of people often goes to organisations they are linked to, like cancer or cuddly animals, but if we can show added value then…’ (case 9). Whereas organisations have previously used a combination of persuasion and personal contacts to secure resources, impact measurement was reported to be an increasingly an important additional tool in this process: ‘if we have data we have the facts and figures… it sets you apart from other organisations, it’s a strength’ (case 6).
Social impact measurement complements an existing strategy based on personal contacts and networking that aims at getting the attention of resource holders. In this sense it is not seen as a threat, but rather as a welcomed additional tool. Analysis shows how complying with an agenda set by outside stakeholders can lead to initial discomfort and resistance, followed by acceptance. One children’s support organisation stated: ‘We have been asked to use this (impact tool) by authorities….I thought at first ‘What a waste! Just a lot of paper with no value!’. The very same manager then goes on to say: ‘But now I have changed. It has made us change what we do and how we do it, as a response to the evaluations we do now’ (case 29). Initial resistance has been replaced by compliance and agreement. It has become embedded in several dimensions of organisational operations. The evaluation is not just a task carried out by a particular group of staff (managers) at specific points in time, but has become part and parcel of organizational activities and engages staffs that are involved at the interface with service users. One organisation (case 30) was explicit about the dual function of impact evaluation. Data produced from formal measurements and evaluations is used to persuade resource holders: this is an important and external audience. Evaluations are also used for internal management to “boost the morale of staff”.
Creating Space for Discretion in Social Impact Measurement
The findings show how there is discretion in what is measured and organisations can therefore shape the evaluation process to meet other objectives. By discretion we mean a space for organisations to make choices about what to measure, how to measure, and in some cases what value to attribute to costs and outcomes.
The discretion embedded in evaluation procedures is due to the lack of consistency or agreement on what counts as good social impact evidence (Arvidson et al. 2013; Hall 2012; Hall and Arvidson 2013). A similar lack of consistency is found with social reporting by for profit organisations. The indicators and evidence chosen to evaluate social impact can be highly subjective. The selection of suitable indicators is a particular challenge for those organisations that are using evaluation frameworks that monetise social impact and use a cost-benefit analysis approach. Such evaluations rely on existing data (i.e., data that pre-dates the start of the evaluation process) and data that are easily quantifiable. One organisation (case 1) using Social Return on Investment (SROI) explained: “we have some [indicators] but she [the consultant] wanted some [indicators] like the impact on Accident and Emergency… she encouraged us as that is where we can get financial information”.
There is scepticism about the value of evaluations based on frameworks that allow organisations with a vested interest to make such a choice. The discretion is a space where organisations can be creative and formulate their own evaluation template. They are then able to disseminate and present findings in the way they feel comfortable with and that suits their needs.
An evaluation has a ritualistic, symbolic value that is believed to prove the organisation to be professional. One organisation (case 9) was very clear about the effect of using the results of the evaluation they had undertaken: “The headline figure is what is quoted and what we are interested in… so it is marketing but marketing with teeth.” Measures based on financial values were seen as providing particularly powerful messages. While recognising the limitations of using financial ratios and quantitative data in the context of understanding social impact, organisations wanted to present the results for marketing purposes. This affected the way organisations made strategies regarding the nature of the information they would include in reports.
A university based external evaluator of one case study was concerned that “it was only the summary that was published, and there was so much fuss about being able to produce the numbers that show the great work they are doing… they were not interested or keen in having this (the full report) widely published”. In a glossy brochure, the organisation published the final findings of the evaluation, emphasising the impressive return-on-investment their work generated. A second report, containing case-studies, was published online in a location unlikely to be found without direction. This detailed study included caveats regarding the cause and effect related to specific interventions that are trying to address complex problems and which are involving multiple other agencies at the same time.
Using results selectively to present different stories to different audiences was made particularly explicit in the following quote by one interviewee who stated: ‘There are different things to prove to different people… we had to mix and match those different things. We are quite frank about all those different things but had to recognize different agendas and different expectations in terms of demonstrating what we did.’ (case 1). Such approaches to impression management play a central role in strengthening the position and legitimacy of organisations.
We have described how organisations experience and respond to recent pressure to provide evidence of achievements based on social impact measurements. The cross-case analysis of the material reveals debates within organisations about the role, use, and effect of evaluations, and how social impact measurement can be used to mediate control and power in organisations’ relationships with resource holders. We find that both discomfort and enthusiasm can be used to describe these debates. Interestingly, these two opposing emotions cannot be used as a basis for dividing organisations into two camps, one characterised by strong resistance, and another by organisations engaging with social impact measurement. Rather the experience of engaging in social impact measurement and reporting generate both discomfort and enthusiasm.
The concept of decoupling, the loose connection between an organisation’s formal strategy and its actual actions (Weick 1976), suitably describes a range of strategies used by non-profit organisations to handle pressure to comply with social auditing regimes. In the literature review on the concept we furthermore identified both defensive (Levay and Waks 2009) and proactive qualities in decoupling as a strategic response (Hirsch and Bermiss 2009). By applying the concept on the behaviour of non-profit organisations the study contributes on two accounts. Firstly, it directs focus to organizational agency and thereby questions isomorphism and an implicit view of organisations as passive responders to directives from resource holders (Verbruggen et al. 2010; Burger and Owens 2010). Secondly, the study contributes with a detailed understanding of responses to competing demands. We suggest two key points that supports a more precise model of what informs the way non-profit organisations respond to demands for social impact auditing. The nature of relationships with resource holders is characterised by asymmetries of power based on resources and access to information. Furthermore, the nature of social impact evaluation as both regulation and voluntary disclosure offers room for manoeuvre that allows organisations to manipulate the way evaluations are carried out and information is presented.
Compliance with requests to conduct impact evaluations can be based on an acceptance of directives and internalisation of norms within an organisation. This can be seen in Fig. 1, where strategy 1 represents those organisations that comply and accept the directives coming from resource holders. A second response (strategy 2) is a simple outright rejection of the need to measure social impact, particularly when organisations have greater financial independence or more control over resources. In such cases, they may have no need to seek resources from those demanding social impact measurements.
The third response is more interesting and shows evidence of types of decoupling. Where there is an element of resistance and reluctance to respond to a directive, there can be symbolic compliance with minimal reporting. Organisations see the performance measurement as an identity threat since the basis on which measurement is carried out does not tally with organisational values and logics: ‘Financial and non-financial conceptions of performance live uneasily side by side’ (Power 1999, p. 97). Nevertheless, by complying they can ascertain that relations with resource holders are not disturbed.
In such cases, social impact measurement is seen as an additional burden or ‘red tape’ rather than a useful set of tools that can be embedded in actual strategies of non-profit organisations. Social impact reporting is seen as a requirement to gain access to resources and as a means to satisfy stakeholders based on a minimal level of compliance. It is there a symbolic tool that is used to signify that the organisation is professional and open to scrutiny.
This study also identifies a forth response (strategy 4) where there is a shift to greater acceptance after initial resistance. Organisations that have engaged with social impact evaluation for some time begin to appreciate the value of this exercise to the organisation itself. Managers see how they can use it to encourage performance among its staff, and how evaluation results lead to some reconsidering the use of resources and strategies within the organisation: there are incentives to embed evaluation procedures in the organisation. As a result of engaging in social impact measurement, managers begin to learn more about how this exercise offers ways of protecting the organisation from outside control. However, there may continue to be a disjuncture between the objectives of leadership and other staff.
Figure 1 finally shows a fifth strategy, where there is a proactive decision to undertake voluntary social disclosure. In such cases, an organisation can use their own discretion. Organisations symbolically adopt the practices of social impact measurement and use it to implement alternative strategies, such as ensuring accountability to users and other stakeholders, improving services or marketing their own organisation. The latter objective is more problematical as the tools are designed to provide accountability or organisational learning, and are not designed to be used for comparative purposes (as they do not have consistent methodologies). However, the competitive environment means that setting an organisation apart from others and demonstrating competitive advantage is an important motivation and justification for allocating an organisation’s own resources to the measurement process.
Decoupling as used by the case study organisations reveals a creative process, characterised by agency and critical reflection much like ‘strategic decoupling’ described by Hirsch and Bermiss (2009). Social impact measurement becomes a marketing tool and a means to ‘set your organisation apart’ from other organisations that have not yet mastered impact evaluation; it becomes an identity marker rather than an identity threat.
The proactive decoupling strategy is based on an engagement with the impact evaluation but one that is determined by organisations themselves. By simultaneously understanding the intricacies of social impact measurement and the context in which it is used, organisations can use their discretion to shape the evaluation process and use evaluations to promote their organisation. This gives organisations opportunities to prevent their organisation from being exposed to outside forces they cannot control, and to protect it from evaluation logics that may negatively affect the organisation.
The results of this study show how social purpose organisations experience and respond to the pressure to conduct social impact evaluations. There is unease about what and how things are measured and how this interferes with organisational operations and values (Christensen and Ebrahim 2006; Townley 2011; Turco 2012). This paper shows that there is resistance to the demands of funders but at the same time organisations can voluntarily decide to measure social impact for their own purposes. Our analysis brings to the fore how organisations critically reflect on what social impact measurement means to them and how they consequently develop strategies to deal with these requests from funders or to meet other objectives. Reflections reveal how organisations consider how they can balance the need to comply in order to gain legitimacy, status, and comparative advantages in a new and competitive climate, but also a need to resist the power and control exercised by outsiders from undermining their independence.
Theories of decoupling can explain how organisations may respond to resource holders’ demands while resisting full compliance with these demands. However, this paper develops the theory of decoupling by examining the agency of organisations and their ability to use evaluations for alternative approaches, beyond satisfying resource holders and internal learning. We refer to this as proactive or strategic decoupling. What characterises the proactive and strategic decoupling, as used by the non-profit organisations, is how it both subverts and confirms power-relations. The ambivalence and tensions expressed by organisations reveal a great deal of the changes in organisational fields, of the nature of relationships and that of evaluation frameworks.
The analysis distinguishes different organisational responses to the discomfort raised by external demands for impact evaluation. It points at two key issues to take into account when exploring what underlines these responses: the nature of relationships with outside stakeholders, and the nature of the demand (Pache and Santos 2010). There is much scope to explore further issues here to arrive at more specific models of responses. There are also questions concerning the sustainability of strategic decoupling that may seek to explore how it affects relations with external stakeholders and the morale of staff within the organisation in the long term.
This paper is part of a programme of the Third Sector Research Centre (TSRC). The support of the UK’s Economic and Social Research Council (ESRC), the Office for Civil Society and the Barrow Cadbury UK Trust is gratefully acknowledged. The authors are grateful for the input from Ian Vickers, Sarah Dodd, David Etherington, Richard Palmer-Jones, Simon Teasdale, the attendees of the TSRC workshop on Researching Impact Measurement, and the participants of the 2012 EGOS panel on Collective Processes of Social Innovation. All views expressed in the paper are those of the authors.