Designing Ad Hoc Impact Monitoring Systems

Abstract

This chapter explores the significance of evidence-based decision-making, providing valuable insights into creating comprehensive impact follow-up systems and a battery of indicators to assess the outcomes of transformative governance efforts.

We can continuously improve and add value

by meeting the evolving needs of communities and actors.

This chapter explores the significance of evidence-based decision-making, providing valuable insights into creating comprehensive impact follow-up systems and a battery of indicators to assess the outcomes of transformative governance efforts. It highlights the importance of measuring progress and tracking the impact of transformative governance initiatives across various dimensions. By designing ad hoc impact monitoring systems, we can ensure that our efforts are aligned with the overarching goals of a just, sustainable and equitable future.

Institutions across sectors play a critical role in shaping societies, addressing complex challenges and producing societal, economic and environmental outcomes and impacts.

Up till now, Gross Domestic Product (GDP) has been the main measure of a country's economic performance. However, it is becoming clear that GDP does not capture the full picture of progress and well-being. The European Commission's Strategic Foresight Report [1] is taking a crucial step by exploring how to better measure progress and prosperity as they are developing Sustainable and Inclusive Well-being metrics that go beyond GDP. The goal is to include factors like quality of life, health, education, environmental impact and inequalities in measuring a country's well-being. The report introduces a pilot ‘adjusted’ GDP metric that considers the ‘health’ dimension of well-being, and further work will be done to incorporate other important factors.

In this new context, Environmental, Social and Governance criteria (ESG) have become a powerful framework that encourages responsible business practices, attracts investors seeking sustainable opportunities and promotes transparency and accountability. It is inspiring to witness the positive impact this holistic approach is having on the business world and our planet.

In a nutshell, the ESG framework can be described as the triple bottom line approach, encompassing environmental, social and governance factors.

The environmental dimension focuses on the effect of business activity on nature. Organisations must not only mitigate the possible adverse effects of their activity but are also required to undertake actions that generate a direct positive impact, for example, by reducing pollution, waste generation or the emission of greenhouse gases.

The social dimension relates with the communities where the company is present. It entails an analysis and an assessment of whether human resources promote equality and diversity among the workforce and social inclusion. Actions should seek to create a healthy space for human capital and the local community in general.

Governance encompasses all issues related to the company's corporate governance, purpose, culture, production processes and management. Corporate governance includes the following tasks: consider the composition and diversity of the board of directors, develop a code of ethics and best practice guide, check the supply chain to ensure compliance and provide transparent fiscal information in its accounts and all public reporting (Fig. 6.1).

Fig. 6.1

ESG graph © Eoh-for-good

This concept is being widely used today by companies, investors and regulations. Therefore, ESG has become a crucial factor in decision-making processes across various industries. Regulations and governments are playing a significant role in promoting ESG practices. Many countries have introduced regulations and policies that require companies to disclose their ESG-related activities and performance. This transparency not only raises accountability but also allows stakeholders to make informed decisions.

Companies have recognised the importance of integrating sustainability into their operations. They understand that being environmentally responsible, addressing social issues and maintaining strong governance practices are not only ethical but also make good business sense. By adopting ESG principles, companies can enhance their reputation, attract investors and even gain a competitive advantage.

Investors have started considering ESG factors when making investment decisions, even giving rise to the figure of the Socially Responsible Investor [2]. They realise that sustainable and well-governed companies tend to be more resilient and financially sound in the long run. By incorporating ESG criteria into their investment strategies, investors can align their portfolios with their values while potentially achieving attractive financial returns.

Key recent developments include the following [3]:

In 2020, the World Economic Forum and the Big Four accounting firms released a standardised set of stakeholder capitalism metrics to make ESG reporting by companies more consistent and easier to compare.

In 2021, the European Union's Sustainable Finance Disclosure Regulation went into effect, creating new sustainability reporting requirements for financial services and investment firms.

In 2022, the United States Securities and Exchange Commission similarly proposed rules amendments with more detailed disclosure and reporting requirements for investment funds that use ESG criteria. Also, the CDSB and the SASB standards were consolidated into the International Financial Reporting Standards (IFRS) Foundation, which plans to create a unified set of IFRS Sustainability Disclosure Standards.

In 2023, the EU’s Corporate Sustainability Reporting Directive went into force in January. Eventually, it will require 50,000 companies to file annual reports on their business risks and opportunities related to social and environmental issues and how their operations affect people and the environment.

By adopting a holistic and sustainable multi-level regenerative approach to innovation, initiatives could drive positive change that goes beyond mere financial returns. One of the primary outcomes of multi-level regenerative innovation is the creation of innovative products, services and processes that address pressing societal and environmental challenges. These innovations can contribute to the development of sustainable solutions that promote resource efficiency, reduce waste and pollution and mitigate the negative impacts of unfair and/or non-responsible socio-economic activities. By integrating environmental and social considerations into their innovation efforts (from a just triple transition point of view), organisations can not only drive positive change but also gain a competitive advantage in the market.

As explained in the previous chapters, the path towards shared positive outcomes and impacts necessitates several components: (1) unleashing the potential of intra-entrepreneurship; (2) driving institutional change through a collective process and (3) encouraging effective and fair deployment, outcomes and positive impacts of multi-level innovation processes of change, which requires a comprehensive and holistic approach.

6.1 The Importance of Measuring Progress and Tracking Impact Across Dimensions

This section explores the main reasons why all types of institutions, including public, private, social organisations, clusters, academia need to engage in comprehensive measurement and impact assessment across socio-economic, technological, political, environmental and governance dimensions.

If we are or want to become committed agents of change for the common good, we need to integrate the impact design, monitoring and collection from the very beginning in the everyday life of the organisation. This needs to be done at the suitable scale and with the available resources. For instance, it would be difficult for spin offs or start-ups, striving to get their business running to use many resources on monitoring and measuring in early stages. However, if the awareness of the importance of this dimension exists, little by little the design of an ad hoc methodology will start to unfold (Table 6.1).

Table 6.1 Element 10 of the compass

Measuring progress and tracking impact allows institutions to assess their performance and effectiveness for achieving their objectives. It enables them to evaluate their strategies, initiatives and policies in relation to desired outcomes. This evaluation nurtures transparency, accountability and continuous improvement, providing a basis for informed decision-making and resource allocation.

Measurement and impact assessment facilitate effective stakeholder engagement and collaboration. By tracking progress and impact, institutions can demonstrate their value and contributions to stakeholders, promoting trust and support. It also enables them to identify areas for collaboration, aligning goals and interests, leveraging resources for collective impact and enhancing synergies and cooperation among diverse actors.

The enhancement of collaboration and cooperation among different participants, within organisations or innovation ecosystems, fosters, synergistic partnerships and knowledge exchange, as explained in Chap. 4. By working collaboratively, these diverse key players can pool their resources, expertise and perspectives to co-create innovative solutions that are more effective and relevant to the needs of society. This collaboration can also lead to the emergence of new business models, networks and ecosystems that support sustainable development.

Comprehensive measurement and impact tracking generate valuable data and evidence that inform decision-making processes. Institutions can utilise this information to identify trends, strengths, weaknesses and emerging challenges across different dimensions. Evidence-based decision-making enables institutions to prioritise interventions allocating resources efficiently and addressing critical areas requiring attention, ultimately enhancing their strategic planning and policy formulation.

It is fine to follow social and environmental governance criteria, but we see that it is not enough to align with the SDGs or invest in X sectors or in X number of initiatives. It is crucial to monitor and actively manage and measure the impact.

For example: you can apply more global benchmark indicators depending on your type of investment. However, it is needed also having specific indicators for what you want to achieve (according to your theory of change, to the set agenda for the coming strategic period).

Measurement across socio-economic, technological, political, environmental and governance dimensions contributes to sustainable development and impact assessment. It allows institutions to assess the long-term consequences of their actions on society, the environment and novel and more coordinated governance structures. This assessment helps identify potential risks, unintended consequences and areas for improvement, promoting the integration of sustainability principles and responsible practices into institutional strategies.

Tracking progress and impact facilitates institutional adaptation and innovation. By monitoring performance across dimensions, institutions can identify changing trends and emerging needs and evolution of stakeholder expectations with anticipation. This information enables them to adjust strategies, develop innovative approaches and proactively respond to emerging challenges, ensuring relevance and resilience in dynamic environments.

6.2 How Can We Design Impact Follow-Up Systems and an Ad Hoc Battery of Indicators?

Organisations and innovation ecosystems can effectively assess progress, drive sustainable development and promote positive change across multiple levels and agents by establishing robust ad hoc monitoring systems.

Systemic multi-level and multi-agent processes of change within organisations and/or innovation ecosystems is a fascinating but complex task. It requires a clear understanding of the objectives and context, identification of key performance areas, selection of relevant indicators, establishment of baselines and targets, design of data collection methods, implementation of reporting and feedback mechanisms and continuous evaluation and learning.

Defining a battery of indicators and a monitoring system for different types of organisations or ecosystems require different approaches and efforts. The size, scope, maturity level, field or sector, etc., matter.

Effective design and deployment require coordinated efforts, resource allocation and the alignment of shared goals and strategies across the different actors involved in the process of change.

Although the design of an ad hoc system needs to go hand in hand with sustainability and economic results, each entity should start by questioning what is its added value for society and the planet.

First, it is crucial to strike a balance between ad hoc design and approaches that facilitate aggregation, comparison and standardisation. To avoid reinventing the wheel, it is important to invest in capacity building, leverage existing efforts and explore the support that new technologies and tested methods can offer to our specific needs. Existing indexes and methods provide a foundation upon which to develop tailored models for various initiatives, transformations and projects that we undertake. We require monitoring and dissemination mechanisms that accomplish the following objectives:

• (a) demonstrate progress;

• (b) incentivise and recognise exemplary practices and

• (c) identify and penalise implementations that harm ecosystems and communities.

The following steps could guide the process, although it is advisable to count on experts in impact design.

Step 1. The first step in defining a battery of indicators and monitoring systems is to clarify the objectives and the context in which the processes of change are taking place. Decisions made at Corporations’, entities, etc., positively or negatively impact the environment and society in general. Therefore, it is essential to assess each company's action holistically, i.e. individually and in its context and sphere of action. This involves understanding the desired outcomes, the scope of the change and the specific characteristics (the givens) of the organisation or innovation ecosystem. Consideration should be given to the multi-level and multi-agent nature of the processes, as well as the interdependencies and interactions between different interested parties. Key actors need to be identified and invited to lead the process as early adopters.

Step 2. Once the objectives and context are clear, it is important to identify the key performance areas that will be monitored. These areas should align with the goals of the processes of change and reflect the different dimensions of sustainability, such as social, economic and environmental aspects, in line with the triple transition presented in Chap. 1. For example, key performance areas could include innovation capacity, stakeholder engagement, resource efficiency, collaboration and the generation of positive social impacts.

It is important to bear in mind that ‘one size does not fit all’ and that every organisation, community and context will require a tailored, flexible, adaptable and feasible monitoring system. Comprehensive models require careful thought, the ‘perfect system’ is usually not an achievable one in terms of resources and timing.

Define and prioritise criteria in accordance with the dimensions and the areas of action identified in the theory of change of the entity or the innovation ecosystem.

Identify the requirements and regulations of the sector to guarantee compliance (directives, regulations, standards, etc.).

Step 3. Baselines and targets provide reference points for assessing progress over time in a longitudinal basis. Baselines represent the starting point, while targets set the desired level of performance to be achieved. These baselines and targets should be established for each selected indicator. Baselines can be determined through data collection, benchmarking against industry standards, best practices or expert opinions. As mentioned above, targets should be ambitious yet realistic, taking into account the organisation's or ecosystem's capacity and resources (Table 6.2).

Table 6.2 Elements 9 and 11 of the compass

Indicators are quantitative or qualitative measures that provide information about the performance or progress in specific areas.

It is important to select indicators that are relevant, meaningful and aligned with the objectives and key performance areas identified earlier. Indicators should be specific, measurable, attainable, relevant and time-bound (following the SMART¹ formula). They should include various types of indicators used in different fields and disciplines:

• 1. Outcome Indicators assess the desired outcomes and changes that occur as a result of the actions taken by a programme, project, product, intervention or service.

• 2. Process Indicators focus on the activities, tasks or steps undertaken to achieve a specific goal. They assess the inputs, activities and outputs to monitor progress and ensure effective implementation.

• 3. Input Indicators measure the resources, such as financial, human or material resources, allocated to a programme, project, product, intervention or service. They provide information on the investments made and resources used.

• 4. Output Indicators track the immediate or direct results of a process of change, a programme or project. They quantify the tangible products, services or deliverables produced as a result of the intervention.

• 5. Impact Indicators assess the broader and long-term effects of a programme or intervention. They measure the overall changes or outcomes at a societal, environmental or economic level.

• 6. Leading Indicators are used to predict or anticipate future outcomes or changes. They provide early signals or warnings about potential trends or developments.

• 7. Lagging Indicators reflect past performance or changes that have already occurred. They are often used to evaluate the effectiveness or success of a programme or intervention after a certain period.

• 8. Qualitative Indicators capture non-numerical data or information. They provide insights into subjective experiences, perceptions or qualitative aspects of a phenomenon.

• 9. Quantitative Indicators involve numerical data or measurements. They provide objective and measurable information that can be analysed statistically.

• 10. It is important to note that the specific types of indicators used vary depending on the context, purpose and/or field of application.

When generating targeted KPI and suitable impact monitoring systems, key questions are:

• Have we made the right questions from the beginning?

• Are we keeping on track the defined roadmap? And are we flexible enough to adjust and include changes over time during the process?— > efficient oversight to manage and supervise operations in a competent and streamlined manner;

• How can we keep record of the progress and monitor potential impacts?

• How can we show the added value and our contribution to society and to the well-being and regeneration of the ecosystems? So how can we disseminate what we are doing and achieving?

A good analysis on these topics will help us define meaningful workable impact monitoring systems and indicators. Follow up multidimensional:

• Outcomes and results (profit, revenues, jobs created)

• Environmental, Social and Governance (ESGs) indicators and beyond, (e.g. well-being, quality of life of communities, ecosystems).

• Continuous improvement and positive changes in society towards the just triple transition.

Step 4. Data collection methods should be designed to gather relevant information for each indicator. This can include surveys, interviews, focus groups, observation and the collection of existing data from various sources. It is important to ensure that data collection methods are reliable, consistent and suitable for the specific context (e.g. definition of a data management plan, data sets, sources of information, personal data, etc.). Careful though should also be given to data privacy and ethical considerations.

Step 5. Develop reporting and feedback mechanisms which are essential for communicating progress, sharing information and promoting accountability. In this sense, it is desirable for each organisation to communicate its mission, vision and values and provide objective and transparent information about its activity. What is not shown, it does not exist and its return on investment cannot be recalled.

Regular reports should be produced to provide updates on the performance of the selected indicators. These reports can be shared internally within the organisation or ecosystem and externally with relevant actors, such as investors, policy makers and the public. Feedback mechanisms should be established to gather input, insights and suggestions for improvement from interested parties.

Step 6. The monitoring system should be implemented to track the selected indicators over time. Effective monitoring, enforcement and accountability mechanisms are necessary to ensure compliance and address any potential negative impacts.

Continuous evaluation and learning are essential for improving the battery of indicators and monitoring systems. This involves the regular collection, analysis and interpretation of data. What should be done?

• 1. Regular evaluations conducted to assess the effectiveness, relevance and impact of the indicators and monitoring processes. Monitoring should be conducted at appropriate intervals, allowing for timely identification of trends, drivers, challenges and opportunities. The monitoring system should be flexible and adaptable, allowing for adjustments and refinements based on changing circumstances or new information.

• 2. Lessons learned captured and adjustments made to improve the system.

• 3. Stakeholder feedback and engagement that can provide valuable insights for evaluating and refining the monitoring system.

Encouraging effective and fair deployment, outcomes and positive impacts of multi-level regenerative innovation processes requires a combination of collaboration, stakeholder engagement, robust regulatory frameworks, equitable access to resources, continuous evaluation and knowledge sharing. Actors can work together to ensure that regenerative innovations are implemented in a way that addresses social inequalities, respects ethical considerations and maximises sustainable development outcomes. Such efforts are essential for creating a just and inclusive future where the benefits of innovation are shared by all.