One of the authorsFootnote 1 used the development of a theory of change, based on PIPA, when supporting a regional TDC. This consortium was at the initial stages of a 10 year strategic program. The aim was to strengthen economic and social structures in the region; this would arise from the power of collective knowledge and expertise development. The total budget, including in kind contributions from the various knowledge partners, was over half a billion Euros over a period of 10 years. The intention was to fund a total of 15 different projects within this 10 year program. In every project, relevant institutes of higher education (HEI’s), private companies, local governments, NGO’s and local associations and foundations were to collaborate. And every project would have to have a relevant societal contribution to the region. For the regional government, the main funder of this program, involvement in a knowledge intensive transdisciplinary program of this scale, was new.
One of the institutes of HEI’s involved asked for support to develop a set of indicators for monitoring and evaluation. The HEI attached great importance to the success of the programme and understood that a novel way of monitoring was necessary. It wanted to avoid the situation that only after completion of a project, success, or worse failure, could be determined.
Together with the main stakeholders, the regional government and the main HEI, a number of criteria for indicators were developed. For every project the basic requirements were the same. Indicators:
-
function to monitor whether the project will reach the goals set;
-
need to enable learning during the project, so as to stimulate changes and improvements;
-
should reflect characteristics of the specific project;
-
should be realistic to use, and this includes that it should be financially justified to collect evidence;
-
need to be endorsed by policy and politics.
We were then asked to support the development of indicators for the first projects to start. We proposed to use the PIPA approach. The joint development of a shared narrative and a monitoring framework, and the use of that framework to learn, were welcomed as very useful aspects. However, it became clear that it was not feasible to conduct workshops of up to 3 days, as described in the PIPA method. The main stakeholders suspected that that there would be insufficient support for such a time investment. A half day workshop was considered to be realistic. Knowing that STEPS (Ely and Oxley 2014) had positive experience with shorter workshops, we agreed to this timeframe.
The first projects to start, were two projects in the field of education and two in the field of medical imaging and materials. In all four projects, knowledge institutes were to collaborate with local organisations. In the two education projects, partners included public organisations, such as schools, private organisations, such as catering companies, as well as a variety of associations (parent-organisations, sports associations, etc.). In the medical imaging and materials projects, partners were predominantly private (companies, as well as business parks).
The workshops
PIPA inspired workshops were organised for each of the four projects. For these, representatives of the main stakeholders were invited. At every workshop, the regional government, the HEI’s (principal investigators as well as the board) and other stakeholders were present. The number of attendees, apart from the workshop organisers, was between seven and ten.
Each workshop was organised at the offices of the HEI, at a central location in the capital city of the region. The workshops lasted half a day and they were led by the same two researchers. The workshop started with an introduction of the participants, followed by an introduction to the method of logical frames. Then the groups were split in two. Attention was paid to the composition of every subgroup: as diverse as possible. Participants were asked to identify the ultimate goal, activities necessary to achieve this, expected results, outcome and impact. Based on these, each subgroup constructed a logical frame. In the final phase, the two subgroups shared their logical frames and developed a common understanding of the pathways and of causalities. The causal relations and the theory of change were discussed. A final round of reflection marked the end of each workshop.
The result of the workshops was a list of goals and sub goals, activities, outputs, outcomes and impacts. This was used to develop a monitoring framework that consisted of criteria and indicators to monitor the progress towards the goals and sub goals identified.
To illustrate the type of goals formulated and the consequential monitoring framework, we present the goals and a number of proposed indicators for two different projects.
Case example: education
One of the education projects is dedicated to the development and application of a new concept for education. The aim is to improve the health of the schoolchildren. The concept affects the common daily routines and schedules of primary schools.
Three sub goals were identified during the workshop: (1) the development of an evidence based concept for this new education paradigm, (2) the sustainable realisation of four schools based on that concept and (3) the consequential application of the concept in other schools, both within the region, as well as beyond the region.
-
1.
The development of the evidence based concept relates to the scientific basis and the effectiveness of the concept, as well as to user costs, ethical issues and legal frameworks. The researchers involved in the field of educational science, focus on the scientific basis and on the effectiveness. To monitor the quality of their research, output indicators such as publications in academic journals, PhD’s granted or citations (scientific impact) can be used. It was identified during the workshop, that it was necessary to monitor the progress of the research in other ways as well. Intermediate analyses regarding the effects of the concept were proposed as necessary and relevant activities—and indicators to monitor progress. The validity of the legal framework is a different, yet equally important issue. The rules, regulations and laws regarding education are strict and need to be taken into account. Relevant indicators relate to activities such as the delivery of an inventory of legal and ethical boundaries and possibilities; to outputs such as the design for the management and supervisory board and to outcomes such as the decision on the legal form for such a school.
-
2.
The sustainable realisation of four schools is necessary in order to test and introduce the new concept. It relates to the involvement and commitment of teachers, parents and local organisations. Relevant indicators relate to activities such as organising information meetings, consultations of parents and to outcomes such as the participation of a required minimum percentage of parents.
-
3.
The application of the concept in other schools was not discussed during the workshop due to time constraints. However, it was identified as a goal in the long run.
Case example: new materials
One of the medical projects is dedicated to, again, the development and application of a new paradigm. This relates to novel approaches regarding the production of biomaterials. The core issue is the in vivo production of biomaterials, i.e. by living organisms such as plants. The project involved the erection of an institute centred on this new approach, so as to further develop the novel approach.
The sub-goals were: (1) the development of a sustainable research institute; (2) the achievement of a paradigm change regarding the specific biomaterials and (3) the strengthening of the innovation ecosystem in the region.
-
1.
The development of a sustainable research institute addresses the future, when no extra subsidies are guaranteed. Two aspects were identified as crucial: a solid financial base and ample good staff. Indicators that monitor these aspects relate to inputs, such as total income, number of different contracts, variety of contract partners, the increase in staff and the potential to attract excellent researchers.
-
2.
In order to introduce a novel approach, research and examples are needed and mission work needs to be done. Output indicators such as scientific publications and outcome indicators such as citations can be used to monitor. However, this relates to the quality of the research and to some extend to the acceptance of the new paradigm. Another way to monitor the acceptance of the new paradigm, is through collaborations with scientific peers and industrial stakeholders. Therefor the number of collaborations was defined as an indicator.
-
3.
The establishment of the institute in the region was going to have a positive influence on the regional innovation ecosystem. A series of indicators relating to business activities can be formulated ranging from number of collaborations with regional companies to the relocation of companies towards the region and the foundation of new companies.
General observations
The number of attendees at each of the workshops was between seven and ten. This is very little, given the amount and variety of stakeholders in every project, such as project partners (with whom collaboration throughout the project is necessary), next users (people and organizations that can or should apply the results) and end users (those ultimately benefiting from the research). It became clear that even for the main stakeholders involved, it was difficult to convincingly invite core stakeholders to the workshop.
During the workshops and through the discussions, it became clear that the responsibility for each of the projects is shared with many different stakeholders. They have different interests and different ideas about quality and relevance. Representatives of the regional government explained in a convincing way that next to good research it was imperative to have a convincing narrative for the regional politicians. They stressed that the politicians are interested in a more detailed narrative than the Bush narrative we referred to above.
Also, two very different logics became apparent. For the local government, the logical frame aims at how the programme as a whole and how each of the individual projects contribute to strengthening the economic and social structures in the region. For the researchers, the logical frame is about how to get funding that supports the kind of research that can lead to societal applications and to entrepreneurial activity.
An imbalance between the stakeholders came to the fore, regarding the robustness of the quality control. For example, when discussing the quality of research, the non-scientists found it difficult to identify ways to assess the quality and relevance of research in relation to the project, and thus to formulate indicators. The participating scientists rather easily formulated quality indicators (peer reviewed publications, PhD granted, conferences organised). This establishes research quality but not relevance to the project. When non-scientific aspects were discussed, such as the reallocation of certain companies to a campus or the willingness of a school to participate, the societal partners were able to identify milestones to assess the progress towards the ultimate goal.
It became clear form the workshop that it takes time to understand and accept each other’s approaches to quality and relevance control. In this case, it was the first time that the project aims and the impact narrative were discussed on such an intensive scale, and with an extended group of participants. The workshop proved to be useful as a means to bring stakeholders together and discuss the aims of the funding and the project. But it became clear as well that it takes more time in order to develop a true shared understanding and a shared responsibility.