The Institutionalisation of Evaluation Theoretical Background, Analytical Concept and Methods

Abstract

This book is offering a systematic selection of European countries and a theory-based approach for national case studies. The theoretical background is derived from social sciences and particular modernisation theories. Three social sub-systems are selected for further investigations: the national political system as the main authority for decision-making, the civil society for controlling and supporting political governance, and the systems of professions for implementing and improving knowledge and skills production. In legislation, for instance, evaluation can be linked to primary acts or to decrees and regulations. Additionally, the book emphasises evaluation policies, embeddedness in structures of parliament and other authorities, and the way evaluation is used in political decision-making. In civil-society, evaluation may improve services and state control, while academic study programs and research enhances together with self-regulating associations exchange and learning within the systems of professions. The introductory chapter describes this theoretical and methodological framework in some detail.

Appendix

1.1 Analytical Guideline: Compendium on the Institutionalisation of Evaluation

I. Institutional structures and processes (Political system)
I.1 Evaluation regulations	I.2 Evaluation practice	I.3 Use of evaluations
• Are there national laws or regulations about evaluation or use of evaluation? If yes, which? • Are there sectoral laws or regulations about evaluation or use of evaluation (e.g. a law about school evaluation or evaluation in the higher education system as example for laws in the educational sector)? If yes, which? • Are there policies or strategies about evaluation or use of evaluation, either national or sectoral? If yes, which? • Are there administrative regulations about evaluation or use of evaluation in different policy fields (instructions, guidelines, etc.)? If yes, which? • What is the content of these laws/regulations/policies/strategies or administrative regulations regarding independence of evaluation, quality, impact orientation and available budget? – Is use of evaluation specified? If yes, how? – How binding are specifications regarding use of evaluation?	• With regard to the whole country: How would you describe the scope of conducted evaluations? Is it possible to speak of a frequent rhythm of evaluations, for instance for every new legislation or for every national program? Or is it rather non-specific? Does evaluation take place in all sectors/policy fields of a country (instead of only in the field of development cooperation, e.g.)? And within one sector, is evaluation applied for measures funded in different ways or maybe only the ones that received funding by the European Union? • With regard to the whole country: How would you describe the relation between internal and external evaluations? Which form is carried out more often and for what purposes? • What are possible reasons for this (e.g. determination in laws, policies or regulations)? • Is this relation differing with regard to sector or state level?	• Which sectors are “good performer” regarding use of evaluation and evaluation findings? Please describe up to 3 sectors that can be considered as leading in the field of evaluation’s use • Which sectors are “bad performer” regarding use of evaluation and evaluation findings? Please describe up to 3 sectors that are lagging behind in the field of evaluation's use • On which aspect do most evaluations focus in these sectors (e.g. Planning and Steering, Accountability and Legitimacy, Enlightenment)? – In the case of different findings in different sectors: What might be possible reasons for these differences? • Which professional groups use evaluation and evaluation findings regularly (e.g. political decision makers, program or project manager, administrative staff)? – For what reasons are evaluations and/or evaluation findings used by these groups?
– What are aspired functions of evaluation (e.g. Planning and Steering, Accountability and Legitimacy, Enlightenment)? • Is evaluation and use of evaluation findings embedded in parliamentary structures? If yes, how? – Do parliamentarians in your country deal with evaluation findings for their own political work? If yes, to what extent (how often/how detailed do they use evaluation findings)? – Do parliamentarians in your country demand evaluations for their own political work? If yes, to what extent? (How often? Do they commission evaluations? Do they publicly demand evaluations)?	• With regard to the whole country: How would you describe the relation between process and impact/outcome evaluations? Which form is used more often and for what purposes? – What are possible reasons for this (e.g. determination in laws, policies or regulations)? – Is this relation differing with regard to sector or state level? • Does an independent evaluation institute exist in your country? – With a national responsibility? – With a responsibility for a specific sector or policy field? • Do independent internal departments exist, in ministries or elsewhere?	– How is the use of evaluation findings guaranteed (f.i.: management response mechanisms, implementation of monitoring for evaluation results, others)? – Are there differences with regard to different sectors? • How is the quality of evaluations guaranteed (e.g. regular conduction of meta-evaluations analyses, competence requirements for evaluators, quality requirements for evaluations)?

II. Societal dissemination/acceptance (Social system)
II.1 Institutionalised use of evaluations by civil society	II.2 Public perception and discussion of evaluation and evaluation findings	II.3 Civil societies demand evaluations
• Is it usual practice in your country that evaluations are used to provide knowledge for referenda or political decision making on a communal basis? – If yes, how regularly does this happen? If not, what might be possible hindering factors? • Are evaluations and evaluation findings used by individual citizens/civil society organisations and or private enterprises or other actors? – If yes, for what reasons (e.g. enforcement of their interests, knowledge or proof for work related issues, knowledge or proof for voluntary activities etc.) and how regularly? If not, what might be possible hindering factors?	• How well-known is the instrument of evaluation in society? • Are evaluation reports (full version) made publicly available? • Is the general use of evaluation publicly discussed in media (benefits of evaluation, quality of evaluations, and professionalisation of evaluation)? – If yes, to what extent? If not, what might be possible hindering factors? • Are findings of actual evaluations publicly discussed (surprising findings, different possibilities of dealing with these findings)? – If yes, to what extent? If not, what might be possible hindering factors?	• Do individual citizens, civil society organisations, private enterprises or other actors in your country demand evaluations, e.g. from political decision-makers? – If yes, how often does this happen and under which circumstances/for what reasons? If not, why not? What might be possible hindering factors?
• Is it usual practice in your country that citizens or civil society organisations (NGOs, CSOs, churches etc.) are participating in evaluations (as stakeholder)? – If yes, how regularly does this happen? What are different forms of participation (e.g. as interview partners, as clients, as users of evaluation findings etc.)? If not, what might be possible hindering factors?

III. Professionalisation (system of professions)
III.1 Academic study courses, further training et cetera	III.2 Profession/discipline	III.3 Compliance to standards and quality obligations
• Do programs of higher university education for evaluators (Diploma, Master) exist in your country? If yes, how many and where? • In which other scientific disciplines is evaluation instructed as scientific subject? Please give as many examples as possible. • Do other forms of academic or non-academic training exist? (e.g. e-learning, training by consultancies, else)?	• Which professional journals, newsletters or other ways/media of communication (e.g. e-Mail or discussion lists) exist? • Which professional journals from other scientific disciplines deal with evaluation regularly? • Does a professional organisation (VOPE - Volunteer Organisations for Professional Evaluation) exist in your country? • Do standards, guiding principles for evaluators or s.th. similar exist in your country? – Developed by the VOPE? – Adopted from another VOPE? • Would you say that the evaluation market in your country is mostly dominated by freelancer (people calling themselves evaluators), consulting firms or scientific research institutes?	• Do professional organisations ask their members to follow standards or guiding principles? If yes, how obligatory is this? • Do clients demand a certain evaluation quality and/or compliance to standards? How does this demand look like (is it obligatory)? • To what extent do evaluators (and clients) follow these standards and/or quality obligations?
	• Does a certification system for evaluators exist in your country? • Does an authority, which might be asked to conciliate in case of a dispute, arbitration board exist in your country, like an arbitration board or ombudsman? • Does a professorship for evaluation exist in your country?