Keywords

1 Introduction

Despite the prolific number of studies investigating the effects of EU funds, empirical evidence shows mixed, and, sometimes, contradictory, results (Gramillano et al., 2018a, b). While some authors found a positive effect of EU funds on the economic growth of regions (Puigcerver-Penalver et al., 2007), others highlighted the existence of a maximum desirable level of funds transfer, beyond which the funds may increase regional gaps within countries (Kyriacou & Roca-Sagales, 2012). Also, some contributions acknowledged no statistically significant impact of Cohesion Funds (CF) on convergence, underlining how disparities persist in the EU (Dall’Erba & Le Gallo, 2008).

As Scotti et al. (2022) emphasized, although the EU Cohesion Policy has progressively diversified the sectors targeted for funding, with possible heterogeneous impacts on local growth, the literature is still largely oriented to the analysis of aggregate impacts. Thus, these authors proposed a granular investigation of the sectoral impacts of Structural and CF on European NUTS 2 regions for the period 2007–2014 and concluded that expenditures in energy, Research and Development (R&D), and transportation sectors stimulated a higher Gross Domestic Product (GDP) per capita, consistently reducing production costs, increasing accessibility and innovation in the beneficiary regions.

Moreover, the assessment of funded programs is still a largely discussed topic. In this regard, the EC identified two main assessment problems with the system of indicators (Nigohosyan & Vutsova, 2018): difficulties in establishing the cause-and-effect relationships between actions, results, and impacts, due to the influence of external factors; and difficulties in measuring the impacts, because they are usually the cumulative effect of many actions, affect diverse populations, and it takes time for them to show their actual effects. For the 2014–2020 period the EC tried to overcome these difficulties in the ERDF context by discarding the concept of impact indicators and introducing a new intervention logic. Nevertheless, Nigohosyan & Vutsova (2018) claimed that despite the changes, some of the identified problems with the indicators were not solved. Besides, these authors suggested that ongoing evaluations should include activity’s in-depth analysis of the relevance of the selected output indicators. All in all, Nigohosyan & Vutsova (2018) consider that the introduced changes are well-intended, but they still need to be perfected. The system of common output indicators should be re-examined, while the requirements for the result indicators should not be contradictory. The current practices show that even solid and logical principles, such as the fact that EU policies should reflect economic, social, political, and institutional differences to maximize both the local and the aggregate potential for economic growth (Barca & McCann, 2012), might not be sufficient when the different countries select indicators.

Furthermore, in the current programming period (2021–2027), the lessons learned from other programming experiences within and beyond the European Structural and Investment Fund (ESIF) framework can make a difference, since ESIF operates in very different local contexts and handles very heterogeneous economic and social regional environments. In this framework, a previous study by Bachtler & Wren (2006) reported that even if Cohesion Policy had a unified regulatory framework, it should address different national and regional circumstances embedded in a variety of institutional arrangements, bearing in mind that its operations comprise a multiplicity of measures and a diversity of national, regional, and local rules and systems.

Additionally, programs consist of a range of interventions (physical and economic infrastructures, business and technological developments, human resources, innovation, and environmental improvement) based on a mix of financial instruments for many types of beneficiaries. This variety of targets and contextual conditions is per se a challenge for any evaluation exercise (Henriques & Viseu, 2022a, b).

In this vein, Nigohosyan & Vutsova (2018) emphasized that the EC should try, in the next programming period (i.e., 2021–2027), to unify as much as possible the understanding of indicators under the different EU funds. An easy step would be the inclusion of all indicators into a single guidance document. Their study assessed the possibility of expanding the current list of common output indicators and the feasibility of developing a list of common direct result indicators for post-2020 ERDF and CF interventions.

Regarding the allocation of the EU budget, Gramillano et al. (2018a, b) suggested that spatial and sectoral effects can contribute to the design of more effective distribution of EU funds. Also, due to the wide variety of projects financed by CF, the EU Regional Cohesion Policy has been defined as a “do it all policy”. Hence, policymakers are currently focused on the economic impact of investments across different sectors, since heterogeneous levels of local development may be achieved depending on the economic activity in which the EU transfers are allocated (Cortuk & Guler, 2015). Indeed, investments in certain sectors might have immediate positive effects, while other types of investments might generate a significant impact only in a long-term perspective (Scandizzo et al., 2020).

Moreover, the magnitude of economic multipliers might be different across sectors and dependent on the level of diversification and complementarity of expenditures (Auerbach & Gorodnichenko, 2012; Duranton & Venables, 2021).

With the foregoing in mind, we investigate the current major issues inherent to the assessment of ESIF committed to three TOs: R&I, LCE, and ICT. A literature review on ERDF implementations is conducted, with a specific focus on the PT situation, as well as its evaluation and reporting practices.

The research questions that we want to address are the following:

RQ1: “What are the main areas of concern of the studies devoted to the assessment of ERDF?”

RQ2. “What are the main challenges inherent to the selection of the variables/indicators in the assessment of ERDF?”

RQ3. “What are the best practices found in the assessment of ERDF?”

RQ4. “What are the main gaps found regarding the methodologies used to assess ERDF?”

This paper is structured as follows. In Sect. 2 a systematized literature review is conducted. Section 3 goes through the Portuguese case. Section 4 describes the main results found for the Portuguese assessment and reporting systems. Finally, some conclusions are presented, and future work developments are indicated.

2 Literature Review

A systematic literature review has been done on the Web of Knowledge database, using a set of keywords combined with Boolean operators, and it covers a wide number of data sources (i.e., scientific journals, books, proceeding papers, etc.). Data collected from EU Reporting were also attained in the analysis. The bibliographic results were saved in data files and comprise the search over all the publication content. These text files were collected and manipulated using Vosviewer (i.e., https://www.vosviewer.com/) to map the bibliographic content—see Figs. 1 and 2.

Fig. 1
An image depicts the complex network structures of Bibliometric pooling and E U merged with E R D F.

Source Authors’ own elaboration

Bibliometric clustering a and timeline b for EU combined with ERDF and funds.

Full size image
Fig. 2
An image depicts the complex network structures of Bibliometric pooling of central thematic and PT-related arcs arranged horizontally.

Source Authors’ own elaboration

Bibliometric clustering with highlight focus on a central thematic and b PT-related arcs.

Full size image

By using the keywords “EU funds”, the search returned 28,809 references that were filtered regarding the type of publication and then coupled with our focal interest, that is the “ERDF”. The latter generated 1,200 references, and while combined conjunctively (i.e., using the “AND” operator) with the “EU funds” keyword returned 532 references. These were extracted from the database to be mapped by Vosviewer. Besides, another collection of references was obtained by adding “Portugal” and “Funds” to the “ERDF” keyword.

As can be observed in Fig. 1a the research published so far regarding the EU funds mostly focus on the cohesion policy and, entering a deep detail, “structural funds” and “regional development” appear as commonly referenced keyword, as, mapped in Fig. 2a.

While introducing the timeline into the data collection (Fig. 1b), the global cluster results show that most references are up to 2016 (green). Also, more recent publications depict a higher concern over the following themes: “sustainability”, “smart specialization”, “high education”, or “regional disparities” (yellow). These are topics quite prominent in today’s EU political agenda and are in line with the latest worldwide concerns.

Finally, regarding Portugal (Fig. 2b), most of the studies address “regional development”, but “competitiveness”, “higher education”, “innovation” or “rural development” are also part of the connecting arcs.

Subsequently, the literature review was systematized by grouping the references into four categories: Data and variables selection; Indicators and monitoring; Best Practices; and Methods.

2.1 Data and Variables Selection

The lack of data and the heterogeneous definitions of relevant indicators further complicate the analysis of the Operational Programs (OPs) funded by ERDF (Henriques et al., 2022a, b). Both, policy and economic performance/outcome indicators can be measured/proxied by different variables and the choice of these proxies may have important implications for the results of the various analyses (Pastor et al., 2010). In most cases, the policy variables under study are ‘payments’ or ‘commitments’ and ‘GDP Growth Rate Per Capita’ or ‘Employment Rate’, which are used as proxies for economic performance. Therefore, the choice of the policy variables can be a determinant factor for the design and results of empirical analyses. For example, the use of actual policy ‘expenditure’ data instead of ‘commitments’, means having to consider the duration of the entire OP (Crescenzi & Giua, 2016).

Additionally, the data available to perform the assessments varies considerably with the type of TO under scrutiny. In the case of R&I (TO1), policymakers face additional challenges in the assessment of R&I policies, notably because of the scarcity of suitable data (Henriques et al., 2022a; Ortiz & Fernandez, 2022). The sparsity of regional-level research and data on ICT (TO2) at the firm level has also been highlighted by Ruiz-Rodríguez et al. (2018), Reggi and Gil-Garcia (2021) and Henriques and Viseu (2022a, b). For example, despite the ample set of data available in the latest European Innovation Scoreboard (Hollanders, 2021) with some indicators on ICT (e.g., Digital skills and business sector ICT specialists), Henriques & Viseu (2022a, b) only managed to consider three indicators for assessing the OPs related to boosting ICT adoption in SMEs (i.e., operations supported, eligible costs decided, and eligible spending). An additional difficulty refers to the identification of EU ICT targeted investments (Sörvik & Kleibrink, 2016). Aside from being an economic sector, ICT is still an essential portion of many other activity-related sectors (for example, e-Health) and a tool to assist other activities. Since ESIF actions might have multiple aims, it can be hard to pinpoint the ICT-related activities within the designated categories when the OPs are planned. The OPs’ financial metrics are organized into intervention categories, TOs, and priority domains. Moreover, although the EU guidelines advocate that planned ICT initiatives should be classified primarily underneath TO2, these obtain funds under distinct TOs, and they are also integrated into various smart specialization strategies. For example, to consider the ICT SMEs support, there are only two dimensions of intervention that can be considered (Henriques & Viseu, 2022a, b; Reggi & Gil-Garcia, 2021; Sörvik & Kleibrink, 2016) under codes 4 and 82, that correspond to €1.7 billion and €304 million of planned investments, respectively (Sörvik & Kleibrink, 2016). These totals are available under multi-TO (€810 million), TO2 (€790 million), and TO3 (€349 million) and to a smaller level under TO1 and TO8 (Sörvik & Kleibrink, 2016). As a direct consequence, national and regional policymakers should use additional specific criteria that account for ICT results; tag expenditure that falls under other TO (rather than TO2) but has an ICT component; enhance the quality and completeness of ICT performance data at the regional and SME levels; and unify different data from diverse data sources (Henriques & Viseu, 2022a, b).

Furthermore, as noticed by Henriques et al., (2022a, b), despite the performance framework providing a set of implementation indicators, the data provided is frequently incomplete, and as a result, assessments end up considering a limited number of indicators and OPs. Finally, it is not possible to reach a complete match between the data obtained for the OPs’ achievements and their financial implementation. This is especially true for the investment priority dedicated to SMEs (investment priority 4b), under TO4 (i.e., LCE), which is meant to increase energy efficiency and renewable energies and includes statistics for the accomplishment metrics but not for their financial implementation.

2.2 Indicators and Monitoring

Performance appraisal techniques, as well as the selection of indicators, provide critical insights into supported programs. The different types of indicators are based on the following aspects (European Commission, 2013): relationship with the variables (complete, partial, and complex indicators); information processing (elementary, derived, and compound indicators); information comparability (specific and common indicators); and information scope (context and program indicators). The primary categories of indicators (input, output, and result) are provided in all ESIF monitoring systems; however, there are significant differences. Apart from the division of indicators into quantitative and qualitative, the guidance documents for the ERDF and the European Social Fund (ESF) agree that indicators should be linked to the specific objectives and kept as close as possible to the activities. Regarding the result indicators, the major difference is that ERDF should not be limited to the supported entities since it addresses different local contexts and very heterogeneous economic and social regional realities, whereas ESF result indicators “capture the expected effects on participants or entities brought about by an operation” (European Commission, 2015). Therefore, the ERDF result indicators should capture the change in the Member States (MS), regions, areas, or affected populations because of a particular program. This leads to two main problems: difficulties in establishing a program’s contribution to results and problems determining the target values.

Nonetheless, the results of Nigohosyan & Vutsova (2018) show that not all of the common indicators suggested by the EC follow its guidelines and the new intervention logic model could still lead to unclear logical links between activities, outputs, and results of the programs. These findings led to the recommendation for a mid-term review of the common ERDF indicator system.

Nigohosyan & Vutsova (2018) focused on the evolution of the ERDF indicators in the 2014–2020 programming period. The main question addressed was whether the new indicator system solved the problems of the past. These authors argued that despite the changes, the evolution of the ERDF indicators is incomplete, and it will likely be the reason for serious monitoring deficiencies and evaluation challenges.

A new understanding of ‘impact’ and the exclusion of impact indicators was put forward in the 2014–2020 guidance for evaluating the EU Cohesion Policy and the new ERDF-supported OPs (Nigohosyan & Vutsova, 2018).

Nigohosyan & Vutsova (2018) argued that despite the good justification for these changes, the 2014–2020 ERDF intervention logic and indicator system did not solve some of the well-known problems and brought new challenges to the evaluation of the ERDF-supported programs. The main challenges that remain to be solved are differing indicator concepts across the EU funds; inconsistency of the common output indicators; difficulties in establishing a program’s contribution to results; persistent problems in determining the target values for results; and broad result indicators with an indirect link to interventions.

The same idea was reported by the EC (European Commission, 2018) clarifying the core differences between the period 2014–2020 and the ongoing Period 2021–2027. Supported on the key concept of simplification (i.e., more comparable data based on the use of fewer indicators), the 11 TO originated 5 Policy Objectives; the 3,573 Specific Objectives gave place to 21 Specific Objectives. Regarding the Results, the 5,082 program-specific indicators were replaced by 85 common indicators plus program specific; and on the outputs reporting side, 46 common indicators (6,481 records) and 4,813 program-specific indicators, were replaced by 85 common indicators plus program specific.

Also, in the 2014–2020 period, roughly half of the common ERDF output indicators could be viewed as another type of indicator (inputs or results). Indeed, some indicators can be viewed as output indicators in one context, and result indicators in another (e.g., the indicator “Reduction of greenhouse gases”).

All along the developed research, Gramillano et al. (2018a, b), identified and emphasized the need for the harmonization and simplification of the used indicators. More comparable data based on the use of fewer indicators than those defined for the period 2014–2020 is desirable. Apart from the provided framework of common indicators, a match between the data gathered for the achievement indicators and the data from financial implementation is not fully feasible. Moreover, the data reported is often lacking (Henriques et al., 2022a, b). This is particularly true regarding the PT case, where the level of data details and the reporting practices frequently overlook common standards that hamper the programs evaluations and national OPs’ comparability.

2.3 Best Practices

Many authors agree on the importance of “learning from the past”. To make it feasible, some additional care must be placed on the chosen variables, the selected and monitoring indicators, and the implemented methodologies, since these are relevant to enhance that learning process.

Furthermore, to hasten the execution of the OPs, best practices from other MS should be examined, and administrative hurdles to obtaining funds should be reduced. MA should look for methods to boost project delivery by promoting the streamlining of payment request protocols and giving more guidance and support (Henriques et al., 2022b).

Frequently, assessment reports indicate that certain firms have revoked their subsidies, most likely owing to bank credit difficulties. In this respect, MA should be prepared to support businesses in securing additional financing options while also simplifying the conditions for attracting other institutional investors (Henriques et al., 2022b).

Gramillano et al. (2018a, b), presented a system of common indicators for the ERDF and CF interventions after 2020. The analysis performed covers the 11 TOs and is structured in two parts (I, for TOs 1, 3, 4, 5 and 6, and II for TOs 2, 7, 8, 9, 10 and 11). In this study, indicators (i.e., common output and direct result indicators) were evaluated according to their quality assessment, supported by key RACER principles (i.e., R (relevant), A (accepted), C (credible), E (easy to monitor), R (robust)) and other revised criteria (e.g., CREAM matrix assessment that sets out five criteria: Clear, Relevant, Economic and Available at a reasonable cost, adequate to provide information useful to assess performance and able to Monitor).

Moreover, time-bound indicators are also critical since they provide dates for measurement over time and monitoring is based on annual reporting or at least takes place at the end of the project. Besides, a debatable criterion should also be emphasized. Here, MA consultation collects information on the use of common output indicators to conduct benchmarking analyses or at least to use them in the future. This is to verify how much comparable information from 2014–2020 common outputs has been exploited since comparability is a major advantage of common indicators.

Globally, apart from the continuity of the best achievements from 2014–2020, the EC focused on: (i) the match of common outputs and results for interventions; (ii) simplification, harmonization, and data comparability; (iii) broader policy coverage; (iv) flexibility; (v) alignment with ESF; (vi) aggregation from project level; (vii) RACER—Financial Regulation criteria (Gramillano et al., 2018a).

Some clarification on the common ERDF and CF indicators was prompted by the EU (2018, Annex I and II), namely for the indicators to be selected in the programs, the data to be collected from projects via the monitoring systems, and the aggregated data reported by MS to the EC.

So, as suggested by Nigohosyan & Vutsova (2018), the EC should try in the next programming period to unify as much as possible the understanding of indicators under the different EU funds. An easy step would be the inclusion of all indicators into a single guidance document.

2.4 Methods

The literature evaluation identified desk research, monitoring data/data analysis, interviews, focus groups/facilitated workshops, surveys, and case studies as the major applied techniques to analyze ERDF TOs. Notwithstanding, the MS efforts to improve cohesion policy appraisal, only very few evaluations use more reliable methodologies, such as statistical methods or other mathematical techniques (Henriques et al, 2022a, b). Non-parametric approaches, like DEA, have turned into a noteworthy methodological alternative to the traditional approaches employed in similar contexts. The key benefit of utilizing this mathematical approach is the type of information that it can provide to MA on the inefficiency of the OPs when compared to their counterparts.

The benchmarks of inefficient OPs are also determined by DEA, and significant information about the best practices to follow to reach efficiency may be obtained. Nonparametric approaches, such as DEA, may readily manage many assessment criteria. Furthermore, DEA can help identify the key reasons that hamper efficiency, supplying policymakers with relevant knowledge on how to solve them. For example, Gómez-García et al. (2012) evaluated the efficiency of the implementation of ESIF allocated to TO1 in this context. Furthermore, Gouveia et al. (2021) used the Value-Based DEA method to evaluate the implementation of an ESIF aimed at enhancing the competitiveness of SMEs throughout multiple OPs (national and regional). In addition, Henriques et al. (2022b) evaluated 102 OPs from 22 EU nations dedicated to supporting an LCE in SMEs using the output-oriented variant of the slack-based measure (SBM) paired with cluster analysis. Lastly, Henriques et al. (2022a) evaluated 53 OPs from 19 MS committed to boosting R&I in SMEs using the non-oriented form of the network SBM approach in conjunction with cluster analysis.

Furthermore, DEA models are easily adaptable to evaluate different TOs if the basic rule proposed by Golany et al. (1989) is followed, namely, the number of DMUs (in this case, the OPs, EU funds, regions, countries, etc.) under evaluation should at least double the number of inputs and outputs (the indicators used in the evaluation). Though DEA offers undeniable benefits over other conventional methods (for example, microeconomic analyses that utilize control groups and case study analysis), there is currently a dearth of academic interest in its application in the context of ESIF efficiency appraisal.

This form of analysis is especially important if the programs are still in progress since it allows MA to predict the influence that prospective changes in output/input levels would have on the levels of efficiency attained by the OPs.

Unlike other approaches and methodologies that are specially applied for ex-post or ex-ante evaluation of cohesion policies, the DEA approach allows us to assess the efficiency of OPs’ deployment across the programmatic time horizon (thus allowing us to perform midterm/terminal assessments), so that the required initiatives can be implemented within the time necessary for producing the appropriate changes during the timeframe in headway.

Due to the lack of more robust approaches during midterm/terminal assessments, the adoption of nonparametric methodologies can be particularly beneficial and suitable, mostly because the existing metrics for appraising the Cohesion Policy can be employed with other methodologies and contextual indicators. This can be done by combining this sort of analysis with Stochastic Frontier Approach (SFA), for example, thus allowing us to understand if the inefficient results obtained are mainly related to managerial failures or to the contextual environment or statistical noise (Henriques & Viseu, 2022a, b).

3 The Portuguese Case: Main Findings

For the PT OPs, different funding dimensions on ERDF were exploited and characterized. Apart from data collection, data curation (i.e., identification of missing and null values, data consistency, etc.), data characterization (i.e., evaluation of the main statistics), and some data visualizations were used to analyze the PT public dataset.

The ESIF comprises the allocation of €461 billion and is distributed by six funds—See Fig. 3a. In 2014, a partnership agreement (PA) between Portugal and the European Commission, called Portugal 2020 (P2020), was signed. Based on that partnership an ESIF budget of €25,860 million was assigned to Portugal, which represents about 5% of the ESIF for all MS and about 2,561€ per capita. The P2020 programming and implementation were divided into four thematic domains (Vaquero et al., 2020): competitiveness and internationalization, social inclusion and employment, human capital and sustainability and efficiency in the use of resources, and seven regional operational programs (OPs). The distribution of the ESIF funds shows that approximately 79% of the total amount assigned to Portugal, goes to the three funds included in the Cohesion Policies, that is, ERDF (45.3%), ESF (24.7%), and CF (9%) (European Commission, 2022).

Fig. 3
An image depicts the less developed regions and more developed transitions. The outermost regions are arranged horizontally.

Source https://cohesiondata.ec.europa.eu/funds/erdf

ERDF “Total Eligible Costs Decided” and “Total spending” (Eligible Expenditure Declared) for a Less Developed Regions, and b More Developed, Transition, and outermost regions.

Full size image

In terms of eligibility for the ESIF (ERDF, CF, ESF, EAFRD, and EMFF), the seven Portuguese regions are divided into Less developed regions (GDP per capita <75% EU average): Norte, Centro, Alentejo, and the Azores, with a co-financing rate of 85%; Regions of transition (GDP per capita between 75 and 90%): Algarve, with a co-financing rate of 80%; and More developed regions (GDP per capita >90%): Lisboa and Madeira, with a co-financing rate of 50% to Lisboa and 85% to Madeira. Figure 3 depicts the execution of the EU funds in Portugal per type of region. As formerly stated, the programming and implementation defined by the PA for Portugal are based on four key thematic domains, also considering two transversal dimensions and seven OPs regarding the integrated intervention at the territorial level (Vaquero et al., 2020).

The defined TOs are concentrated on a series of priorities as R&I (TO1), ICT (TO2), business competitiveness (TO3), and the LCE (TO4) that fulfill regulatory requirements (74% in less developed regions; 69% in the region of transition of Algarve; 67% in the outermost regions of Madeira and Azores; 73% in the more developed region of Lisbon). According to the data available on the P2020 website, about 95% of the budget allocated to Portugal was already spent.

Regarding the evaluation of the PT programs, many difficulties were found regarding data collection. These were due namely to the usage of different reporting nomenclatures across the various PT regions and OPs (see e.g., the reports available at https://portugal2020.pt/portugal-2020/o-que-e-o-portugal-2020/). The autonomous reporting of the lists of Operations referring to P2020 can partially justify those differences (e.g., an ID of investment priority named as 1.1, 1.a, and 1a; the same about indicator CO01 or O.04.02.02.C for the same indicator in investment priority 4.b). Also, these generate some inconsistencies while comparing data at the regional level (i.e., Norte2020, Centro 2020, Alentejo2020, Lisboa2020, Algarve2020 and Madeira e Açores 2020). Another important issue regarding the reporting practices is related to the monitoring periods. The details on data are not uniform for the various programs or the monitored variables. Therefore, making it very hard or even impossible the development of comparative studies.

Concerning the assessment indicators, the EU Observations Report on the PT-funded programs noticed a very positive evolution in Norte OP regarding the construction and definition of the indicator framework. That evolution resulted from (i) the interaction between the regional entities responsible for the programming process and the evaluation team; (ii) the process of harmonizing the bases of indicators at the national level; (iii) the interaction between national entities and the EC (European Commission, 2014a, b).

Following the main funding priorities for the PT programs, our study focuses on three TOs: LCE; R&I; and ICT. To perform the analysis, data were collected from public EU sources on ERDF, the dataset was afterward analyzed, and some preliminary results are illustrated in Fig. 3a and b.

As it can be observed in Fig. 3a, for the “Less Developed Regions”, the programs showing higher values are “Competitiveness and Internationalization” (i.e., TO3) followed by the OPs of “Norte” and “Centro” regarding both, the “Total Eligible Costs Decided” and the “Total spending (Eligible Expenditure Declared)” (see the ESIF_2014-2020 categorization file available at https://cohesiondata.ec.europa.eu/funds/erdf).

In 2020, more developed regions, “Lisboa” and “Madeira”, registered quite different behaviors, with Madeira having a “Total spending” around half of the “Decided value”, while Lisboa registered a lower rate of execution. As “Outermost Regions”, Azores and Madeira (i.e., funds 17,740,636,416€ and 11,207,915,405€) both presented an even better execution rate in 2020—see Fig. 3b.

In less developed PT regions, i.e., Norte, Centro, Alentejo, and Azores; the highest “Decided values” were assigned to Multi Thematic Objectives, immediately followed ex aequo by “Educational and Vocational Training” and R&I, subsequently followed by the “Competitiveness of Small and Medium Enterprises”. The LCE and ICT are in the detached area of the graphic (i.e., split for representing less than 10%), with percentages around 1% or even lower—see Fig. 4.

Fig. 4
Two pie chart highlights less developed regions in total eligible costs. Information and communication technologies has least percentage.

Total eligible costs decided for less developed regions, 2014–2020, ERDF allocated to PT, evaluated for 2020 as a function of the TOs

Full size image

Concerning the regions of Transition i.e., Algarve; the highest percentage is allocated to “SMEs’ Competitiveness” with 43% of the funds. R&I, LCE and ICT have higher representativeness than that observed in less developed regions, but the decided value represents less than 10% of the total, see the detached area of the graph—Fig. 5.

Fig. 5
Two pie chart highlights transition regions in total eligible costs. Competitiveness of S M E s has the highest percentage.

Total eligible costs decided for transition regions, 2014–2020, ERDF allocated to PT, evaluated for 2020 as a function of the TOs

Full size image

In the more developed regions, the R&I TO appears again with a high representative percentage, followed by the SMEs’ competitiveness (TO3)—see Fig. 6. The TOs related to LCE, and ICT registered again a lower percentage of the “Total decided”.

Fig. 6
Two pie chart highlights more developed regions in total eligible costs. Research and innovation and multiple thematic objectives have the highest percentage.

Total eligible costs decided for more developed regions, 2014–2020, ERDF allocated to PT, evaluated for 2020 as a function of the TOs

Full size image

For the “Total eligible costs decided” it can be noticed that Multiple Thematic Objectives have the highest values, followed by R&I (TO1) and by Competitiveness of SMEs (TO3), whereas LCE (TO4) and ICT (TO2) obtain the lowest values—see Fig. 7a. Similar behavior is observed for the “Total spending”, only with the “Educational and Vocational Training” registering the highest values—see Fig. 7b.

Fig. 7
An image illustrates total costs and total spending are arranged horizontally marked as a, b respectively.

Source EC, ESIF, available at https://cohesiondata.ec.europa.eu/2014-2020/ESIF-2014-2020

a Total eligible costs decided and b total spending (Eligible Expenditure Declared) for 2014–2020.

Full size image

4 Conclusions

This study explores the present key challenges associated with the evaluation of ERDF committed to three TOs: R&I, LCE, and ICT. A study of the literature on ERDF deployment is done, with a particular emphasis on the PT case, as well as its assessment and reporting processes.

The answers to our research questions are given below.

RQ1: “What are the main areas of concern of the studies devoted to the assessment of ERDF?”

The literature review allowed us to conclude that more recent publications dedicated to the assessment of ERDF show a higher concern over the following themes: “sustainability”, “smart specialization”, “high education” and “regional disparities”. In the case of Portugal, most of the studies focus on “regional development”, also contemplating concerns on “competitiveness”, “higher education”, “innovation” or “rural development”.

RQ2: “What are the main challenges inherent to the selection of the variables/indicators in the assessment of ERDF?”

Overall, there is a lack of data availability that makes it difficult to assess all the OPs targeted to be funded. Besides, there is no full match between the financial data and the corresponding achievements per TO and dimensions of intervention.

We were also able to ascertain that the data available to perform the evaluations differ significantly with the type of TO under scrutiny. In this sense, there is scarce data availability on ICT (TO2) both at the regional and firm levels.

Regarding the PT case, the data analyses allowed us to conclude that there is a significant gap in some of the ERDF initiatives, particularly for the TO2 and TO4, i.e., ICT and LCE. Besides, it was possible to identify a scarcity of data publicly available for the PT OPs, in general, and various reporting conflicts. These issues hampered the possibility of performing deeper analysis, involving, for example, an in-depth comparison across regions’ performance or even enabling productivity analysis.

RQ3: “What are the best practices found in the assessment of ERDF?”

We were able to conclude that the best practices highlight the necessity for indicator harmonization and simplification. A possible way to do this would be the inclusion of all indicators in a single guidance document. It would be desirable to further enhance the quality and comprehensiveness of ICT performance data both at the regional and SMEs levels. Finally, it would be preferable to have more comparable statistics based on the usage of fewer metrics than those established from 2014 to 2020.

RQ4: “What are the main gaps found in the methodologies used to assess ESIF funds?”

Only a small number of assessments employ more consistent methodologies, such as statistical analyses or other mathematical tools. Non-parametric methods, such as DEA, have emerged as a significant quantitative option to the standard methodologies used in comparable circumstances. The primary advantage of employing this mathematical technique is the source of data that it can supply to MA on the inefficiency of the OPs when compared to their peers.

DEA also determines the benchmarks of inefficient OPs, and relevant data on the best procedures to follow to achieve efficiency may be gathered. Nonparametric techniques, such as DEA, may handle multiple evaluation criteria. Additionally, DEA can assist in identifying the primary causes of inefficiency, providing policymakers with helpful information on how to address them. Moreover, the DEA approach is easily adaptable to assess various TOs. This type of analysis is particularly relevant if the programs are already in progress because it enables MA to foresee the impact on the efficiency of future changes in output/input levels. Because more robust techniques are lacking during midterm/terminal evaluations, the use of this nonparametric methodology can be especially advantageous and appropriate, because the current metrics for evaluating the Cohesion Policy can be combined with other methods and contextual factors. This may be accomplished by integrating this type of study with the SFA, for example, enabling us to determine if the inefficient outcomes achieved are mostly due to managerial failings, the contextual environment, or statistical noise.

All in all, our findings emphasize the need for harmonization and simplification of the usage of indicators to evaluate the funded OPs. Besides, an important effort should be placed on the reporting of results to allow for better assessments and to avoid poor outcomes. Finally, we have identified a trade-off between the required detail of the achievements reported and the number of indicators used to support their description, i.e., comprehensiveness versus simplicity.