1 Introduction

The early school-leaving figures for Spain reveal that, although the rate has declined over last few years [from 30.9% in 2009 to about 17.3% in 2019 (MECD 2021)], it is still far from the 10% goal set by the European Horizon 2020 Strategy. By 2030, the target is to ensure that all girls and boys complete free, equitable and quality primary and secondary education leading to relevant and effective learning outcomes. In view of this situation of the Spanish economy, the reduction in school dropout rates and the improvement of labour force qualifications, are key objectives in the design of national and regional economic policies for the following years.

As in Spain, the governments of several developed countries have implemented different measures to address labour market problems in order to reduce the mismatch between labour supply and demand. These measures can be classed as active labour market policies (ALMPs). At the international level, these policies consist of several programmes such as job search assistance (JSA), public employment, educational and training for unemployed people or incentive schemes in private and public sector. Various meta-analyses and comprehensive surveys of the effectiveness of ALMPs (Betcherman et al. 2004, Card et al. 2010, 2018, Kluve 2010) agree on highlighting the positive impact of JSA programmes on employment probability. The same holds for training programs, although they are more likely to be effective in the medium and long term, while public works have on average negligible or even detrimental effects on employability in all horizons. By contrast, Vooren et al. (2019), analysing 55 experimental studies published between 1990 and 2015, found out that incentives schemes in the private sector are the most effective labour market policy, followed by training and re-training programmes. Additionally, public employments appear to have negative effects and JSA interventions seem to be ineffective.

As regards the effects of ALMPs in Spain, Malo and Cueto (2016) state that almost all the labour market programmes assessed impact positively on employment, however, the results found are extremely heterogeneous. Focusing on the labour market effects of training and JSA measures, Blázquez et al. (2019) confirm their effectiveness for the Spanish case. Both type of interventions have a positive effect on employment, being more intense for training programmes beneficiaries and among long-term unemployed.

Shifting to policies aimed at improving the employability of young people, very few studies have gathered evidence on the effectiveness of ALMPs targeted at youths (Caliendo and Schmidl 2016; Hardoy et al. 2018; Kluve et al. 2019). In general, no consensus exists on whether there is a certain type of programme that outperforms others in reducing young unemployment and it seems that youth unemployment programmes are less effective in increasing employment levels than policies targeted at general population of unemployed (Card et al. 2010; Kluve 2010).

One of the above-mentioned training programmes was the so-called Programa 18–25. The Programa 18–25 was launched by the Regional Government of Extremadura (an Autonomous Community located in southwest Spain in the border with Portugal) in November 2012 and aimed to reduce the number of unemployed and uneducated people in this Autonomous Community. The target population was unemployed people aged between 18 and 25 years who had not completed compulsory secondary education. Through a monetary incentive of 1000 euros, this conditional cash transfer (CCT) programme was meant to motivate these people to return to the formal education system and earn the lower secondary school diploma in the shortest period. Therefore, Programa 18–25 combines the use of CCT for achieving education results as an intermediate outcome for the final goal of improving young adults labour market outcomes.

On one hand CCT are widely applied in developing countries for reducing poverty and inequality but also for boosting education at primary and secondary schools. Different previous meta-analysis studies (Baird et al. 2014) conclude that on average the effectiveness of CCT on improving test scores is, at best, small. Nevertheless, imposing achievement conditions, in addition to school enrolment or attendance, is associated with larger primary attendance effect sizes (Garcia and Saavedra 2017). CCT programmes have been also applied in combination with ALMPs (López-Mourelo and Escudero 2017; Baird et al. 2018). Aeberhardt et al. (2020) analyse, through a randomized controlled trial, the effects of a CCT programme for young, unskilled jobseekers in France receiving a monthly cash transfer conditional on their participation in the French national career guidance program. Their results show that although the CCT programme fosters programme participation, it also results in no effects on beneficiaries to their higher employability, even observing a lower rate of participation on the labour market in the first six months of the program.

On the other hand, there is also the vast literature focused on estimating the causal impact of public policies intended to improve educational outcomes (for the literature review in economics of education see for example Webbink 2005; Schlotter et al. 2011 and Cordero et al. 2018). Regarding empirical evaluations in Spain, we find some previous works analysing the causal relationship between different specific education programmes or large public policies and educational outcomes. For example, Anghel et al. (2015) find that the publication of the results of standardized external test in the region of Madrid had a significant positive effect in reading by the end of secondary education. Likewise, Anghel et al. (2016) evaluated the introduction of bilingual education in a group of public primary schools in the region of Madrid concluding that the programme had a strong negative effect on general knowledge subjects taught in English but not in Mathematics and Reading taught in Spanish. In the same vein, Feliciano et al. (2021) examine the impact of Escuela 2.0, a one laptop per child programme developed in Spain between the academic years 2009–10 and 2011–12, on PISA test scores concluding that on average there was a 2.9% performance fall across all regions that applied this programme.

In relation with the evaluation of general public policies in education conducted by law changes, Felfe et al. (2015) conclude that the extension of free universal public preschool education from four to three-year-old students during the 90s in Spain had a sizeable influence on children’s reading skills by the end of secondary education. Furthermore, Salinas and Solé-Ollé (2018) conclude that the early decentralization of educational competencies for some regions of Spain during the 80s reduced the dropout rate in secondary education by around 13% in the treated regions with respect to the controls.

The aim of this paper is to evaluate whether participation in the above-mentioned Programa 18–25 increased the probability of earning the lower secondary education diploma or, at least, the likelihood of successfully completing the academic year. To do this, we use administrative data provided by Regional Department of Education and Culture in Extremadura about the students enrolled in adult secondary education (ASE) during the 2013/14 academic year. This paper focuses on males only, as Programa 18–25 made a distinction between males and females. Although the potential beneficiaries were unemployed people aged between 18 and 25 years who did not hold a lower secondary education diploma, the regional Government decided to remove the age limit for long-term unemployed women. It means that women over 25 years old could apply for the programme if they were unemployed for 12 months or more. As a result, it is not possible to evaluate the impact on the total population. We consider that this study is relevant on several grounds. To the best of our knowledge, this is the first time that this kind of programme has been evaluated in Spain (at regional or national level). Second, we believe that the results of this study are likely to be of interest to other regional or national governments worldwide currently considering similar labour market policies based on conditional cash transfers to fighting against youth unemployment.

The paper is structured as follows. Section 2 gives an in-depth description of the Programa 18–25 features and presents the research design and our identification strategy. Section 3 explains the dataset used and the selected variables included in the empirical analysis. Section 4 report the results, and the article winds up with the main conclusions.

2 Intervention, research design and identification strategy

2.1 Description of Programa 18–25

According to the Employment Observatory, around 41% of the people registered with the regional Public Employment Service (Servicio Extremeño Público de Empleo, hereafter SEXPE) as unemployed in ExtremaduraFootnote 1 had not completed their secondary education in 2012, figures that have remained stable in more recent years. In this context, the Regional Government of Extremadura signed an agreement with SEXPE with the aim of developing programmes specifically targeting unemployed people. Its goals were to provide education and training for people with employability issues, particularly young people aged between 16 and 25 years, long-term unemployed women, disabled people, or people at risk of social exclusion.

The Programa 18–25 was part of this above-mentioned agreement invoking Extremadura’s Education Law 4/2011, of March 7th.Footnote 2 This law states that the Regional Government has the power to promote policies aimed at ensuring the right to lifelong learning for people who dropped out of the education system, encouraging them to return to education in order to improve their personal and career prospects. Programa 18–25 was launched in November 2012 and ran for the following two academic years. The programme was cancelled when the political party in charge of the regional government changed after the regional elections (May 2015).

The aim of this programme was to reduce the number of unemployed people who had not completed compulsory secondary education in Extremadura by offering a financial incentive with a view to improving their job opportunities. The potential beneficiaries were unemployed people aged between 18 and 25 years who did not hold a lower secondary education diploma. Additionally, no age limit was applicable to long-term unemployed women. This means that the Extremadura Government decided to extent the Programa 18–25 to all long-term unemployed women regardless their age. Thus, long-term unemployed women could benefit from the programme even if they were over 25 years old, unlike men who had to meet both the age and unemployment requirements to be eligible. For this reason, the conditional cash transfer effects are analysed in this work just for men.

Both the educational and administrative programme management conformed to the provisions of ASE in Extremadura region.Footnote 3 The Programa 18–15, like ASE, was organized in modules instead of subjects, distributed in two different levels and three different areas of knowledge (communication, scientific-technological and social). Each level was composed of two modules per area (one per semester) and took place in a different academic year, making a total of 12 modules. Each academic year consisted of a total of 35 teaching weeks, with 18 teaching hours per week. Thus, in order to benefit from the programme, students applying for admission to adult education were required to state their interest in joining Programa 18–25 and enrol for up to a maximum of six modules per academic year, which was the maximum number of modules allowed per academic year in ASE.

The monetary incentive was conditional upon three conditions. First, programme beneficiaries were obliged to attend school regularly. Second, beneficiaries should pass the ordinary exams (like any other ASE student) and, finally, pass programme-specific tests set for each module in which they enrolled. Only students who applied for and met the above-mentioned requirements received the cash transfer, composed of two separate 500-euro payments due at the end of each semester.

In Spain, the only previous programme with similar features was the Second Chance (Segunda Oportunidad) scholarship offered by the Regional Government of Andalusia, which entered into force in the 2011/12 academic year. This scholarship targets 18- to 24-year-olds who enrol for compulsory secondary education, vocational training programmes or the Spanish baccalaureate. It consists of a monthly cash transfer [around 75% of Spanish Public Income Index Multiplier (IPREMFootnote 4)] conditional upon students regularly attending school, doing homework and passing exams, where the maximum entitlement per academic year is 4000 euros.

2.2 Research design

We assess the impact of Programa 18–25 by comparing the academic achievement of programme beneficiaries throughout the 2013/14 academic year with students who enrolled in ASE in the same academic year, but they did not benefit from the programme because they did not meet the age or employment status requirements. For this purpose, the Government of Extremadura’s Department of Education provided us with administrative data for conducting the evaluation.Footnote 5

A preliminary analysis of the evaluated students’ behaviour revealed key issues for determining our identification strategy. First, some of the programme beneficiaries qualified for the lower secondary education diploma but did not receive the monetary incentive because they did not attend the programme-specific tests or did not apply for the payment at the end of the semester. Second, we found that some individuals received the 500 euro per semester payment (which entailed meeting the attendance requirements and passing regular exams and programme-specific tests), even though they did not qualify for the lower secondary education diploma. This was because despite they pass the six modules, they still had more than six modules to pass to earn their diploma. Third, as Programa 18–25 was not a mandatory public policy, some eligible individuals did not self-select into the programme showing a ‘never-takers’ behaviour. Finally, some individuals managed to enrol in the programme even though they did not meet the age requirement.Footnote 6 Figure 1 shows the composition of the treatment and control group according to the cut-off age and the different motivations of males enrolled in the ASE and the programme.

Fig. 1
figure 1

Source Authors’ own elaboration

Treatment and control groups.

Full size image

The treatment group includes males aged 25 years and under enrolled in ASE.Footnote 7 This group includes people who did and did not enrol for the programme. A distinction can be made between programme beneficiaries who are intrinsically motivated, i.e. they would enrol in ASE regardless of the policy, and individuals who are extrinsically motivated, i.e. they were induced by Programa 18–25 to return to education. The control group includes people who enrolled in adult education but were not eligible for the programme because they were aged over 25 years and for this reason all of them were intrinsically motivated. In order to assess the programme properly, we should compare treated and non-treated individuals who are intrinsically motivated. However, it is not possible to make a distinction between these two types of motivation among individuals within the treatment group. However, in Sect. 4.1 we explore whether or not the programme led to a ‘pull effect’ inducing more males aged 25 or under to enrol in adult education.

2.3 Identification strategy

For assessing the impact of this policy on the variables of interest we decided to use a regression discontinuity design (RDD). Thanks to this approach, we can compare individuals with similar characteristics within a narrow band above and below the cut-off point (25 years), where only students who do not exceed this threshold are eligible for the programme.

Thistlewaite and Campbell (1960) introduced RDD into the evaluation literature in an attempt to study the effect of a scholarship granted exclusively to students who achieved specific test scores above a threshold.Footnote 8 Since then, this method has been applied to evaluate educational issues as diverse as the effect of class size on student performance (Angrist and Lavy 1999), the impact of university financial aid awards on college enrolment (Van der Klaauw 2002), the influence of grade retention on educational attainment (Jacob and Lefgren 2004), the impact of the month of birth on cognitive and non-cognitive skills (Crawford et al. 2014) or the effect of college remedial policies (Duchini 2017).

RDD is applicable for programmes or policies that have a continuous eligibility index (running variable), \({X}_{i}\), with a strictly defined cut-off point, \(\overline{x }\), to determine who is and who is not eligible. Then, with respect to the analysed policy, \({D}_{i}\) denotes the treatment as follows:

$$D_{i} \left\{ {\begin{array}{*{20}c} 1 & {{\text{if}}\,\,{\text{Age}}_{i} \le 25 \to {\text{Treated}}} \\ 0 & {{\text{if}}\,\,{\text{Age}}_{i} > 25 \to {\text{Non - treated}}} \\ \end{array} } \right.$$

The main advantage of the RDD approach is that our comparison of the results for units in a close neighbourhood around (above and below) the eligibility cut-off will be as good as if we had conducted a randomized trial (Lee and Lemieux 2010). Therefore, differences in outcomes can be entirely attributed to the intervention itself (Gertler et al. 2016).

There are two main general settings within RDD. The sharp regression discontinuity design is applied when the running variable precisely defines the treatment and control groups. In a fuzzy regression discontinuity design, on the other hand, the running variable does not perfectly determine the treatment group but creates a discontinuity in the probability of receiving the treatment (Abadie and Cattaneo 2018; Schlotter et al. 2011). The RDD scenario is fuzzy when the eligibility rules are not strictly adhered to because treatment assignment is governed by some unobserved variables (Hahn et al. 2001).

In this case, as Fig. 2 shows, the percentage of participants in Programa 18–25 is less than one to the left of the cut-off point and greater than zero to the right of the cut-off point.Footnote 9 This means that the running variable (Age) did not perfectly match the treatment, and, therefore, a fuzzy RDD arises here.

Fig. 2
figure 2

Source Authors’ own elaboration

Percentage of participants in Programa 18–25 by age.

Full size image

Fuzzy RDD can be analysed in an instrumental variables framework by defining a simple dummy variable, denoted by \({I}_{i}\), to indicate whether the running variable \({X}_{i}\) is below or above the eligibility cut-off. \({I}_{i}\) is used as an instrument for treatment variable \({D}_{i}\) in the estimation of the outcome equation (Angrist and Pischke 2008).

There are several concerns that should be taken into account when applying RDD. First, the running variable should not be manipulated in order to ensure treatment assignment. In this study, the running variable (students’ age) meets this requirement (Imbens and Wooldridge 2009). Second, the specification may be sensitive to the functional form used to model the relationship between the assignment variable and the outcome variable (Gertler et al. 2016). Consequently, alternative specifications must be tested including higher-order polynomials on age and interactions. Third, RDD produces local average treatment effects that may not necessarily be generalized to units far away from the cut-off point (Khandker et al. 2010); then, the potential effects should not be extrapolated to other subsets of the population. Finally, it is not always possible to find enough available observations close enough to the threshold. In order to solve the small sample size problem, the interval around the cut-off point can be increased. However, eligible and ineligible units will become more different as we move further from the eligibility threshold and this difference might bias the comparison (Schlotter et al. 2011). The inclusion of covariates may eliminate some bias resulting from the larger bandwidths (Imbens and Lemieux 2008). In this research, the bandwidth was extended due to the limited sample size. Thus, a set of control variables were included in the empirical model aimed at avoiding bias. Therefore, the fuzzy RDD is estimated by means of the following equations:

First stage or treatment equation:

$$D_{i} = \gamma_{0} + \gamma_{1} I_{i} + \gamma_{2} {\text{Age}}_{i} + \gamma_{3} Z_{i} + \varepsilon_{1} ,$$
(1)

Second stage or outcome equation:

$$Y_{i} = \beta_{0} + \beta_{1} \widehat{D}_{i} + \beta_{2} {\text{Age}}_{i} + \beta_{3} Z_{i} + \varepsilon_{2} ,$$
(2)

where \({Y}_{i}\) stands for the measures of interest chosen to evaluate the programme impact of individual i; \({D}_{i}\) indicates the real treatment; \({\widehat{D}}_{i}\) denotes the probability of receiving the treatment and is estimated using a generated instrumentFootnote 10\({I}_{i}\); \({\mathrm{Age}}_{i}\) is the running variable, and \({Z}_{i}\) denotes the set of covariates.

3 Data and variables

The information used in this paper was gathered from administrative records provided by the regional government of all students enrolled in ASE in Extremadura during the 2013/14 academic year. This initial sample is composed of 5485 observations, males and females, aged between 18 and 60 years old. These data include information about students’ gender, their age as of December 31st, 2013, whether they were Programa 18–25 beneficiaries, the number of modules for which they enrolled, the school at which they enrolled and their results per semester.

Prior to the empirical analysis, the administrative sample was constrained according to the following criteria. First, as mentioned earlier, the evaluation focused on males. Second, due to the wide disparity in the Age variable, the sample was confined to individuals aged between 20 and 31 years old, that is, six years above and below the cut-off point set by the programme (25 years). Besides, students were allowed to enrol for one to six modules. Most participants enrolled for four or more modules, and the effort that it took to pass was clearly proportional to the number of modules for which they enrolled. In order to reduce the heterogeneity among the data caused by these concerns, the study focused on individuals enrolled in six modules (the maximum number of modules allowed per academic year in both ASE and the Programa 18–25). Therefore, the final sample was composed of 1049 observations.

Following recommendations in the literature, we looked for a large enough bandwidth in order to get a large enough sample size for a reliable empirical analysis, while, at the same time, narrow enough to assure the individuals included are alike. To this end, we replicated the analysis using different bandwidth sizes to test for robustness. The largest bandwidth is composed of 1049 males aged between 20 and 31 years, six years to the left and right of the threshold. The major problem with this bandwidth is that, due to its extent, the sample is likely to include males who may not be comparable in terms of motivational, personal and socio-economic characteristics. By narrowing the bandwidth down to four years above and below the cut-off point, we got a second group composed of 686 males aged from 22 to 29 years. Finally, the narrowest bandwidth consists of 336 24- to 27-year-old males, i.e. two years above and below the eligibility cut-off.

The impact of Programa 18–25 is measured providing two different outcomes. The first dependent variable, Diploma, is a binary variable whose value is 1 if the student earned a diploma for lower secondary education at the end of the 2013/14 academic year and 0 otherwise. The second dependent variable, Success, is a binary variable whose value is 1 if the student passed every module for which he or she enrolled in that academic year and 0 otherwise. This dependent variable should capture individuals who received the per semester payment but did not earn the diploma because they still had to pass some modules in the following academic years.

The running variable for the Programa 18–25 is the students’ age as of December 31st, 2013 (Age). The real treatment is a dummy variable that is equal to 1 if the student belongs to the treated group and 0 otherwise. The assignment of individuals to the treatment or control group should be equal regardless of which of the previous variables is employed. However, as mentioned in Sect. 2.2, we detected the presence of always-takers and never-takers. This led us to build the variable CutOffAge, whose value is 1 if the individual is, at December 31st, 2013, aged 25 or under and 0 otherwise, and that will be used to estimate the generated instrument, Ii.

A preliminary statistical analysis showed that the above- and below-threshold students differed statistically and significantly for some variables. For this reason, we included a set of covariates in the empirical model to taking into account this potential difference. As is mentioned in Sect. 2, the programme were organized in modules, instead of subjects, distributed in three different areas of knowledge (communication area, scientific-technological area and social area). Three dummy variables are defined to account for the area(s) of knowledge in which student was enrolled taking value 1 if the individual was enrolled in some module(s) belong to a specific area and 0 otherwise: Communication, Sci-tech and Social. Student environment is accounted for by two different variables: Rural, defined as 1 if the student lived in a rural environment and 0 otherwise; and the unemployment rate (U-rate) of the municipality in which the student was attending ASE. Finally, a discrete variable (Enrolled modules) accounts for the number of modules enrolled per student in the academic year under review. This last variable is only included in the robustness test models.

Table 1 reports the main descriptive statistics by bandwidth. Students belonging to the eligible group and those belonging to the non-eligible group differ mainly in their environment (rural area and municipalities’ unemployment rates) within the large bandwidth, although this difference vanishes when the bandwidth is narrowed. Finally, by construction, there are also differences in their mean age.

Table 1 Descriptive statistics: group means and group differences
Full size table

4 Results

4.1 ‘Pull effect’

With regard to the existence of a potential ‘pull effect’, Fig. 3 plots the number of students enrolled in ASE by age in the range 20–31 years. In this manner, we can explore whether the treatment boosted enrolment, i.e. whether the number of programme beneficiaries enrolled in ASE was considerably greater than regular ASE students.Footnote 11 A visual inspection does not appear to reveal any evidence of there being a jump in enrolment around the cut-off point; all that is observed is a negative linear relationship between students’ age and enrolment rates. This finding is consistent with human capital theory that states that younger people are more likely to invest in education due to their lower opportunity cost and their longer working life with regard to return on investment (Becker 1962).

Fig. 3
figure 3

Source Authors’ own elaboration

Number of people enrolled in adult education by age.

Full size image

Since the data for comparing the enrolment rates of males in ASE in Extremadura with the same rates for previous academic years are not available, we used the figures of the national trend in enrolment as a proxy of the enrolment rate in the region. Table 2 reports the figures for male enrolment in ASE in Spain over the four academic years analysed. The negative relationship between students’ age and enrolment rates observed in the Extremadura region is similar to the nationwide trend in the reviewed academic years. Besides, note that, while Programa 18–25 was running in Extremadura, national ASE enrolment for males increased considerably too. Hence, if regional enrolment rates in ASE increased in the years under review, it might not due to the introduction of the programme but to the Spanish and global economic crisis, which prompted people of all ages to return to education. Similarly, several OECD countries report crisis-related increases in enrolments in compulsory secondary education and vocational training (Damme and Karkkainen 2011).

Table 2 Figures for male enrolment in ASE by age groups in Spain
Full size table

All in all, we cannot conclude that the introduction of the Programa 18–25 induced a ‘pull effect’. Therefore, most students in the treatment group had the motivation to return to education regardless of the programme. Hence, we conclude that individuals within the treatment group are comparable with individuals within the control group.

4.2 Impact of Programa 18–25

This section discusses estimates of the impact of Programa 18–25 on two dependent variables: Diploma and Success. As a preliminary approximation of a possible effect of the programme on these two outcomes, we plot the average value of each dependent variable as a function of age in Figs. 4 and 5, respectively. Figures 4 and 5 show that there is no clear evidence of a significant jump in the average outcomes near the cut-off point. Although this graphical analysis suggests that the programme did not have any impact, we used a fuzzy RDD to check this preliminary result.

Fig. 4
figure 4

Source Authors’ own elaboration

Average of Diploma variable against age. Note The circles are the average outcomes for student with a given age. The fitted lines are predicted probabilities from a linear probability model, estimated separately on either side of the threshold. Alternative scatter-diagrams, applying nonparametric local regression functions, were plotted, resulting in the same findings.

Full size image
Fig. 5
figure 5

Source Authors’ own elaboration

Average of Success variable against age. Note The circles are the average outcomes for student with a given age. The fitted lines are predicted probabilities from a linear probability model, estimated separately on either side of the threshold. Alternative scatter-diagrams, applying nonparametric local regression functions, were plotted, resulting in the same findings.

Full size image

For this purpose, we estimated four models for each dependent variable (Diploma and Success) and per age bandwidth (20–31 years; 22–29 years; 24–27 years). Model 1 is the straight fuzzy RDD model estimation with no covariates. Model 2 accounts for control variables defined in Sect. 3 to avoid any potential bias resulting from using a wide range of data. Finally, Models 3 and 4 replicate the previous models by incorporating age-squared in order to capture any nonlinearity on age.

It is worth noting here that in our empirical model the two dependent variables, Diploma and Success, the endogenous explanatory variable, \({D}_{i}\) and the instrument represented by the CutOffAgei variable are binary variables. In order to avoid the weak identification issue caused by the first-stage linear projection onto linear instruments in the context of a binary endogenous explanatory variable we resort to Wooldridge (2010, p. 939) and Xu (2021). The procedure proposed by these authors is, in short, a two-steps estimation approach. In the first step, we estimated a binary response model by maximum likelihood (probit) of the endogenous explanatory variable (\({D}_{i}\)) on CutOffAgei, Agei and the set of covariates (\({Z}_{i}\)). In the second step, the fitted probabilities \({\widehat{D}}_{i}\) of the previous step are used as the new generated instrument \({I}_{i}\) for the IV estimation of Eqs. (1) and (2) through a linear two-stage least square model.Footnote 12

Tables 3, 4 and 5 report the impact of Programa 18–25 on the dependant variables selected from the widest to the narrowest age bandwidth, respectively.Footnote 13

Table 3 Impact of Programa 18–25 on dependant variables. Bandwidth from 20 to 31 years
Full size table
Table 4 Impact of Programa 18–25 on dependant variables. Bandwidth from 22 to 29 years
Full size table
Table 5 Impact of Programa 18–25 on dependant variables. Bandwidth from 24 to 27 years
Full size table

Regarding estimates of the probability of earning a lower secondary education diploma at the end of the academic year (Diploma), the different models estimated result in similar findings. First, the treatment is not statistically significant, even accounting for covariates in order to control for potential bias. Second, the results are robust to the inclusion of nonlinear terms and for the age windows selected.

Turning to the second dependent variable, Success, neither of the models report a statistically significant effect of the programme, irrespective of the bandwidths. These results suggest that Programa 18–25 beneficiaries are not more likely to pass every enrolled module in the 2013/14 academic year.

Despite the lack of statistical significance of the treatment, it should be mentioned how the sign of the coefficient associated with the treatment changes when analysing different age windows. More specifically, for the widest (20–31 years) bandwidth the effect of the treatment on the probability of obtaining the diploma and success is mostly negative, however, when the bandwidth is reduced, the coefficient sign becomes positive for Success variable in the intermediate (22–29 years) window and in both outcomes for the narrowest (24–27) interval. This could be explained by the influence of students’ motivation (which cannot be observed). Thus, in the widest age window, the older students’ intrinsic motivation seems to cancel out the motivation that the monetary incentive may generate in younger students. However, the role played by older students’ motivation apparently disappears when looking at individuals closer to the threshold.

In sum, the programme does not have any statistically significant impact when a fuzzy RDD is applied, which is consistent with the visual evidence (Figs. 4 and 5). The findings reveal that both the probabilities of earning the secondary education diploma and of passing all enrolled modules during the 2013/14 academic year were not statistically and significantly different for the treatment and control group.Footnote 14 Hence, having benefited from Programa 18–25 did not have, at least for the evaluated males, a clear impact on expected outcomes.

4.3 Robustness check

In order to check whether it was low statistical power caused by the small sample size that was behind the lack of programme impact, we replicated the estimates using an extended sample that takes into account individuals enrolled for between 4 and 6 modules. Again, three different age bandwidths are employed. The main descriptive statistics and sample sizes per age window are reported in Table 6. As for the sample of individuals enrolled in six modules, eligible and non-eligible students differ in terms of the rurality, although again differences disappear as the bandwidth narrows, and their age. In addition to that, some statistical differences are found in the areas of knowledge students enrolled. However, the average number of modules studied is almost identical in both groups in the three bandwidths.

Table 6 From 4 to 6 modules sample: descriptive statistics
Full size table

The second-stage results for this robustness test are presented in Tables 7, 8 and 9.Footnote 15 Once again, the policy has no effect on any of the dependent variables. This is consistent with the findings of our main analysis using only the information about students enrolled in six modules. The results suggest that, in general, students with more enrolled modules had less probabilities of obtaining the Diploma because in these cases the workload is higher but, as we mention above, the intervention seems to have no effect on the outcome. Likewise, for the outcome Success, regardless the bandwidth and other covariates, the regional programme seems to bring about no result on the outcome.Footnote 16

Table 7 Robustness check for bandwidth from 20 to 31 years
Full size table
Table 8 Robustness check for bandwidth from 22 to 29 years
Full size table
Table 9 Robustness check for bandwidth from 24 to 27 years
Full size table

Although the findings were in line to those obtained previously and it appears that the Programa 18–25 effect does not depend on the number of observations employed in the evaluation, we must acknowledge that the limited sample size remains a weakness for our analysis.

5 Conclusions

The bleak early school-leaving and youth unemployment figures for Spain point to the need to develop policies specifically aimed at raising the average educational level of the Spanish population in order to improve job opportunities and economic growth prospects for society as a whole. The aim of this paper was to evaluate the impact of Programa 18–25 on males in the 2013/14 academic year. First, we explored whether this policy caused any ‘pull effect’, i.e. led to a greater increase in the enrolment rates of under 25-year-old males than other males who were not eligible for the policy. Second, we set out to assess the effect of the programme on the probabilities of earning a lower secondary education diploma (first dependent variable: Diploma) and of passing every enrolled module (second dependent variable: Success). We carried out a RDD analysis using administrative data provided by the Regional Department of Education and Culture about all students enrolled in ASE during the academic year under review.

The findings show that, at least for the evaluated males, the likelihood of earning the diploma or passing every module for which they enrolled did not depend on being a Programa 18–25 beneficiary. Furthermore, these results are robust to alternative regression specifications and a variety of bandwidths. To elaborate briefly, our results suggest that those that decided to follow the ASE in the 2013/14 and earned the Diploma or Success all enrolled modules received in addition 1000 euros, but the money did not fulfil with the expected target. A potential reason for these findings might be that the monetary incentive was not enough to compensate for the strict programme requirements (regular attendance, ordinary exams, and programme-specific tests). Thus, the motivation of males aged 25 years or under who returned to education did not differ from that of those aged over 25 years old, showing no statistically different results in terms of earning a diploma o passing the modules.

It should be noted that only short-term outcomes (earning a secondary school diploma or passing every module enrolled in the academic year) are observed in our analysis. Although these findings provide some insights for the analysis of the programme impact, more research would be needed to further explore the potential effects on medium to long-term outcomes (such as the time it takes to get the diploma or to find a job).

After Programa 18–25 was implemented, some similar educational policies were launched. First, the Regional Government of Castile-La Mancha introduced a programme identical to Extremadura’s in the 2013/14 academic year. It was in effect for the following two academic years. Second, the programme called Graduate2 (Gradua2) was launched by the Regional Government of Castile-Leon in January 2015. This programme targeted people aged over 18 years that do not hold a compulsory secondary education diploma. It consisted of a free six-month course for training students to pass the secondary education diploma exam. Beneficiaries were eligible for a scholarship covering transport, accommodation and/or meals depending on average family income. Additionally, students who earned the diploma and enrolled in vocational training or the Spanish baccalaureate in following academic year received a monetary incentive of 500 euros. A grant called Second Chance Programme (Programa de 2ª Oportunidad), targeting unemployed and uneducated people aged from 16 to 30 years, has been in effect in the region of Madrid since the 2016/2017 academic year. It consists of a monthly cash transfer, amounting to up to 5000 euros per year. To qualify for payments, beneficiaries have to be enrolled in vocational training or attending training courses to qualify for admission to vocational training or to earn the secondary education diploma. Although this kind of programme has proliferated in recent years throughout Spain, to the best of our knowledge, none of them have been evaluated.

Because of that, we should stress here, first, the need to carry out more formal impact evaluations capable of distinguishing causation from accidental associations or correlations. It is worth noting that even for small programs, in terms of public spending, like this, rigorous evaluation provides a valuable feedback tool for making rational decisions about whether or not to continue the public programmes in operation with the same incentives scheme. In this respect, before they are implemented, future public programmes should be designed to enhance the evaluation results. Second, the null effect of Programa 18–25: the financial incentive offered failed to increase either the probability of successfully completing the academic year or the likelihood of earning the lower secondary education diploma. Finally, it is worth considering that results for Extremadura should act as a caveat emptor for other Spanish regions and other countries whose governments are currently running or considering this type of policies, subject to the caveat that the results might not be the same in other contexts.