Teaching Efficiency of Italian Universities: A Conditional Frontier Analysis

Abstract

The aim of this chapter is to provide a comparative analysis of the performance of teaching at Italian university by evaluating the efficiency of heterogeneous faculty courses at the national level. For this purpose, we use advanced and robust nonparametric tools recently developed in nonparametric efficiency frontier literature. This performance assessment does not rely on hypotheses about the relationship between inputs and outputs and allows us to account for the heterogeneity of the analyzed courses. The overall analysis carried out at the national level for Italy extends the traditional and limited one-dimensional indicators available through SUA-CdS data and ad hoc surveys on graduates conducted by ANVUR (National Agency for the Evaluation of Universities and Research Institutes) and MIUR (Ministry of Education, University and Research). The estimated efficiency scores are used to analyze current trends and changes in the teaching activities of Italian universities.

1 Introduction

Economic theory models public entities’ behavior using paradigms typical of the economic sciences. However, empirical econometric or operational research is needed to obtain comparative evaluations and effective policy implications (Daraio, 2018; Johnes, 2006, 2015; Johnson & Ruggiero, 2014; Ruggiero, 1996). In the case of universities, the quantity, quality, and mix of services produced are largely due to autonomous decisions, influenced by preferences of the different categories of stakeholders (Klumpp, 2015; Nigsch & Schenker-Wicki, 2015). Universities in the current period of crisis are at a crossroads. Evaluation, rankings, and governance are at the core of recent policy agendas, because it is crucial to invest in science and education to fully implement revitalizing strategy in terms of innovation and growth (Daraio et al., 2019).

Universities can be seen as “loosely coupled systems” characterized by autonomous decision-making processes with respect to the quality, quantity, and mix of products and services provided (Bonaccorsi & Daraio, 2007a, 2007b, 2007c). Italian universities, as all other universities over the world, are involved in a series of institutional activities, namely teaching, research, and the diffusion of knowledge in society (called “Third Mission”). An important limitation of available Italian university rankings relies on an inadequate consideration of the role of teaching, one of the fundamental pillars of academic activity, causing information bias regarding the performance of individual universities and the academic system as a whole.

This chapter investigates the performance of university teaching by evaluating the efficiency of different faculty courses. The aim is to propose a new measure of university efficiency, where the unit of evaluation is not the university but the course of study. The aim is to highlight the diversity and autonomy of individual universities in relation to teaching organization, providing an empirical measure, based on comparable available data, of the results obtained in terms of the efficiency of the various courses of study.

A preliminary analysis was carried out on the University of Salento data, and the results are reported in Mastromarco et al. (2019). We extend the analysis at the national level, including all courses of all Italian universities. For this purpose, we use advanced and robust nonparametric tools recently developed in nonparametric efficiency frontier literature (Bădin et al., 2010, 2012, 2019; Daraio et al., 2018; Daraio & Simar, 2005, 2007a, 2007b).

The novelty in the performance assessment of university teaching activities consists in an extension of traditional and limited indicators, extracting information available through the SUA-CdS data sheet and ad hoc surveys on graduates conducted by ANVUR (National Agency for the Evaluation of Universities and Research Institutes) and MIUR (Ministry of Education, University and Research), in order to improve the quality of teaching monitoring and promote its dissemination through the different universities.

The chapter is organized as follows. In the next section, we summarize the relevant literature. Section 3 outlines the methodology. Section 4 describes the data and the empirical strategy. Section 5 illustrates the results, and Sect. 6 concludes the chapter. Finally, the Appendix reports additional details on the methodology.

2 Brief Literature Review

Although the role of universities in the knowledge society is increasingly relevant, there is a lack of systematic quantitative evidence at the micro level and, in particular, at the level of individual faculty courses. Bonaccorsi and Daraio (2007a, 2007b, 2007c) examined original data from universities in six European countries, including Italy, by applying for the first time new generations of nonparametric efficiency measures on a large scale and providing micro-based evidence on the evolution of the strategic profile of universities in terms of scientific research, contract research, education, and the Third Mission. In another study, Agasisti and Johnes (2010) evaluated the efficiency of Italian universities, demonstrating that there is a close relationship between the size and efficiency of universities; moreover, in their work, they highlighted that the growth in the size of universities reduces the overall efficiency of scientific research in the same. Even earlier, more specific analysis on the efficiency of teaching by Italian universities includes: Ferrari and Laureti (2005), Laureti (2008) and Laureti et al. (2014). A rich survey on the efficiency of universities can be found in Worthington (2001); while De Witte and López-Torres (2017) is the most recent and comprehensive survey available in the field.

The main critical points that emerge from the empirical literature relating to the evaluation and rankings of universities can be summarized as follows: (i) one-dimensionality; (ii) lack of statistical robustness; (iii) dependence on the size of the university and on its subject mix; (iv) lack of consideration of the input–output structure. See Daraio et al. (2015a, 2015b) and Daraio and Bonaccorsi (2017) for a deeper description of the literature on rankings and empirical investigations. In addition to these critical aspects, university education is subject to rapid changes, due to continuous reforms, which make the evaluation process particularly complicated. In Italy (see, e.g., Agasisti & Dal Bianco, 2009), thanks to law n.270/2004, which reformed the university system, the differentiation between three-year bachelor courses and master’s degree courses has grown considerably, gradually allowing for the adoption of different teaching practices depending on the level of university education. This aspect has not yet been adequately assessed and monitored, even though it concerns one of the main activities of universities, which involves almost two million students every year, and which represents the first source of funding for universities (both public and private) and therefore the main area of competition.

The difficulty in applying efficiency methodologies at the level of individual degree courses consists mainly in finding the data: in fact, as reported in the comprehensive analysis of the literature proposed by De Witte and López-Torres (2017), only two papers, Cooper and Cohn (1997) and De Witte and Rogge (2011), carried out a study of the single degree courses, and neither used the methodology applied here, which allows for taking into account the temporal dimension, and therefore the adjustments dynamic, following the work of Mastromarco and Simar (2015).

In this chapter, we aim to develop new models for estimating the efficiency of Italian faculty courses exploiting the new information contents that can be processed from the SUA-CdS forms, useful for identifying the most efficient teaching practices for better monitoring the performance of teaching, comparatively, among Italian universities.

The indicators identified for the analysis of efficiency are detailed in Sect. 4, and relate to basic institutional information, geographic information, training activities conducted, personnel (including gender), and size.

3 Methodology

Our econometric approach is based on recent developments of Data Envelopment Analysis (DEA), which originated from the seminal papers of Farrell (1957) and Charnes, Cooper, and Rhodes (1978). DEA uses linear programming to compare and benchmark a sample of observed units—in our study, faculty courses—against the efficient production frontier, which consists of combinations of observed production possibilities. DEA relies on a minimum set of hypotheses that are: i) free disposability (that is, the possibility to destroy goods without costs) and ii) convexity.

The main advantage of the approach is its multidimensionality; i.e., multi-input multi-output performance evaluation, without any assumption on the functional relationship between inputs and outputs. DEA is nonparametric because it does not make assumptions about the distribution of inefficiencies or the functional form of the production function. On the contrary, it uses the input and output data themselves to estimate the production possibility frontier. This nonparametric approach does not require assumptions about the behavior of the analyzed units, such as cost minimization or profit maximization, which are not appropriate for the Higher Education context, and does not require the knowledge of input and output prices, which are often unknown in the Higher Education context (Daraio, 2018; Johnes, 2006). In this study we use advanced and robust nonparametric tools recently developed in nonparametric efficiency frontier literature called “conditional efficient frontier” models (Cazals et al., 2002; Daraio & Simar, 2005; Mastromarco & Simar, 2015) whose main ideas are outlined below. Additional methodological details are reported in the Appendix.

The aim of this study is the analysis of the performance of teaching at Italian universities by comparing the efficiency of university faculty courses at the national level. For this purpose, we consider the university faculty courses as the relevant unit of analysis, whose efficiency in producing knowledge can be evaluated by applying a conditional efficiency approach. In an efficiency analysis, performances are measured with respect to the best-practice frontier (or efficient frontier) constructed by comparing the outputs achieved, given the inputs possessed, of the units analyzed. In a conditional efficiency analysis, conditioning or environmental factors are included in the measurement of performance. These factors are neither inputs nor outputs of the production process but may influence the performance of the units analyzed (see, e.g., Daraio & Simar, 2007a).

Let the production process of teaching activities in university faculty courses be characterized by one input X ∈ R₊ (number of teachers weighted by teaching hours), and two outputs$\mathrm{Y}\in {\mathrm{R}}_{+}^2$, (the percentage of graduated students within the legal duration of the degree course and the percentage of graduates satisfied with their faculty course).

The objective is to evaluate the efficiency with which the inputs (number of teachers compared to the number of students, weighted by the hours of teaching) of the different study programs determines the outputs (percentage of graduates within the legal duration of the degree course and percentage of students satisfied with the course of study). This model considers the quality of teaching as given and not observable, and instead analyzes the teachers/students ratio in relation to the completion of the course of study on time and student-satisfaction rate. It would also be important to take into account the quality of teaching, but data limitation does not allow us to control for quality. However, our conditional approach enables us to include other factors, which are neither inputs nor outputs, which may affect and limit the units under analysis in terms of reaching the efficient output levels. We assume that there are some external variables $\mathrm{Z}\in {\mathrm{R}}_{+}^{\mathrm{d}}$ that may influence the performance of the faculty courses. We consider the size of the different faculty courses, which may capture the heterogeneity among them. To avoid distortion, due to time delays or time adjustments, in the evaluation of the performance of different faculty courses, we take into account temporal dynamics.

Being nonparametric, the conditional efficiency models suffer from the so-called curse of dimensionality, which means the need to use parsimonious models in terms of input and output numbers to avoid inaccurate estimates of efficiency scores.

To overcome the curse of dimensionality, then, we limit the dimension of our model and consider only one input (weighted teacher-to-students ratio), two outputs (% of graduates within regular duration and student-satisfaction rate), and two environmental variables (size and time).

4 Data and Empirical Strategy

Data come from SUA-CdS forms and ad hoc surveys on graduates conducted by ANVUR and MIUR and refer to the period 2013–2017.

The analysis was conducted at the level of a single university course of study, using annual data from 2013 to 2017. The entire dataset was divided into three-year degree courses and master’s degree courses in order to make the analysis more homogeneous. We have accounted for the heterogeneity among disciplines by carrying out the analysis distinguishing among subject areas, considering sciences, health sciences, social sciences, and humanities. This approach makes the empirical analysis more homogenous and allows us to consider the peculiarities of the courses in different subject areas, as with, for example, the infrastructure and the size in health science and science degree courses.

Unique-cycle degree courses (like law and veterinary) were annexed to the datasets related to master’s degree courses. In this way, we obtained eight datasets and, therefore, eight analyses: the three-year degree courses were divided into the four subject areas, as were the master’s degree courses. The summary statistics and the results of the analysis are therefore being presented for each subject area.

All degree courses with a temporal dimension fewer than three years were eliminated from the dataset. This threshold was chosen in accordance with the period needed to conclude one study cycle that is a three-year period. The same threshold is useful in order to improve the homogeneity of the analysis and to be able to consider the temporal trend of the efficiency of different degree courses. After cleaning the data, we obtained 1907 BA courses and 2138 master’s courses.

Table 1 shows the distribution of the degree courses and master’s courses by disciplinary area. We note that in Italy there is a prevalence of courses in the scientific field as scientific courses represent 41% and 45% of the overall degree courses and overall master’s courses, respectively.

Table 1 Number of analyzed courses by subject area

The purpose of this study is to analyze the performance, in terms of efficiency, of the various university courses. We chose the following outputs:

• (i) The “percentage of graduates within the legal duration of the degree course” (iC02)

• (ii) The “percentage of graduates overall satisfied with the study course” (iC25).

The chosen input is the “number of teachers over the number of students (weighted by teaching hours),” which is the inverse of the ANVUR iC27 indicator. As an external factor, we use “size”; that is, the number of enrolled students of the considered study courses.

See Table 2 for some descriptive statistics on the input, outputs, and external factor “size” by degree and master’s courses and by discipline. It can be seen that the areas with the highest number of students are those related to Science and Social Science. The average of “percentage of graduates within the legal duration of the degree course” (iC02) is quite similar between the different disciplines, while the average of “percentage of graduates overall satisfied with the course of study” (iC25) varies greatly depending on the subject area. The input “number of teachers over the number of students (weighted by teaching hours)” (the inverse of the iC27) has a lot of heterogeneity between the different groups of degrees.

Table 2 Summary statistics on input, outputs, and size

The objective is to evaluate the efficiency with which the input (number of teachers compared to the number of students, weighted by the hours of teaching) of the different study programs determines the outputs (percentage of graduates within the legal duration of the degree course and percentage of students satisfied with the course of study). Regarding the choice of the input variable, we need to make some considerations. The Italian education system does not report costs and human resources at the level of a single university course of study. In general, the data are aggregated at the level of departments (or faculties). The input indicator we chose, then, is the only one available at the study-course level. Nevertheless, we think it may adequately proxy the concept of teaching resource at the study-course level.

The data are from a dashboard of indicators that each university uses for its own evaluation and self-assessment. These indicators are the first and only source of data available at the level of individual university courses. The originality of the work consists in the comparative efficiency analysis conducted at the level of the single course of study, carried out for the first time in the education literature considering all the courses at the national level.

As described in Sect. 3, the methodology applied in this work is conditional efficiency. It allowed us to obtain the conditional efficiency on some environmental (or contextual) factors that may influence the production process. We conditioned on size, as the number of enrolled students in the subject area of the study course. In particular, we analyzed whether and how the number of students influences the efficiency of the course. Size was measured by the number of students enrolled in bachelor’s and master’s degree courses of the specific subject area of the course analyzed. For example, the efficiency of the degree in economics at the University of Salento was conditioned by the number of students enrolled in social science degrees at the University of Salento.

Following the approach proposed by Mastromarco and Simar (2015), we carried out a time-dependent analysis, which allowed us to measure the time-dependent efficiency of university courses and to assess the effect of time on the performance by taking into account time delays and adjustment lags.

To illustrate the conditional efficiency scores calculated for each course of study, we considered (i) the geographic area of the universities and (ii) the gender composition of teachers.

We distinguished between universities in central-Northern and Southern Italy. This focus greatly strengthens the policy recommendations for a strategic sector, such as education, for the economic development of Italy, which has always suffered from serious geographic disparities.

Considering the importance of gender balance, especially in advanced studies in the STEM (Science, Technology, Engineering, and Mathematics) fields, we calculated an indicator of gender prevalence. Regarding this topic, to which policymakers have recently been paying attention, we found it difficult to find data at the level of an individual study course. To overcome this problem, our indicator was calculated for each International Standard Classification of Education (ISCED) Field of Education and Training (FOET2013) of each university as follows. If in each FOET2013, the number of female-dominated courses exceeded the number of male-dominated courses, the field was assigned the female-oriented discipline label. If the number of male-dominated courses exceeded the number of female-dominated courses, the FOET2013 classification of the university was assigned the male-oriented discipline label. If the number of male-dominated courses was equal to the number of female-dominated courses, the field was assigned the gender-neutral discipline label. Finally, the relevant label was assigned to each degree course based on the correspondence between the Italian degree classes and the FOET2013 nomenclature.¹

5 Results

In the first step, we studied the impact of time and size on the efficient frontier of the production process of the study courses. This is done by investigating the ratios of conditional and unconditional efficiency scores for the robust full frontier calculated with α = 0.99 (see Appendix). Subsequently, we analyze the effect of the conditioning variables (time and size) on the distribution of efficiency—that is, on the distance of the units from the efficient frontier—by inspecting the graph related to the robust partial frontier estimated in the middle of data with α = 0.5 (see Appendix).

The second step involved obtaining the average efficiency results of the individual university degree courses. For the sake of clarity, only the results of the 20 best (on average) degree courses and 20 worst (on average) courses of study for each subject area are presented here.

The third and last part of the results focuses on the efficiency trends of the study courses, with particular attention given to the average variation of efficiency scores. Emphasizing the tendency of improvement or deterioration is an important aspect in evaluating a university course. The purpose of the efficiency analysis is not, in fact, to punish or reward educational institutions but to stimulate a process of change. Following the Mastromarco and Simar (2015) approach, we were able to evaluate in a robust way the dynamics of efficiency during the time of each course, which provides useful insights. Scatterplots are shown to visualize the relationship between improvements and efficiency starting levels. In the following we present the results, tables, and figures of the three-year degree courses. For space reasons, in the supplementary material, available on line, we report all the figures for the master’s degree courses.

5.1 Effect of Size on Efficiency

We investigated the effect of size and time on the boundary of the efficient frontier, hence on the best-performing courses, in different areas. We started with science courses that represent more than 40% of our sample. Figure 1 shows the impact of size on the efficiency of three-year degree courses in science. Size (on the x-axis) is the number of enrolled students in science. On the y-axis R_0,(x,y|z,t) (𝜶 = 0.99, see Appendix for more details) are the ratios of the conditional to unconditional efficiency scores. An increasing (decreasing) trend of ratios identifies a positive (negative) impact of the dimension on the efficient frontier of courses in science. A flat trend shows no effect of size on the efficient frontier. Inspecting Fig. 1, we note an important effect of the size—as number of enrolled students in the subject area—on the efficient frontier. In particular, there is, first, a negative and then positive effect on the efficiency frontier (i.e., the maximum achievable output values, given the available input). This suggests a negative effect of number of students in small-medium universities and a positive effect of size on the most efficient courses in sciences in larger universities. This finding is explained by the fact that science courses need specialized structures as well as laboratories. Thus, only larger universities, which are able to invest in large infrastructures, may obtain good results when they increase the number of students enrolled. Indeed, the best courses in science in big universities may afford new investments to avoid congestion costs. Figure 1 also highlights some universities with the best courses in science.

Fig. 1

Impact of size on the efficient frontier of three-year degree courses in science. Size (on the x-axis) is the number of enrolled students in science. On the y-axis R_0,(x,y|z,t) (𝜶 =0.99, see Appendix for more details) are the ratios of the conditional to unconditional efficiency scores. An increasing (decreasing) trend of ratios identifies a positive (negative) impact of size on the efficient frontier of courses in science. A flat trend shows no effect of size on the efficient frontier

Fig. 2

Impact of size on the distribution of the efficiency of three-year degree courses in social science. Size (on the x-axis) is the number of enrolled students in social science. On the y-axis R_0,(x,y|z,t) (𝜶 =0.50, see Appendix for more details) are the ratios of the conditional to unconditional efficiency scores. An increasing (decreasing) trend of ratios identifies a positive (negative) impact of size on the distribution of the efficiency of courses in social science. A flat trend shows no effect of size on the distribution of the efficiency

The effect of the number of students enrolled in degree courses in the health sciences does not seem relevant for the most efficient courses. This result is not surprising; in fact, in order to have a degree course in health sciences, universities must collaborate with important clinical centers (e.g., hospitals), and the number of students who can access medical degrees is limited by law. In addition, the presence of some medical laboratories and other infrastructures is necessary to offer these courses. This makes the size of this area, measured as the number of students enrolled, similar for all universities with medical degree courses and, hence, not particularly influential. To save space, we do not include graphs for the other university areas, which are available upon request.

A result similar to health science is obtained for the degree courses in social sciences and humanities. In these cases, since no particular laboratories or infrastructure is necessary, the degree courses allow for a great deal of flexibility in the number of enrolled students. Despite this great variability of size, as can also be seen from the descriptive statistics in Table 2, there is no particular influence of the size on the efficiency of the best courses in these areas. Concerning the analysis of the possible effect of the size on the distribution of the efficiency scores of the university courses, we evaluated the trend of the ratios between the conditional and the unconditional efficiency scores using a partial frontier with alpha = 0.5, which captures the middle of the distribution of the efficiency (see Appendix). For sciences, the effect of size on the ability of degree courses below the frontier to reach full efficiency (i.e., maximum outputs value, given the available input) is similar to the previous case on the most efficient degree courses (see Fig. 1). In particular, size (i.e., the number of students enrolled in the area) has a negative impact on universities with a medium- to small-sized scientific area, while it has a positive impact on universities with a bigger scientific area. The effect of size in the health sciences does not seem relevant for the achievement of efficient output values of three-year degree courses. For social science courses (see Fig. 2), size has a slightly negative effect for medium-small universities and a positive effect for larger ones. Furthermore, for universities with a number of enrolled students higher than 20,000, a slight negative effect of size on the efficiency of university courses over time is observed. This finding demonstrates how excess overcrowding can cause problems and negatively affect the efficiency of the courses.

For humanities, the number of enrolled students has a negative effect on efficiency of courses in universities with a small number of students, while it has a positive effect on efficiency of courses in medium-sized universities (i.e., 6000–11,000 enrolled students). The effect is again positive for universities with over 12,000 students.

5.2 Analysis of the Efficiency Scores

We now analyze the twenty best and worst degree courses over the five years under analysis. The top panel of Table 3 shows the ranking of the 20 most efficient degree courses in science, among which 19 universities are located in the center-North and only one university in Southern Italy. This result is not surprising. It is well known that the industrial sector is a natural job outlet for science degrees. The universities in Northern Italy have more relationships with companies and industries, as they are concentrated in the North of the country. As a result, there are many job opportunities for students who may build relationships with these companies during their university studies. In this area, with strong STEM characterization, from a gender point of view, we find a female predominance in six out of the 20 best courses (highlighted in bold) and one neutral. Among the twenty least efficient degree courses (bottom of Table 3), we find 14 degree courses in Southern Italy and six in the center-North, which is substantially reversed compared to the ranking of the best courses. From a gender point of view, we find five degree courses with female prevalence among the worst ones. In both rankings—those with the best and those with the worst courses in science—we find female gender-dominated degree courses in limited numbers. This result is common in the science area, as confirmed in the scientific literature on the subject (see Card & Payne, 2021). Despite this evidence, in Italy, in recent years, there has been a change in this trend.

Table 3 List of the 20 most efficient three-year degree courses in the scientific area (top) and the 20 least efficient three-year degree courses in the scientific area (bottom)

In the ranking of the 20 most efficient degree courses in the health sciences, shown in the top panel of Table 4, we only find courses in the center-North, with a very high frequency of degree courses in Milan universities (11 out of 20). The regional aspect is important for health degree courses. Any university offering a course in medicine is required to establish an agreement with the regional health system. Therefore, it is clear that interregional differences in health systems are also reflected in this ranking. All of these courses are female-gender prevalent (highlighted in bold in the table). This result is not surprising, because it is well known that the presence of women is predominant in this disciplinary area. Among the 20 least efficient degree courses, displayed in the bottom panel of Table 4, considering an average of the period being analyzed, we find 13 degree courses in Southern Italy and seven in the center-North; therefore, this situation is substantially reversed compared to the ranking of the best university courses. Only one course (the worst, in terms of efficiency) is male-gender prevalent, and three out of 20 are neutral-gender prevalent.

Table 4 List of the 20 most efficient three-year degree courses in the health science area (top) and the 20 least efficient three-year degree courses in the health science area (bottom)

In the ranking of the 20 most efficient degree courses in the social sciences, shown in the top panel of Table 5, we only find courses from the central-Northern regions, with a very high frequency of degree courses in Milan universities (12 out of 20). This evidence depends on the presence, in the center-North, of universities considered to be excellent in the economic and social sciences. Many of these courses (15 out of 20) are female oriented. This result is due to pedagogy and social science courses, which mainly attract female students (Francesconi & Parey, 2018). Continuing with the evaluation of the average efficiency of courses, the bottom panel of Table 5 reports the 20 degree courses deemed least efficient. We find 12 degree courses from Southern Italy and eight from central-Northern Italy; therefore, this situation is substantially reversed with respect to the ranking of the most efficient social science courses. All of these courses, except the first one, are female oriented (highlighted in bold). This result is in line with what was previously described—namely, the fact that many courses of study in this area find the presence of many female students.

Table 5 List of the 20 most efficient three-year degree courses in the social science area (top) and the 20 least efficient three-year degree courses in the social science area (bottom)

In the ranking of the 20 most efficient degree courses in the humanities, displayed in the top panel of Table 6, we find many courses in the central-Northern regions, but, unlike the previously analyzed area (social sciences), the frequency of degree courses in universities of Milan is only four out of 20. This result confirms that, at the level of three-year degree courses, the disparities between the North and South of Italy are great among the best courses, based on our methodology. With the exception of three courses, all humanities courses are female oriented, in line with expectations. The bottom panel of Table 6 presents the least efficient courses in this area and illustrates a substantially different situation compared to the best ones. Among the 20 least efficient degree courses, we find 18 degree courses in Southern Italy and two in the center-North. These strong territorial imbalances, even in the humanities (thus, degree courses that do not require many resources in terms of infrastructure), are worrisome. Strong disparities exist in terms of educational efficiency between the Northern and Southern universities. The Italian university system seems strongly characterized by polarization. The courses in humanities belong to a female-oriented area (highlighted in bold).

Table 6 List of the 20 most efficient three-year degree courses in the humanities area (top) and the 20 least efficient three-year degree courses in the humanities area (bottom)

5.3 Trends and Prospects

We conclude the analysis of the three-year degrees with comments on the geometric mean of the efficiency variation rates to better understand the direction of the dynamism of the individual degree courses. It is important to understand which degree courses have improved the most over the period considered. Continuous improvement, evaluation, and self-evaluation are among the key principles of the Italian legislation on the evaluation of the university teaching system. Figure 3 refers to science courses and offers an overview of the relationship between the efficiency in the first year of analysis and the average rate of change during the observation period. The courses belonging to Southern universities are colored in red, while those belonging to the Northern-central universities are colored in blue. The top and bottom panels of the chart report the courses with highest and lowest levels of efficiency at the first year, the left and right panel the ones with lowest and highest levels of efficiency variation. It is possible to note how the study courses that started from lower efficiency levels are characterized by high volatility in the rate of change of efficiency. Some courses that started at a low level of efficiency registered significant improvement rates, while others displayed a consistent deterioration rate, especially for courses of universities in the South.

Fig. 3

Change in efficiency of three-year degree courses in science. Efficiency in the first year of analysis (y-axis) versus the average improvement rate of the degree courses (x-axis). Blue circles are courses belonging to Northern-central universities, while red circles are courses belonging to Southern universities

Figure 4 shows us the same relationship for degree courses in health sciences. These courses exhibit low variability. This is typical of medical degree courses. Accreditation and regulatory provisions are stringent and limit the autonomy of degree courses, which implies a reduced variability in terms of efficiency. From the same picture, we can conclude that degree courses that have shown a consistent positive rate of growth belong to universities located in the North.

Fig. 4

Change in efficiency of three-year degree courses in health science. Efficiency in the first year of analysis (y-axis) versus the average improvement rate of the degree courses (x-axis). Blue circles are courses belonging to Northern-central universities, while red circles are courses belonging to Southern universities

Figure 5 provides useful information on the relationship between the rate of change of efficiency and the level of efficiency in the first year for social sciences courses. It is evident that the most efficient courses are located in the center and North. The courses that started at low efficiency levels are equally distributed among the geographical areas. What is important to underline in this graph is that many courses have significant improvement rates. The best courses during the first year, in contrast, have a low variability and, hence, they keep high performance during all observed periods.

Fig. 5

Change in efficiency of three-year degree courses in social science. Efficiency in the first year of analysis (y-axis) versus the average improvement rate of the degree courses (x-axis). Blue circles are courses belonging to Northern-central universities, while red circles are courses belonging to Southern universities

Figure 6 shows the same relationship for the three-year degree courses in humanities. We can appreciate that the courses that register low efficiency in the first year of analysis have positive average rates of variation. Few courses have worsened their low position and find themselves in the lower-left quadrant of the chart. Yet regarding degree courses with a medium-high level of efficiency, the tendency toward a moderate deterioration is generally apparent (top-left panel of the graph).

Fig. 6

Change in efficiency of three-year degree courses in humanities. Efficiency in the first year of analysis (y-axis) versus the average improvement rate of the degree courses (x-axis). Blue circles are courses belonging to Northern-central universities, while red circles are courses belonging to Southern universities

6 Results on Master’s Degrees

6.1 Effect of Size on Efficiency

The analysis of the efficiency of the master’s degrees follows the same structure of the analysis carried out on the three-year degree courses presented in the previous section.

Figure 7 in supplementary material shows the impact of size on the efficient frontier of master’s degree courses in science. In universities with small numbers, in terms of enrolled students in sciences, there is no effect of the number of enrolled students on the efficiency of the best master’s courses. Therefore, the number of enrolled students does not affect the outputs (percentages of graduates within the legal duration of the course and satisfied by the course of study), given the input (the number of teachers per students weighted by the number of teaching hours). There is a negative effect when moving from universities with small-sized science courses to those with medium-sized ones, and then a positive effect of the size in the transition from universities with medium-sized science courses to those with large-sized ones.

The impact of size on the efficiency of the health science master’s courses is illustrated in Fig. 8 in supplementary material. There is no influence of the number of students enrolled in degree courses in the health sciences on the most efficient courses of study. The same conclusion can be drawn for the social sciences (see Fig. 9). It is useful to emphasize how the distribution of size in this discipline is quite homogeneous.

Figure 10 in supplementary material shows the impact of size on the efficiency of master’s courses in the humanities. The ratios show a negative influence of the size on the efficiency of degree courses in the humanities. As the size of the disciplinary area increases, the effect becomes ever greater and remains constantly negative, indicating that master’s courses in humanities are more efficient when the number of students in this area is small.

The relationship between conditional and unconditional efficiency, calculated with respect to the partial frontier with α = 0.5, shows the effect of the conditional factor (in our case size) on the distribution of efficiency and therefore on the convergence process of the least efficient courses compared to those at the efficiency frontier (the best-performing ones).

Figure 11 in supplementary material shows the impact of size on the efficiency distribution of master’s courses in science. The effect of students enrolled in this area on the master’s degrees courses efficiency distribution is negative in universities with small-sized science departments (0–25,000) and positive in universities with medium-large science departments (30,000-45,000). Hence, the increasing number of enrolled students in this area improves the efficiency of master’s courses. Furthermore, it is worthwhile to note that the size has a positive effect on efficiency only starting from some values. Therefore, it seems there is a threshold effect, and this area needs to reach a certain number of enrolled students to have a positive effect on efficiency from an increasing number of enrolled students.

In health and social science (not reported for space reasons), size does not seem to affect the process of convergence of less efficient master’s courses toward the efficient frontier, so size does not improve efficiency for health and social science master’s degrees. In the case of master’s degrees in the humanities, the effect of size on the distribution of efficiency is quite heterogeneous (see Fig. 12 in supplementary material). Fig. 12, however, shows that the size of the humanities (number of enrolled students) certainly has a negative effect on universities with a small area in this field. Concerning universities with medium-sized humanities areas, the effect seems to be positive, which indicates that growth of enrolled students favors improvement of the efficiency of master’s degree courses. This trend is quite similar to science master’s degrees.

6.2 Analysis of the Efficiency Scores

Now we move on to assess the efficiency of master’s degree courses, focusing on the 20 that performed best and worst on average during the five years considered in the study.

In the ranking of the 20 most efficient master’s degree courses in the sciences, shown in the top panel of Table 7, we mainly find courses from the center-North (16 out of 20). The majority of these courses (13 out of 20) belong to female-oriented disciplines and are highlighted in bold in the table.

Table 7 List of the 20 most efficient three-year degree courses in the science area (top) and the 20 least efficient three-year degree courses in the science area (bottom)

The bottom panel of Table 7, concerning the less efficient master’s degree courses, illustrates a substantially different situation compared to the more efficient master’s degree courses. It is interesting to note that this is the first case in which all three of the worst courses in terms of efficiency belong to universities in the center-North. Among the 20 least efficient degree courses, on average, in the five years considered, we find five degree courses from Southern Italy and 15 from central-Northern Italy. Overall, master’s degree courses in the sciences present a more balanced situation in the distribution of inefficient courses between the North and South of the country.

All of the courses at the top of the ranking of the least efficient master’s degree courses in the sciences are male oriented. The situation is similar to the three-year degrees in the sciences. This finding corroborates our hypothesis that the more developed industrial context in central-Northern Italy plays a central role in improving the efficiency in sciences master’s courses. Moreover, the two outputs we have selected—regularity in studies and satisfaction in the courses—are aspects related to the labor market.

In the ranking of the 20 most efficient degree courses in the health sciences, shown at the top of Table 8, we mainly find courses in the center-North (16/20), with a high frequency of degree courses in universities in the Lombardy region (5/20). The ranking shows that a large part (16/20) of health science degrees is female prevalent (highlighted in bold in the table). The bottom part of Table 8, regarding the less efficient master’s courses in health science, illustrates a substantially similar situation compared to the more efficient courses in the discipline. Among the 20 least efficient degree courses, in the period studied, we find eight degree courses in universities in Southern Italy and 12 in central-Northern Italy. All these courses, except for “Medicina e Chirurgia” at the University of Pavia, belong to the female-dominated field.

Table 8 List of the 20 most efficient master’s degree courses in the health science area (top) and the 20 least efficient master’s degree courses in the health science area (bottom)

In the ranking of the 20 most efficient degree courses in the social sciences, presented in the top panel of Table 9, we find courses all in universities in the center-North (except one that belongs to Federico II University), with a very high frequency of degree courses in universities in Milan (10/20). The majority of these courses (13/20) belong to the female-dominated field and are highlighted in bold in the table. The bottom part of Table 9, regarding the less efficient courses in social sciences, illustrates a substantially different situation compared to the most efficient ones. It is interesting to note that most of these courses are in law disciplines and belong to universities in Southern Italy. Among the 20 least efficient degree courses, on average, for the period under analysis, we find 11 degree courses in Southern Italy and nine in the center-North. It is, therefore, a vastly different situation compared to the ranking of the best courses.

Table 9 List of the 20 most efficient master’s degree courses in the social science area (top) and the 20 least efficient master’s degree courses in the social science area (bottom)

In the ranking of the 20 most efficient degree courses in humanities, exhibited at the top of Table 10, we find nearly all courses in the center-North (18/20) with a high frequency—although a lower one than the three-year degrees—of university degree courses in Milan (6/20). Except for the cognitive science program at the University of Trento, all the best humanities master’s degrees belong to female-dominated fields and are highlighted in bold in the table. The bottom part of Table 10 reports degree courses rated as less efficient. Among the twenty less efficient degree courses, we find 13 degree courses in Southern Italy and seven in the center-North; therefore, the situation is substantially reversed compared to the ranking of the best master’s courses in humanities.

Table 10 List of the 20 most efficient master’s degree courses in the humanities area (top) and the 20 least efficient master’s degree courses in the humanities area (bottom)

6.3 Trends and Prospects

This section is devoted to the analysis of changes in efficiency of master’s degrees in order to better understand the direction of the dynamism of the degree courses. University legislation is geared toward improving quality. Therefore, it is useful to understand which master’s course has improved the most or which one has deteriorated the most, starting from the initial level of efficiency of the degree course. This will help in understanding the dynamism of the master’s courses of the Italian university system.

As done in the previous section, we present the results by relating the level of efficiency in the first year of analysis and the average improvement rate of each master’s degree course. In the following graphs, the units in red are the master’s degree courses offered in universities located in Southern Italy, and the circles in blue are the master’s degree courses offered in universities in central-Northern Italy.

The results for the science master’s degree programs are shown in Fig. 13 in supplementary material. For most of them, the rate of change is low. Master’s degrees that have a low to medium level of efficiency in the first year and improve significantly can be seen in the lower-right quadrant of the figure. Regarding the courses in the health sciences, as presented in Fig. 14 in supplementary material, we note a greater variability compared to the three-year degrees in the same area. Furthermore, it is evident from the graph that master’s courses that started from a situation of high efficiency show positive variation rates (upper-right quadrant of the figure). Therefore, these courses further increase the efficiency. The courses that started from a low level of efficiency further worsened their performance (lower-right quadrant of the figure), except few of them, and especially the courses in Federico II University, which start from a very low level of efficiency but improve their performance and register the highest rate of positive efficiency change. As for the social sciences, displayed in Fig. 15 in supplementary material as a pyramid structure, the courses that started with particularly good efficiency have undergone little variation. The degree courses that started from lower efficiency, in contrast, have registered a high level of variability in the average rate of change. As for the humanities master’s degree courses, as revealed in Fig. 16 in supplementary material, the courses that already started from a good position have mostly improved. Unfortunately, many post-graduate courses that had lower levels of efficiency have deteriorated, especially in the North (blue circles in the lower-left quadrant in the figure).

7 Conclusion

This chapter contributes to the literature on the evaluation of universities by proposing a new method of evaluating the ranking and performance of Italian universities from a perspective of the efficiency of the teaching offered at the level of study course, overcoming simple monodimensional indicators used in this context.

For the first time, an evaluation of the efficiency of study courses is proposed and assessed by considering as outputs the number of graduates within the legal duration of the courses and their satisfaction with the course of study followed, and as input the number of teachers per student weighted by the number of teaching hours. The efficiency is calculated taking in account the effect of time and size (number of students enrolled in the disciplinary fields). Furthermore, for the first time for this type of analysis, the results are analyzed in a gender-balance perspective, identifying to which gender-oriented area the best and worst courses belong.

The results highlighted a greater efficiency of the university courses of the universities of the center-North. In addition, the analysis has highlighted how the universities in the islands, especially in Sicily, suffer from serious problems of inefficiency for their courses of study, with a tendency to worsen, as evidenced by the negative temporal variation rates of efficiency.

High efficiency values are obtained in particular from the scientific and health science areas, which involve disciplines with a high technological content strictly connected with the industrial sector. The values of the efficiency of the courses of study in these areas seem to reflect the economic reality of the territories in which they operate. In particular, high values of efficiency in the scientific and health science area study courses in the center-North stand out, where there is an industrial sector capable of absorbing the human capital formed by universities, enhancing their skills, and interacting in the formation of the same through partnerships and projects with the university world aimed mainly at the professional integration of specific figures required by the world of work. This happens mainly for scientific areas, such as engineering, which in Northern Italy has employment rates that exceed 80%. With regard to gender policies, in this STEM area, an interesting aspect is highlighted: the number of the best courses that are female oriented is growing, while the worst courses are highly male oriented. As far as the health science sector is concerned, the gap between North and South is even wider in terms of degree course efficiency. The management of health care on a regional basis and greater efficiency in the allocation of economic resources in these territories indicates that the universities, which work alongside the hospitals of excellence, widely present in the center-North, also incentivize efficiency, as well as the satisfaction of recent graduates, as seen from the results obtained on the study courses of this disciplinary area.

The humanities area, on the other hand, presents a trend reversal in relation to the efficiency of its study courses, which are also efficient in the South—certainly an encouraging result for this disciplinary area.

A possible extension of this work could consist in the inclusion of variables of the socioeconomic context of the territories in which Italian universities operate—territories with different economic resources and employment scenarios, which follow their own dynamics, often divergent from those of the efficiency of university centers. One possibility would be to include postgraduate employment data. The only source in this regard is the AlmaLaurea database, which, however, does not provide information on the type of employment contract.

Another interesting extension of this work could be the application of recently developed efficiency methodologies (see Daraio et al., 2021) able to estimate quality as a latent heterogeneity factor in the efficiency of the university faculty courses providing quality-adjusted rankings of Italian university faculty courses. All these extensions are left for future research.

The main contribution of this work is to have shown how, by using university faculty courses as a relevant unit of analysis, it is possible to apply the new conditional efficiency analysis methodologies to provide multidimensional comparisons and rankings by discipline at the national level. Obviously, the proposed analyses suffer from limitations due to existing data, highlighting the need to invest in the creation of more complete databases based on information coming from different sources.

Appendix

Let the production process of teaching activities in the university faculty courses be characterized by a vector of inputs$\mathrm{X}\in {\mathrm{R}}_{+}^{\mathrm{p}}$that produces a vector of output$\mathrm{Y}\in {\mathrm{R}}_{+}^{\mathrm{q}}$ and a vector of environmental variables $\mathrm{Z}\in {\mathrm{R}}_{+}^{\mathrm{d}}$ may affect the performance of this production process. For each time period t, the attainable production set ${\boldsymbol{\varPsi}}_{\boldsymbol{t}}^{\boldsymbol{z}}\subset {\boldsymbol{R}}_{+}^{\boldsymbol{p}+\boldsymbol{q}}$ can be defined as the support of the conditional probability of being dominated (Cazals et al., 2002; Daraio & Simar, 2005; Mastromarco & Simar, 2015), given by

$$ {\mathrm{H}}_{X,Y\mid Z}^t\left(x,y|z\right)=\mathrm{Prob}\left(X\le x,Y\ge y|Z=z,T=t\right). $$

(A1)

As Mastromarco and Simar (2015) suggest, for each period t, the conditional output-oriented efficiency of a production plan (x, y) facing conditions z, is defined as

$$ {\uplambda}_t\left(x,y|z\right)=\sup \left\{\uplambda |\left(x,\uplambda \mathrm{y}\right)\in {\Psi}_t^z\right\}=\sup \left\{\uplambda |{{\mathrm{S}}^t}_{Y\mid X,Z}\ \left(\uplambda y|x,z\right)>0\right\} $$

(A2)

where ${\mathrm{S}}_{Y\mid X,Z}^t\left(y|x,z\right)=\mathrm{Prob}\left(\mathrm{Y}\ge y\right|X\le x,Z=z,T=t\Big)$ is a nonstandard conditional survival function.

The unconditional output-oriented efficiency of the production plan (x, y) is given by

$$ \uplambda \left(x,y\right)=\sup \left\{\uplambda |\left(x,\uplambda y\right)\in \Psi\ \right\}=\sup \left\{\uplambda |{\mathrm{S}}_{Y\mid X}\ \left(\uplambda y|x\right)>0\right\} $$

(A3)

where S_Y ∣ X (y| x) = Prob(Y ≥ y|X ≤ x) is a nonstandard survival function conditioned only to the inputs but non-conditioned to time and Z.

The estimation of conditional distributions ${\mathrm{S}}_{Y\mid X,Z}^t\left(y|x,z\right)$ where we condition on X ≤ x and a particular value of Z = z and =t, is given by

$$ {\hat{\mathrm{S}}}_{Y\mid X,Z}^t\left(y|x,z\right)=\frac{\sum_{j=\left(i,v\right)}\mathrm{I}\left({x}_j\le x,{y}_j\ge y\right){\mathrm{K}}_{h_z}\left({z}_j-z\right){\mathrm{K}}_{h_t}\left(v-t\right)}{\sum_{j=\left(i,v\right)}\mathrm{I}\left({x}_j\le x\right){\mathrm{K}}_{h_z}\left({z}_j-z\right){\mathrm{K}}_{h_t}\left(v-t\right)} $$

(A4)

where K_(.) are kernels with compact support and h_(.) are the bandwidths or smoothing parameters (see Bădin et al., 2010 and 2019, for technical details).² Optimal bandwidths are selected by least squares cross-validation (LSCV), which is asymptotically equivalent to the maximum likelihood (see, for example, Li & Racine, 2007). Daraio and Simar (2005, 2007a) and Bădin et al. (2010) discuss in detail how to choose the appropriate bandwidths. They are determined by the estimation of conditional distributions ${\mathrm{S}}_{\mathrm{Y}\mid \mathrm{X},\mathrm{Z}}^t\left(y|x,z\right)$ on X ≤ x and a particular value of Z = z and T = t, following the approach suggested by Hall et al. (2004) and Li and Racine (2007).

For our analysis, we follow Daouia and Simar (2007) and Mastromarco and Simar (2015) and apply order-α partial frontiers, to provide efficiency scores more robust to outliers and extreme observations. Unconditional and conditional output-oriented robust (or partial) efficiency scores are defined for any α ∈ (0, 1), respectively, as follows:

$$ {\lambda}_{\alpha}\left(x,y\right)=\sup \left\{\lambda |{S}_{Y\mid X}\left(\lambda y|x\right)>1-\alpha \right\} \vspace*{-12pt}$$

(A5)

$$ {\lambda}_{t,\alpha}\left(x,y|z\right)=\sup \left\{\lambda |{S}_{Y\mid XZ}^t\left(\lambda y|x,z\right)>1-\alpha \right\} $$

(A6)

The partial frontiers do not depend on full support of Y under the conditioning, but on a less extreme quantile (unless α = 1) and for this reason are also called robust frontiers. The partial frontiers estimated with values of α close to one provide the same information as the full frontier estimates, but do not envelop all the data points and for this reason are more robust to extremes and outliers. By choosing a central quantile, such as the median (i.e., α = 0.5), it is possible to investigate the effect of Z on the distribution of inefficiencies. On the contrary, values of α close to one (i.e., α = 0.99) allow us to analyze the effect of Z on the efficient frontier.

The ratios to be analyzed are

$$ {\mathrm{R}}_{\mathrm{O}}\left(x,y|z,t\right)=\frac{\uplambda_t\left(x,y|z,t\right)}{\uplambda \left(x,y\right)} $$

(A7)

whose numerator has been defined in Eq. (A2) and the denominator has been defined in Eq. (A3). To be more robust to extremes and outliers, in this study we applied the robust ratios calculated using the partial robust frontiers of order -α given by

$$ {\mathrm{R}}_{\mathrm{O},\upalpha}\left(x,y|z,t\right)=\frac{\lambda_{t,\alpha}\left(x,y|z\right)}{\lambda_{\alpha}\left(x,y\right)} $$

(A8)

whose numerator and denominator have been defined, respectively, in Eqs. (A6 and A5).

We apply these efficiency ratios to explore the influence of the external variables Z on the efficient frontier (using α = 0.99) and on the distribution of the efficiency (using α = 0.50).

To detect the impact of the external (environmental) variables Z on the efficient frontier, Daraio and Simar (2005, 2007a, 2007b) and Bădin et al. (2012) propose to plot the ratios of the conditional to unconditional efficiency scores as a function of the Z variable. In our output-oriented framework, we consider given the input and look at the maximum feasible expansion of the outputs. In this framework, an increasing trend of the ratios denotes a positive impact of Z on the efficient frontier. On the contrary, a decreasing trend of the ratios points to a negative impact of the Z variable on the efficient frontier. A flat trend of the ratios identifies no impact of the Z on the efficient frontier. In our case, the ratios R_0,(x,y|z,t) are calculated using α = 0.99 to ensure a robust estimation of the full efficient frontier.

To investigate the impact of Z on the distribution of efficiency scores, it is necessary to inspect the plot of the ratios R_O(x, y| z, t) calculated using a frontier that captures the center of the distribution, specifying a α = 0.50. Again, an increasing (decreasing) trend of the ratios identifies a positive impact (negative) of the Z on the distribution of the efficiency scores. A flat trend shows no effect of Z on the distribution of efficiency scores.