Introduction

Abstract

Universities are showing increasing interest in measuring research quality, teaching quality and the relationship between them. Research quality affects individual academic careers and has become important for the efficient allocation of public funding which in many countries and especially countries in Europe is the main component of university financial resources. Teaching quality affects students’ careers, and higher quality teaching can reduce dropout rates, improve student performance, and facilitate graduates’ transition to the labor market. The quality of the research and teaching in universities has mutual effects, since good quality and effective teaching is often related to good research performance. Mobility can emphasize these dynamics; the best students and the best teachers may be concentrated in a few universities creating potential quality gaps among public universities.

Universities are showing increasing interest in measuring research quality, teaching quality and the relationship between them. Research quality affects individual academic careers and has become important for the efficient allocation of public funding which in many countries and especially countries in Europe is the main component of university financial resources. Teaching quality affects students’ careers, and higher quality teaching can reduce dropout rates, improve student performance, and facilitate graduates’ transition to the labor market. The quality of the research and teaching in universities has mutual effects, since good quality and effective teaching is often related to good research performance. Mobility can emphasize these dynamics; the best students and the best teachers may be concentrated in a few universities creating potential quality gaps among public universities.

Italy provides an interesting international case study. Rates of tertiary education enrolment in Italy are relatively low and completion rates are even lower, while tuition fees are relatively high compared to those in other European countries. The 2021 edition of OECD Education at a Glance (Table C5.1) reports that in the academic year 2019–2020, average tuition fees in Italian public universities were $2013 for an undergraduate degree and $2252 for a master’s degree compared to $148 and $233 respectively for undergraduate courses in Germany and France and free tertiary education in all the Nordic countries. The Italian university system is mostly a public system but is characterized by one of the lowest public funding rates in Europe. At the same time, according to several indicators research output (measured as number of journal articles) and research quality (in terms of number of citations) are comparable to most other countries at a similar level of economic development.

In Italy government allocates funds to public universities based on their teaching and research performance, the two main missions of academic institutions. About two-thirds of this funding is proportional to the number of students enrolled in the university (weighted according to disciplinary field), with the remainder allocated based on research output weighted by research quality. This funding mechanism abstracts from possible complementarity or substitutability between the two missions. The interaction between teaching and research and the implications for researcher incentives are the focus of the contributions to this volume.

The book brings together contributions from a range of economists, statisticians, and social scientists involved in an ANVUR-sponsored project.¹ The various chapters analyze different dimensions of research and teaching quality and their interaction, using sound statistical methods allowing comparison with other European countries. The aim is to address the question of whether the evaluation of universities and university departments should focus on both quantitative indicators (such as number of published papers, or number of graduates) and other dimensions of teaching and research, since academic careers, teaching, and students’ achievements are strictly intertwined.

The evaluation of teaching and research is addressed also from a gender perspective to try to understand where and when gender discrimination occurs. There is considerable evidence that the glass ceiling is prominent in the Italian academia: women have higher enrolment rates and lower dropout rates relative to men, are represented almost equally at entry to an academic career but despite comparable research productivity are gradually side-lined among higher ranks.

1 Which University Model?

The Italian university system adopts the principle of a regulated autonomy (see Chapter “Governance Reforms in Comparative Perspective and Their Path in the Italian Case” by Capano). The Government, the Parliament, and the Ministry of University and Research decide on the allocation of funding and the rules governing its allocation. There are also detailed rules related to the content of the courses that universities can offer (undergraduate, master’s, doctoral), access to the states of an academic career (PhD, postdoc, and assistant, associate, and full professor), university governance, and department organization. However, universities in Italy can follow different paths.

For example, they might decide to offer courses in most or all disciplines at the undergraduate or graduate level, or choose to specialize in particular subjects and/or levels, and concentrate the resources in particular areas or departments. They might choose to allocate funds internally and reward those researchers and departments that are more productive, or use the funds generated by these departments to invest in the weakest departments. In terms of governance, they might open their boards to external stakeholders, or confine them to mostly incumbent professors. Finally, in terms of organization, they might decide to focus on specialized departments covering specific research areas, or include more and larger departments focused on heterogeneous research areas. Therefore, whether to specialize and give more weight to research than to teaching, or invest in those research areas likely to attract more funding and focus on reputation and international visibility, or offer master’s level and doctoral level courses in only a few fields become relevant issues.

A problem particular to Italian academic institutions is their high drop-out rates especially during the first study cycle. This is exacerbated by the fact that for the average student the duration of the course of study can exceed the authorized length by one year or more, depending on the subject. At the national level, only about two-thirds of the student enrollment graduate, and less than half manage to graduate in three years. Drop-out rates and slow careers are particularly prevalent among students from relatively low-income classes. In Chapters “Do Financial Conditions Play a Role in University Dropout? New Evidence from Administrative Data” and “Drop-out Decisions in a Cohort of Italian Universities” Contini et al. and Atzeni et al. show that there is insufficient public support for students in the form of subsidies, services, and scholarships. However, problem goes deeper with the very small fraction of secondary school students from lower-income classes who enroll in tertiary education. Since Italy has one of the lowest rates of college graduates in Europe, one of the objectives of Italy’s university system should be to improve the country’s human capital² by increasing enrolment after secondary school and reducing dropouts during the course of study.

From a policy perspective there are at least three options. One would be to focus on creating a few national university “champions” with the remaining universities offering mainly undergraduate education. A second option would be to pursue a specialization model which would mean that each university would aim for international parity in the areas of its comparative advantages. The third and more traditional option would be to try to maintain a more balanced tertiary education system involving all universities offering master’s level and doctoral level courses in all fields. These options have different implications for academic careers which do not distinguish between research and teaching positions and apply the same standards to all at both entry level and promotion.

The choice between specialization and a universal model depends on the relation between teaching quality and research quality. In Chapter “The relationship Between Teaching and Research in the Italian University System” Carillo et al. show the complementarity between research and teaching; that is they find that good researchers are also good teachers which would imply that specialization is a sub-optimal solution.

Past contributions by Bratti and coauthors suggest that students “vote with their feet”, and that the best students enroll in higher-ranked universities. In addition, paucity of high-quality study courses results in student mobility to other countries, and in domestic universities attracting only in lower quality national and international students which has long term consequence for human capital formation and growth. Novel results by Bratti et al. in Chapter “Degree-Level Determinants of University Student Performance” suggest that higher education institutions play an important role in ensuring the academic success of their students. Indeed, several degree-level characteristics significantly predict students’ progression and satisfaction with their university education.

Since research and teaching go hand-to-hand, it is important to offer economic incentives and career prospects for young researchers and teachers in particular. Checchi and Cicero (Chapter “Is Entering Italian Academia Getting Harder?”), and De Paola et al. (Chapter “Academic Careers and Fertility Decisions”) show that in the recent past the ability to do this has been limited severely by budget cuts.

2 Incentives

In the case of incentives while in principle the regulatory framework allows funding to be channeled to the best universities, university departments, and individual researchers, in practice this often does not happen.

At the university and department levels, incentives are designed based on the evaluation of their research output. Ferrara et al. (Chapter “Topic-Driven Detection and Analysis of Scholarly Data”), and De Stefano et al. (Chapter “Social Network Tools to Evaluate Individual and Group Scientific Performance”) points out main-streaming and adapting to the rules of the game can occur after the results of three evaluation exercises, and particularly in fields involving research conducted by large research teams rather than individuals.

Each university might implement local incentives for periodic salary increase for the most productive researchers, and might also mobilize internal and external funds to incentivize research, teaching, external finance, and other activities which would allow increased compensation for the most active researchers. However, in practice, these types of rewards are not relevant for differentiating among academic salaries within universities, and most academics boost their income by engaging in consultancy and/or professional activities (lawyers, clinic doctors, architects, etc.).

3 Evaluating University Performance

Several of the chapters in this volume are methodological contributions. Mastromarco et al. in Chapter “Teaching Efficiency of Italian Universities: A Conditional Frontier Analysis” suggest that resources must be distributed efficiently across fields of study and geographical areas which requires measurement of the extent to which current allocations are efficient. Their analysis considers only the distribution of human resources (teachers) but could be extended to integrate other inputs such as infrastructures and staff. Ferrara et al. (Chapter “Topic-Driven Detection and Analysis of Scholarly Data”) propose a framework that could be used to benchmark the research produced by particular universities or particular themes against international research. Also, ANVUR, which sponsored the research for this book, has published several indicators of the impact of teaching on students’ academic careers.

4 A Tour of the Book

The book includes five sections and eleven chapters. This introduction by the editors and a chapter by Capano on “Governance reforms in comparative perspective” comprise the first Chapter. Capano discusses models of university governance in a European framework, and whether the Italian model of steering at a distance is consistent with university autonomy. Lack of guidance about prioritization of research, teaching, and knowledge transfer limits the ability of individual institutions to identify clear strategies to improve their performance. Analysis of the reforms the Italian higher education system implemented since 1990 should help the reader to contextualize the dynamics of the institutional and policy arrangements within which research, teaching, and an academic profession have developed.

Section 2 discusses evidence based on administrative data related to the determinants of career completion by university students. Italy is an interesting case study due to the relevance of students’ initial socio-economic conditions for academic achievement which is underlined by studies based on the OECD’s PISA (Programme for International Student Assessment) scores. Obtaining evidence about how family background affects a university career is difficult since students tend to be sorted into academic and vocational tracks at the secondary level. Females exhibit higher completion rates. Experiments have been conducted in local Italian universities to study the causal impact on drop-out rates of introducing extra-tutoring.

In Chapter “Do Financial Conditions Play a Role in University Dropout? New Evidence from Administrative Data” Contini and Zotti discuss the role played by economic conditions on student university careers in Italy. They use administrative data from the University of Turin – a large public institution in the North of Italy – and information on family background collected at matriculation to analyze how family economic conditions influence the probability of first-year dropout from university. While parents’ education and parents’ occupations have been shown to have a major effect on education outcomes for school-age children it seems that they do not have a sizable effect on university student drop-out. Instead, there is evidence that despite the progressive character of tuition fees and the existence of scholarships provided to low-income students, economic conditions do have a substantial impact on the likelihood of completing university studies. This suggests that current student aid policies in Italy are insufficient to close the gap that exists between high- and low-income students, and that increasing financial aid could be a tool for promoting equality of opportunity in education and eventually increase the share of young individuals with higher education degrees.

In Chapter “Drop-out Decisions in a Cohort of Italian Universities”, Atzeni et al. study the determinants of students’ drop-out decisions using data on a cohort of over 230,000 students enrolled in the Italian university system. The empirical analysis controls for course-of-study and university fixed effects, and shows that the probability of dropping out of university is correlated negatively with high-school grades and student age. However, it shows also that women have a lower propensity for drop-out especially among students enrolled on science, technology, engineering, and math where they are under-represented. Atzeni et al.’s data differentiate between students who leave home to enroll at university (off-site students) and students who continue to live in the family home (on-site students). They find that drop-out is significantly lower among off-site students. Self-selection into studying off-site is estimated using an instrumental variable approach to identify the causal relationship. The authors use detailed administrative data on students enrolled at the University of Sassari to investigate another self-selection channel affecting the estimation of the determinants of drop-out. They employ bivariate probit estimation to account for self-selection into the course of study, and show that estimates of the traditional determinants are modified. The unconditional comparison among degrees is misleading since some degrees attract more heterogeneously skilled and motivated students. While the estimation without selection suggests that women’s dropout rates are lower after accounting for selection, the contribution of women to the drop-out rate turns either positive or negative depending on the chosen study course.

Section 3 focuses on the increasing precariousness of an academic career especially for younger researchers. The two chapters in this section exploit longitudinal administrative data. They show that the standard transitions (PhD-postdoc-assistant professors-(tenured)associate professor) are discipline specific but also are gendered since job instability has different costs as women and men age (reflected in fertility decisions, conditional on obtaining tenure).

In Chapter “Is Enter to Italian Academia Getting Harder?” Checchi and Cicero consider the traditional steps in an academic career. While a doctoral degree is often considered the first necessary step only a small fraction of doctoral graduates (less than 10%) obtain an academic position within 6 years of degree award. Despite the absence of information on labor market outcomes, the authors focus on the determinants of this transition in order to study whether entry into an academic job is becoming more selective and/or more precarious. Merging three national administrative datasets on completed PhD degrees, postdoc collaborations, and new hiring into the academia they find a decline in appointment probability after the 2010 cohort, due to the effect of the hiring freeze imposed by fiscal austerity. They find also that combining a doctoral degree and postdoc experience increase successful application to academia. Women and foreign-born candidates are shown to be negatively discriminated, and there is evidence of career disadvantages for candidates from Southern universities.

In Chapter “Academic Careers and Fertility Decisions” De Paola et al. investigate how academic promotions affect the propensity of women academics to have a child. They use 2001–2018 administrative data on female assistant professors employed in Italian universities and estimate a model with individual fixed effects. They find that promotion to associate professor (a tenured position) increases the probability of having a child by 0.6 percentage points, which translates into a 12.5% increase at the mean. Results point in the same direction using a regression discontinuity design that exploits the eligibility requirements in terms of research productivity introduced in 2012 by the Italian National Scientific Qualification (NSQ) related to promotion to associate professor. Their study has important implications for policy by showing that reducing career uncertainty leads to increased fertility among academics.

Section 4 deals with methods designed to assess research productivity at a time when co-authorship and team production are becoming standard practice. Co-authorship and team working complicate the assessment of research quality, and the individual contribution of a research project and research output. The increased pressure to publish may induce the risk of excessive conformism in the choice of topics which can be mapped using text analysis. Gender issues may also matter since co-authorship, research networks, and research impact might not be gender neutral.

Chapter “Social Network Tools to Evaluate Individual and Group Scientific Performance” by De Stefano et al. analyzes patterns of scientific collaboration which recently has been considered an important driver of research innovation. Collaboration allows scientists to benefit from methodological and technological complementarities and synergies which can improve the quality and quantity of their research output. For example, collaboration among scientists has been shown to be increasing in all disciplines, and the rules governing international exchange programs are aimed at promoting collaboration among researchers. Collaboration among scientists can be mapped into networks and co-authorship linkages which makes social network analysis a useful theoretical and methodological approach. Several empirical studies identify a positive association between the researcher’s position in the co-authorship network and the individual researcher’s productivity, although the results differ depending on the discipline, scientific performance measure, and the data source used to construct the co-authorship network. De Stefano et al. propose the use of social SNA tools for scientific evaluation purposes. Network indices at the individual and subgroup levels are introduced to analyze the relation with both the individual research productivity and scientific output quality measures provided by the Italian academic researchers involved in the latest national evaluation of research quality (2011–2014).

In Chapter “Topic-Driven Detection and Analysis of Scholarly Data” Ferrara et al. present a mining approach to identify academic research topics based on the idea that research topics emerge through analysis of epistemological aspects of academic publications extracted from conventional publication metadata such as title, author-assigned keywords and abstract. The authors provide a conceptual analysis of research-topic profiling according to the behaviors/trends peculiar to a given topic along a considered time interval. They define a disciplined approach and related topic mining techniques based on the use of publication metadata and natural language processing tools. This approach can be applied to various topic analysis issues such as country-oriented and/or field-oriented research analysis based on scholarly publications. To assess the effectiveness of these techniques when applied to a real situation, the authors conduct a case-study analysis based on national and international data.

Section 5 synthesizes the discussions in Sects. 3 and 4 on student achievement and teacher careers driven by research assessment. By exploiting quality measures derived from research assessment exercises conducted every 5 years by ANVUR, Chapters “The Relationship Between Teaching and Research in the Italian University System”, “Degree-Level Determinants of University Student Performance”, and “Teaching Efficiency of Italian Universities: A Conditional Frontier Analysis” focus on the conditional correlation between student mobility and academic career completion. The relationship between these dimensions can be studied in terms of the joint production of academic services to enable the evaluation of university efficiency using frontier analysis.

In Chapter “The Relationship Between Teaching and Research in the Italian University System” Carillo et al. study the relationship between the quality of research and teaching in the Italian university system at the study program level. The authors use detailed data collected by the Italian ANVUR on undergraduate and master’s degrees offered by Italian universities in the academic year 2016–2017. Their cross-sectional econometric analysis shows a positive relationship between teaching quality and research performance that emerges when taking account of yardstick competition among study programs offered by the same department. The theory suggests that despite the trade-off between teaching and research faced by individual academics, in multi-unit universities which have implemented budget sharing based on research performance and the number of students, the negative relation between teaching and research is reduced or is counterbalanced. However, in the case of universities offering only a small number of study programs the teaching-research relationship is positive and stronger. The results are even more pronounced for master-level degrees where teaching is more aligned to individual research interests.

In Chapter “Degree-Level Determinants of University Student Performance” Bratti et al. use administrative data on higher education degrees in Italy during 2013–2018 to analyze the degree-level determinants of university student performance as measured by ANVUR quality indicators. After controlling for detailed degree subject–geographic macro area fixed effects, their analysis reveals several significant predictors of degree quality including access (i.e., selectivity), language of instruction, teaching body composition, percentage of teachers of core subjects, teachers’ research performance (for second-level degrees), and spatial competition.

The last chapter on “Teaching Efficiency in Italian Universities: A Conditional Frontier Analysis” by Mastromarco and al. presents a comparative analysis of the performance of Italian university teaching by evaluating the efficiency of heterogeneous faculty courses at the national level. According to OECD data, Italian public universities are under-funded: the costs related to individual Italian students are well below the OECD average. This underlines the importance for policymakers of information on the relative efficiency of universities which can be used as an indirect evaluation of how public funding is used. Chapter “Teaching Efficiency of Italian Universities: A Conditional Frontier Analysis” uses tools developed recently in the nonparametric efficiency frontier literature. The analysis is conducted at the national level and extends traditional analyses based on mono-dimensional indicators. The efficiency scores enabled by the statistical analysis are used to interpret current trends and changes to Italian universities’ teaching activities.

5 The Road Ahead

There are many other relevant topics that are not addressed in this volume. Public engagement and knowledge transfer are another university mission but information on these activities is scattered. However, it is being collected for the research evaluation that is currently underway. We also do not discuss the potential effects on teaching and research of the Italian National Recovery and Resilience Plan approved in July 2021 which assigns €5.4 billion to postsecondary education, slightly more than one-quarter of the total intended education budget. This plan is aimed at increasing tertiary education graduation rates, strengthening vocational education, and removing the financial obstacles to university enrolment.

The plan included a measure to raise the transition from upper secondary to higher education and to reduce university drop-out by providing more information on university careers, since the children of less-educated parents are more likely to lack confidence and knowledge about academic courses and careers. This measure is expected to increase school attendance, improve learning levels, increase university enrollment, and reduce the gender gap in university employment and participation in higher education in all fields.

At the same time, financial constraints and/or labor market opportunities also matter. For this reason, half a billion euros have been allocated to student scholarships and tuition exemptions. While empirical evidence that low graduation rates are caused by a lack of public financial support is limited, these measures will make Italy better aligned to other European countries and should promote mobility across universities in Europe. A further €1 billion has been allocated to student accommodation in Europe (to enhance student mobility) in a partnership with the private sector.³ Following the German model, the higher education system is expected to be extended to include vocationally oriented, non-academic tertiary education based on a planned investment of €1.5 billion.

To strengthen university autonomy in program design, and to increase the competition among universities in vocational training, several measures are being suggested to update study course curricula, create new cross-disciplinary programs, expand vocational training programs, and limiting the role of professional associations in the transition of graduates to the labor market.

The number of doctoral scholarships will increase from 10,000 to around 17,500 to try to increase the stock of human capital, with positive spillovers for innovation and R&D activities through partnerships with private companies and research centers, and reduce the doctoral graduation gap with European partners. Firms will be given incentives to recruit temporary junior researchers (20,000 over 3 years), and to establish research hubs and promote spin-off activity. To retain new doctoral graduates and avoid their migration to industry, additional funding will be provided for research programs led by young researchers. These incentives will be subject to gender quotas to encourage greater participation of women.

In summary, Italian policymakers plan to enhance the higher education system by promoting student mobility, providing new job opportunities for young researchers, and including new vocational programs. The former two objectives are discussed in this volume. It is hoped that these contributions will help aid to evaluating the success of the plan and the more efficient use of public resources.