Keywords

Over the last fifteen years, I have aimed to enter into the analysis of broader phenomena and processes by decoding the genesis and uses of quantification devices.1 The accounting reform of Chinese state enterprises, for instance, makes it possible to grasp the essence of the Chinese economic reforms of the 1990s (Eyraud, 1999, 2003). The construction and implementation of a system of performance-based management in French universities informs us about the profound transformations these organizations have undergone in the last two decades (Eyraud, 2014; Eyraud et al., 2011). As Alain Desrosières points out, these various studies suggest that “it is possible to look at the same time at social or political philosophies and seemingly technical tools, considering them as a totality” (Desrosières, 2000, p. 84).

However, my recent research on the transformation of French state accounting2 (Eyraud, 2013) has shown that an accounting system can be employed in relation to different, sometimes conflicting logics and objectives, for example, in attempts to show that the state is heavily indebted or not so much, to improve public management, or to encourage the outsourcing of public activities. The link between a specific type of quantification device and a particular social or political philosophy does not seem to be univocal; both do perhaps not constitute “a totality”. Hence, the nature and conditions of their linkage have to be questioned. I did this first through a renewed analysis of the French state accounting reform (Eyraud, 2016). This led me to put forward a grid for the analysis of quantification devices which seems capable, first, of casting light on the very nature of this link and, second, of making visible the possible choices involved.

The purpose of this chapter is to test this grid by revisiting our work on performance indicators for French universities. I choose thus to start from the analysis of a concrete quantification device—i.e. performance indicators—and conduct a kind of “archaeologic” analysis, using it as a lens for investigating and understanding the changes French universities have undergone since the mid-2000s. Drawing on the works on quantification done by French social scientists, such as Alain Desrosières (1988 with Thévenot; 1998 [1993], 2003‚ 2008a, b, 2014), Robert Salais (1986, 2004, 2010, 2016), Alain Supiot (2010, 2015) and Laurent Thévenot (1979, 1983, 1990, 2016), this paper seeks ultimately to enhance our understanding of reactivity.3

The analytical grid I propose distinguishes between three different levels that exist within a quantification device, each of which is examined in a separate section of this chapter. First, there is what might be called the bedrock level: a quantification device is grounded in a founding vision that is generally congruent with a particular form of state or economic system. Second, there is what might be called the intermediate level: a quantification device contains a conception of the objectives and “raisons d’être” of the entity that is quantified. Third, there is the level that relates to the micro-conventions of calculation: philosophies can be hidden at this microscopic level and give a particular orientation to the device. However, the analysis of these three levels does not tell us everything about the orientation of the device and the effects it can produce; the device is part of a larger configuration, the context of its deployment and its uses have thus to be examined. That constitutes the fourth dimension of the analytical grid, which will be examined in the fourth section of this paper. Returning to the question of the device as a totality, the conclusion will show that this specific device is not a very integrated assemblage, which explains its real but at the same time also limited effects.

The analysis is based, first, on in-depth interviews with government and university officials, second, on the study of official documents, reports and archival materials from parliament, central government, the Ministry of Higher Education and several universities, and, third, on participant observation. I conducted participant observations in my role as a member of the Governing Board and Finance Committee at my own university. Further, I was a special adviser to the President of my university on performance management systems. I attended the training on performance indicators and performance management for universities provided by the Ministry, and I participated in the implementation of the performance indicators in my own university. This variety of materials allowed me to follow the numbers from their birth through their very detailed construction process to their concrete uses.

The Bedrock: NPM, LOLF

Performance indicators are the latest form of public statistics. The development of their current form and uses began in some European countries, such as Sweden or the United Kingdom, in the late 1980s, and was a part of much a broader phenomenon: the rise of New Public Management (NPM). This term is quite ambiguous, even stretchy: it is used to speak about government reforms implemented in Great Britain during the Thatcher government, in the United States during the Reagan and Clinton administrations, in the Netherlands under a Christian-Democrat government, or in Sweden and New Zealand under Labour governments. All these reforms have a number of common features but also many differences (Hood, 1995). The shift in doctrines of public accountability is part of their common ground. Before NPM, democratic accountability depended on limiting corruption, waste and incompetence in public administration. To this end, the public sector was kept sharply distinct from the private sector in terms of ethos, methods of management, organizational design, people and career structure. An elaborate system of procedural rules was designed to prevent favouritism and corruption. In contrast NPM involved a very different conception of public accountability, with different patterns of trust and distrust. As Hood (1995, p. 94) writes:

The basis of NPM lies in reversing the two cardinal doctrines of public administration; that is, lessening or removing differences between the public and the private sector and shifting the emphasis from process accountability towards a greater element of accountability in terms of results. Accounting was to be a key element in this new conception of accountability, since it reflected high trust in the market and private business methods and low trust in public servants and professionals.

In this process, the introduction of the “Loi organique relative aux lois de finances” (henceforth referred to as “LOLF”) was in France a very important step. This law, passed in 2001 and taking effect in 2006, introduced performance management and private accounting to the state and public services. The first objective of the LOLF was to make the government and the public services accountable to parliament for the results of their actions, and to give more power to parliament over budgetary policies and choices. Since 2006, French MPs have two new documents for the budget debate: an annual performance plan and an annual performance report for each public policy. The first one determines the objectives for the following year; quantified indicators (known as performance indicators) are used to quantify these objectives and set the targets which have to be reached. The second one gives an account of the results achieved (relative success or relative failure) over the last few years. National performance indicators were set at each level of government and for all public bodies, so that a performance-oriented form of management was introduced throughout the public services and public administration, which was the second objective of the law.

A performance measurement system for public policies is based on the idea that the state is accountable for the results of its actions, which is based on, as Hood highlighted, a specific conception of public accountability which emerged from the 1980s, and which now prevails.4 This is what might be called the bedrock level of quantification: a quantification device, such as a performance indicator, is grounded in a founding vision, an ontology.5 It would be possible to go deeper and view this conception of public accountability as being rooted in a specific way of governing that might be called “government by objectives” (Thévenot, 2015).6

From these specific conceptions of governing and accountability, the French government made several choices. First, in probably a typically French way, a law (here the LOLF) was used to establish the system of performance management, using a standardized, centralized, top-down approach. Second, the LOLF speaks of “performance” and not of “results”. At the beginning of the 2000s, this was the choice made by many governments (such as the UK and the US), but not all (see Canada, for example). Whereas “results” can be seen as a quite a neutral word, the notion of “performance” carries many connotations (Jany-Catrice, 2016): ideas of outstanding qualities, of achievement (specifically in sports), of excellence, of winning; the notion of competition being not so far from them. Third, the LOLF chose to assess public performance only by means of quantitative indicators, relying on the belief that only numbers are able to report on public action and social reality, and, hence, demonstrating a “trust in numbers” (Porter, 1995).

A quantification device is thus the bearer of some great fundamentals: here, it implies a specific way of governing (government by objectives), a particular conception of public accountability, a standardized, centralized and top-down approach, a focus on performance, and a specific way of assessing public performance (via quantitative indicators) which is based on a belief in numbers.

Performance Indicators for French Universities: What Are Their Raisons D’être?

The formulation of the first annual performance plans introduced by the LOLF involved designing a set of performance indicators. However, what is the “performance” of a public policy? What do we expect from a prison or from a school? What does a “well-performing university” do? Do we expect it to produce graduates adapted to the labour market and fitting the needs of private companies? Do we expect it to allow women and men from different social origins to attain the same levels of education? Do we expect it produce a lot of patents that enhance the competitiveness of national companies? Do we expect it to broaden human knowledge? Do we expect it to provide a fulfilling and motivating working environment for its staff and students? Measuring the performance of a public institution is clearly built on a system of values, and it involves the making of fundamental, societal choices on what is important to measure.

In 2006, the first annual performance plan for higher education and research specified 33 indicators for universities with targets to be attained within five years. Where did these indicators, which French universities and the French Ministry of Education were accountable to Parliament for, come from? The LOLF could have been an opportunity for large democratic debate on the outputs and outcomes we expect from our public policies and public services. However, in my view, the LOLF was a missed opportunity for democracy, the choices of indicators were made in a completely technocratic way: they were the result of discussions and negotiations (from 2003 to 2005) between the ministry in charge of the public policy in question (here the Ministry of Education, Higher Education and Research, which I will refer to as the Ministry of Education), on the one hand, and the Treasury, on the other. Surprisingly, the discussions between the two ministries were not about the expected outcomes of higher education, but directly focussed on the indicators themselves: one ministry proposing certain indicators, which were then very often refused by the other.

To prevent the application of a “realist epistemology” to numbers, Desrosières suggested to talk not about “measurement” but about the “quantifying process”. As he put it:

The use of the verb ‘to measure’ is misleading because it overshadows the conventions at the foundation of quantification. The verb ‘quantify’, in its transitive form (make into a number, put a figure on, numericize), presupposes that a series of prior equivalence conventions has been developed and made explicit […]. Measurement, strictly understood, comes afterwards […]. From this viewpoint, quantification splits into two moments: convention and measurement. (Desrosières 2008b, pp. 10–11)

In our case, that means that the ministries argued about measurements before agreeing on what the performance of higher education is. We can analyse these controversies through the interviews I conducted with officials of both ministries and try to understand the rationales of each.

For the Ministry of Education, the indicators served various objectives. Firstly, the Ministry conceived them as incentives for universities to act in a certain way. As a senior official in this Ministry put it:

There is a political will behind each indicator; we would like each indicator to be an incentive for universities to be better in a certain field or to develop in a certain way.

The indicators, then, are seen as signals.

Secondly, the Ministry thought about how the indicators could be used during its negotiations of the education budget with the Treasury. The Ministry proposed for example the following indicators: cost per student, student–teacher and student–administrative staff ratios, all of which are quite low in France compared to other OECD countries with similar GDP per inhabitant. These indicators would allow the Ministry to justify an increase of its budget, in particular in relation to staff numbers. Thirdly, the Ministry wanted to present a positive image of higher education. This can be seen in several interviews, where interviewees for example said:

We’re not masochists. You try to have indicators which can only improve; you do not want to be shot for it.

We wanted to show what was working well.

In contrast, the Treasury’s main concern was about its uses of the indicators during budgetary negotiations with ministries. It was very aware of their potential use as devices to justify the need of enhanced income and staff numbers. See, for example, the following comment from a senior Treasury official:

We absolutely did not want indicators which allow our counterparts to say ‘to get better results on this indicator, we need more money, more teachers, more premises, more computers or anything else’.

The Treasury refused, for this reason, nearly all the indicators that were put forward by the Ministry of Education. Contrary to the third objective of the Ministry of Education, indicators which presented a negative image of the Ministry to the public, which depreciated or undermined the value of its activities, would turn into a useful weapon for the Treasury. They allowed it to be in a stronger position to negotiate the education budget. Several quotations from the Ministry of Education showed that the Treasury was pressing for “negative” indicators:

The Treasury wanted to impose figures that you can find in tabloid headlines, such as failure rate, dropout rate and so on, a negative picture in fact.

The next section will show that the very limitations of the indicators were exploited in this struggle.

By analysing not only the controversies around the indicators but also the final indicators themselves, we can now try to make visible the value systems on which they are based. The works of Boltanski and Thévenot (2006 [1991]) and Boltanski and Chiapello (2007) are useful for that. To analyse disputes and controversies, Boltanski and Thévenot (2006) identified seven “orders of worth” which imply systematic and coherent principles of evaluation, justification and legitimacy. Each of these “orders” (or “worlds”) gives importance to different values:

  • The inspired world values imagination and creation.

  • The domestic world values tradition, long-term relationships and the respect of hierarchies.

  • The fame world values celebrity and public opinion.

  • The civic world values collective interest, solidarity, equality and democracy.

  • The market world values competition and the exchange of goods and services on a profit basis.

  • The industrial world values efficiency, productivity and technical competences.

  • The projective world, which features prominently in the New Spirit of Capitalism (Boltanski & Chiapello, 2007), values flexibility, mobility, attractiveness and networks.

It is possible to link each of the indicators (some refer to teaching, some refer to research activities) of the annual performance plan for higher education to a specific world; only four of the above listed worlds are relevant for the purposes of our analysis. For each of these worlds, several topics and associated indicators can be identified. For the industrial world, for example: production volume (percentage of people with a university degree); production “failure” (non-completion rates); lead time (rate of PhD students defending their PhD thesis within three years); and efficiency (percentage of university building capacity in use). For the market world: revenues (percentage of revenues coming from intellectual property rights); competitiveness (share of world scientific publications, percentage of patents deposited by universities); and inclusion in the labour market (employment rate of graduates three years after graduating). For the projective world: attractiveness (percentage of foreigners among masters, doctoral and postdoctoral students, and academics); visibility (two-year citation impact); and networks (rate of participation in European Framework Programmes). For the civic world: fairness (ratio between foreign and home students’ success rates).7

In summary, as far as teaching is concerned, the industrial world (with eleven indicators) is dominant, whereas the projective (four indicators), market and civic worlds (three indicators each) are present but in much weaker form. Especially the indicators related to the civic world are quantitatively weak but also qualitatively poor: the indicators chosen do not send out a strong political signal, for example towards widening participation and democratizing higher education. With regards to the indicators for research, six of them can be linked to the market world and five to the projective world. It is ultimately a widely economics-based idea of performance that emerges from the chosen indicators, focussing on revenues, efficiency, competitiveness and insertion in the labour market. Some of the indicators, especially those related to research, also focus on attractiveness and networking, being thus closely akin to “knowledge economy” theories. On the other hand, the political and civic dimensions are not very pronounced at all.

The Upper Stratum: The Micro-Conventions of Calculation

There is a large number of possible choices available at the most granular, even microscopic, level of each calculation. These choices have also been debated by the ministries. The Ministry of Education and the Conférence des Présidents d’Universités (CPU) [Association of University Presidents] often preferred indicators expressed in absolute terms, which they consider better for showing the high activity levels and social usefulness of higher education. The following excerpt from a letter from the CPU to the Secretary of Higher Education clearly illustrates this:

Our general analysis of the indicators put forward [by the Treasury] is that several of them are disadvantageous, are negative for universities. While French universities have to accept all the students who have passed the baccalauréat, they will be judged on qualitative criteria, such as success rates within a certain period of time (three years for undergraduates and PhD students for example). It is absolutely necessary that universities can also show quantitative results, such as the number (that is in absolute terms) of graduates they have trained.

By contrast, the Treasury systematically refused indicators expressed in absolute terms, and even refused their relative expression for a temporal analysis, such as growth rate. Let us consider, for instance, the indicator aiming to measure the objective of “Producing scientific knowledge at the best international level”. The two ministries struggled fiercely over it. The Ministry of Education promoted the number or the growth rate of French publications in internationally recognized journals, whereas the Treasury promoted the percentage of French publications in internationally recognized journals.8 The latter was finally chosen, so the Treasury won the battle on this indicator.

If we go back to the “orders of worth” analysis presented above, it is possible to regard the absolute measure as related to the industrial world, underlining production volume and the increase of this production volume; it is rooted in a productivity-based perspective. The relative measure can be seen as linked to the market world and being rooted in a market-based perspective, all the more so because the term “market share” is used in several speeches of Treasury officials, as well as many of the interviews I conducted with them. Furthermore, the latter choice allows comparisons between countries, and between universities or research centres. This choice, as Desrosières stated, “creates a new world in relation to which everyone has to position himself” (Desrosières, 2008b, p. 15); it makes benchmarking and ranking possible, and it makes it possible to put under pressure the universities or research centres which are at the bottom of the league (see also Dixon & Hood, 2016). As Ozga wrote: “Comparison defines the new mode of governance […]. Comparison is war by other means” (Ozga, 2008, p. 268).

Even the limitations of the indicators can be used for this war, as a member of the Ministry of Education put it:

The Treasury really put pressure on us to calculate some of the indicators in a certain way. For example, we had to fight really hard to make sure that the rate of PhD students defending their thesis within three years should take into account students who have their viva before the 31st of December and not only before the beginning of the new academic year in September. This would change the result by more than 20 percentage points.9

The determination of the targets to be reached in the mid-term constitutes further evidence of this struggle between positive and negative pictures. The Ministry of Education wanted them to be set at an achievable level, the Treasury wanted them to be as high as possible. Furthermore, this negative picture tarnishes the image and perception of the public service in question in the eyes of MPs and in the eyes of the public. Indicators can then be used to justify reforms: this sector is doing badly, hence new policies are needed. For instance, the percentage of French publications in internationally recognized journals has been the most widely publicized indicator when talking about the quality of French research, fuelling alarmist discourses. Nicolas Sarkozy used this indicator in his speech of 22 January 2009 to legitimate the implementation of the Law on Liberty and Responsibility of Universities (the LRU) which was passed in 2007, but denounced by a great part of the French academia.

To sum up, the analysis of the construction process of the indicators picked up three elements: the indicators conceived as signals, the development of comparability and an economics-based idea of performance. This confirms Desrosières’s (2008a) analysis. “Markets, incentives, benchmarks and rankings” have been, since the 1980s, “new and increasing features of public statistics” (Desrosières, 2008a, p. 112). But these indicators are only “loosely linked to each other” (Miller, 1992, p. 84). This way of developing quite an inconsistent set of indicators seems a specificity of NPM, considering that each field of social reality has its own dynamic separated from the others. In contrast, macroeconomic or national accounting aggregates are a very different kind of statistics, highly interconnected and based on a conception of the economy as a whole entity.

The analysis has also shown that a quantification device, such as a performance indicator, is the result, down to its smallest detail, of power struggles between the actors involved. Hence, “the moment of indicator design is a defining moment which will shape the future” (Desrosières, 2014, p. 47), and it is therefore a moment particularly important to analyse.

The Life of the Device: Context, Uses and Developments

To understand the orientation of a quantification device and the effects it can produce, the context of its deployment and its uses have also to be examined.

2006–2012

The development of performance indicators is still at a relatively early stage in France. We must remember that one of the main objectives of the LOLF was to make the government and the public services more accountable to parliament for the outcomes of their actions. The performance indicators were supposed to be used by the MPs during the budget debate. However, all the reports and speeches during the national budget debates 10 were based on budget figures, describing the evolution of each policy measure; and the discussions among MPs turned to the relevance of these budget decisions. Performance indicators were not used at all in these debates. Two reasons at least can explain that. Firstly, the LOLF did not link performance to funding. As Lambert and Migaud, two MPs and fathers of the LOLF, pointed out:

Managing a public entity is not the same as managing a company. For the state, there is no direct link between the level of budget funds and the objectives to be achieved. To decide the level of appropriations, the notion of needs will remain the most important. It is thus possible that achievements will have no budgetary impact. (Lambert & Migaud, 2006, pp. 13–14)

Secondly, the indicators are rather meaningless for the MPs. They are quite technical. Because of the objective of consistency, the indicators have changed little since 2006, and the MPs, as was mentioned, did not participate in their construction. MPs do not find the indicators meaningful in relation to public policy making. Since they discuss policy options and consider to be at the heart of political choices budgetary decisions, they use budget figures. Furthermore, the budget debate is conceived to be more of a debate between MPs from different political parties, rather than an exchange between parliament and public administration.

As shown above, within the government, the Treasury and the Ministry of Education anticipated, while negotiating the indicators, that these might be used during the budgetary negotiations. But the performance indicators were not used at that moment either; as a senior official from the Treasury put it:

We were afraid that the different ministries would use the performance indicators to ask for an increased budget—that is the reason why we were so tough during the discussions about these indicators. But in fact, nobody uses them during the discussions of the ministries’ budgets; we speak about money, about staff, about policies, not about performance, neither them nor us.

The performance indicators were, in the end, not used in the two situations where they were expected to be used. However, they were going to be very powerful in a different, unexpected way. A French state reform, called the “General Revision of Public Policies” (RGPP), launched by Nicolas Sarkozy in 2007 immediately after his election, initiated a new usage of them: performance indicators became tools for resource allocation from the Ministry of Education to universities. To make this understandable, we must briefly explain the history of the French university funding system as summarized in Table 9.1.

Table 9.1 French university funding system (from mid-1980s to 2012) (compiled by the author)
Full size table

The new system of resource allocation, called “Sympa” (which can be translated as “cool”) resulted from discussions between the Department of Higher Education of the Ministry of Education, the parliament and the Conférence des Présidents d’Universités (CPU) [Association of University Presidents]. The system had two parts: one depending on activities (80%) and one depending on performance achievements (20%). Although, this does not seem to be a big change compared to the previous system, it actually was, as the newly introduced activities criterion had also a performance dimension. It is now no longer merely the number of students registered at the beginning of the year, but the number of students who sit the exams, and it is no longer the number of staff, but the number of “publishing academics” that counts (and non-publishing academics hamper the performance of their own university). If one incorporates this last criterion into the “performance share”, the share makes up more than 50% of the budget of most universities,11 although it is the 80–20 ratio which has been taken up in the parliamentary reports and documents and speeches produced by the ministries and CPU.

Although I have not examined the negotiations and controversies behind the production of the new resource allocation system, we can analyse its indicators. Some of these are derived from the LOLF, however, at least four different rationales can be noticed: first, a pure performance logic awarding good and bad marks (counting the number of publishing academics, awarding grades to research centres); second, an attempt to consider social and cultural inequalities (for example, by weighting the success rate by the grant holders’ ratio)12 driven by equity concerns; third, the acknowledgement of a university’s activity (for instance the number of Master degrees and PhD degrees delivered, so an indicator in absolute terms as the CPU asked for the LOLF indicators); fourth, a weighting by fields, which is a remainder from the first resource allocation model (see also Table 9.1).

Quite technical and poorly publicized, the Sympa system received very little attention beyond the circles of the CPU, Ministry of Education and some well-informed MPs. These parties welcomed the new resource allocation system for several reasons. First, it was the result of negotiations in which these parties had been involved, and different rationales had been taken into account. Second, it was deemed to be an objective system based on clear criteria which replaced the 20–30% of contractual resources which previously had to be negotiated between a university’s management team and the Ministry. Third, besides its performance dimension, it was supposed to make visible inequalities existing between universities and, hence, could help address and reduce these. Fourth, parallel to the introduction of the new system, the government committed to a general increase of the higher education budget; so each university was supposed to benefit from Sympa, and under-resourced ones were supposed to benefit more.

However, it quickly became apparent that things should turn out very differently. The Ministry decided to increase the budget by far less than originally promised, and it also decided not to put the increase into Sympa’s envelope, but a separate “Undergraduate Success Programme” (Plan Licence), which should become a key measure for the government. In fact, the Plan Licence is still attached to the name of Valérie Pécresse, then Minister of Higher Education. The Plan Licence was allocated to each university without using the Sympa’s criteria, and through negotiations with each university. As a staff member of the Ministry of Education put it:

Actually, the Ministry wanted to keep the power in its hands, at least what it thought was power; it wanted to have something on the table. And to be able to use it to encourage universities to apply voluntarily some new regulations (for example encourage them to merge). The result is: it was the tougher university president, the one with political support, the one who was president of an already well known university, etc. who earned the most.

Since the Sympa’s envelope remained steady, gains for some meant losses for others. The CPU, on the other hand, refused the notion of an “over-resourced” university and rejected any redeployment of resources between universities. Because of the decision of the Ministry and the position of the CPU,13 the possibility of reducing inequalities faded away, while the effects of Sympa’s performance dimension became quickly evident.

Performance is now financially rewarded, while non-performance is financially punished. In this context, having indicators which take into account the inputs or the conditions of teaching and research is very important. Performance can be linked with the academic level of the students, or with good working conditions, such as the number of administrative staff, which allows academics to do less administrative work and to have more time for research. In the same way, non-performance can be linked to “low” inputs and difficult working conditions, something which Sympa’s weighting tried to take into account, but now the punishment is only going to make worse.

Merton already revealed this phenomenon in the 1960s, and he called it the “Matthew effect” (Merton, 1968) referring to the following passage in the gospel of Matthew: “For everyone who has will be given more, and he will have an abundance. Whoever does not have, even what he has will be taken from him” (New Testament, Matthew 25:29). We can also speak of “cumulative advantages”: a favourable relative position becomes a resource that produces further relative gains, so the rich get richer at a rate that makes the poor become relatively poorer. And the richest universities are generally the ones where most of the students come from privileged social backgrounds. So, performance-based financing, which is a frequent component of New Public Management, leads to a concentration of resources around those who already have the most,14 which has significant implications for our conceptions of equality and justice. Suleiman already noted in 2003 that a lot of the proposals coming from New Public Management theories “have little to do with bureaucracy in itself and much to do with the distribution of public resources” (Suleiman, 2003, p. 20). But often this political dimension is hidden: “The allocation of resources seems to result instead from the dynamism and the quality of individuals and institutions” (Le Galès & Scott, 2010, p. 132).

Generally speaking, this way of funding introduces competition between universities: once the total budget for higher education is decided, the fact that some universities have financial rewards, and so more money, naturally means that other universities will have less. Competition takes place at the heart of performance-based financing systems. Furthermore, these “Sympa indicators” have impacted universities’ policies. Sympa’s weighting pattern has indeed had an incentive effect: an increase of five hundred students who pass their degree (for the same number of students passing the exams) would generate an additional budget of €80,000, while a 2.5% increase of the number of publishing academics would mean an additional budget of €474,000. In this context, a lot of universities chose to redirect financial and human resources from teaching, not towards marketing, as Espeland and Sauder (2007) observed for American Law Schools, but towards research. Furthermore, it is advantageous for universities to recruit new academics for research centres ranked A + or A, and not for the ones graded B or C. Lastly, a lot of universities chose not to hire research officers, but to recruit only university lecturers and professors, because the activity of the former is made invisible by Sympa, although the potential lack of research officers does not bode well for the development of science.

In addition, more and more universities decided to introduce a performance-based resource allocation system internally, especially for their research centres. They introduced a variable part for their budget, for example 15%, based on performance indicators, such as the ratio of publishing academics and the grade of the research centre (which is partly based on the same ratio). But most often the management teams of the research centres do not know anything about the criteria on which the allocation of the remaining 85% is based. They are rather committed to the variable component, arguing that at least here the criteria are known and clear, and that it is a more transparent and fairer way of allocating resources, not depending on personal relationships and lobbying. With this system, they know what they have to improve and so they have the feeling of being able to contribute to the sound management of public money.

But, this link between performance and funding also produced what can be called “punitive practices”: some research centres excluded the non-publishing academics (to increase the ratio of publishing ones); some decided not to pay for the costs when a non-publishing academic gave a presentation at a conference. The presence of such practices depends largely on the disciplines; they are quite rare in the social sciences but quite common in economics and management,15 all the more so because the criteria used to decide whether one is publishing or non-publishing are, as will be discussed later, more strictly defined in economics and management than in the social sciences and the humanities.

This way of governing public services by financial incentives is really new in France. The relationships within the state and public services (and between them) are thought of in terms of microeconomic theory, specifically agency theory: “Society is viewed as a system of essentially self-interested ‘elementary particles’” (Supiot, 2015, p. 216); institutional and individual actors are thus thought of in terms of homo economicus, whose actions can be driven and controlled through a system of punishments and rewards. This is often the basis of management by objectives in the private sector and in the context of New Public Management. Vinokur (2008) summarized these changes when talking about a shift from the model of “obligation of means + trust” (the obligation of means resulting in a bureaucratic ex ante form of control, and the trust in job security for civil servants) to a model of “obligations of results + distrust”. We already highlighted the “low trust in public servants and professionals” when referring to Hood (1995, p. 94), which accompanied the rise of NPM and its new conception of accountability.

Finally, the performance indicators have widely replaced, within universities, the previous statistics they produced internally for a better understanding of their students. Performance indicators, and specifically those used by Sympa, became the dominant metrics compared to the statistics produced for acquiring knowledge about students’ characteristics. This trend, which can also be found in health care and the social services,16 is part of “the shift away from the social welfare state as guarantor of basic solidarities and rights, access and treatment for all, to the state as a provider of services” (Jany-Catrice, 2016, p. 129).

Since 2012

In 2012, the resource allocation model of Sympa was abandoned by the new French government set up under the presidency of François Hollande. This demise has to be linked to several protests from students and academia from 2007 onwards. As mentioned before, the government, set up in May 2007 under the presidency of Nicolas Sarkozy, had immediately passed a new law on Liberty and Responsibility of Universities (the LRU Law).17 This law led to great protests, strikes and demonstrations, from 2007 to 2009. Initially, the protests were against the new law as a whole; later (in 2009), they partly focused on the new status introduced for university lecturers and professors by the new law, under the so-called “modulation des services”. Here, those who were classified as non-publishing might be given, by the president of their university, more teaching. Opponents to the reforms widely criticized the quantitative evaluation of the research centres and individuals, which led to centres being ranked and academics categorized as publishing or non-publishing.

The AERES (Agence d’Évaluation de la Recherche et de l’Enseignement supérieur), whose setting up in 2006 was already controversial, was in charge of these assessments. It developed, during the summer of 2008, the criteria that defined an academic as publishing or non-publishing, and it designed a ranking of scientific journals for each discipline: an academic was considered as publishing, if s/he had published two articles in a journal ranked A or B during the past four years. In the fall of 2008, many petitions emerged signed by academics, academic professional organizations and trade unions, scientific committees and editorial boards of academic journals; some of these petitions were against the ranking system, some against the priority given to publication in journals at the expense of other scientific activities (including the publishing of books), and some were against forms of quantitative evaluation all together. In response to these, in October 2008, the AERES allowed the academic disciplines themselves to identify a list of scientific journals without ranking them, and it made it possible to count books and book chapters as publications (at the discretion of the respective assessors). In 2009, AERES published new lists of journals, and while nearly all the social sciences and humanities decided not to rank the journals listed by them, economics and management studies did and still do so.

To bring the strike against the new status for lecturers and professors to an end, the government added that non-publishing academics might be given more teaching by the university president only if the academics concerned agreed. Critics of the ranking of research centres arose again a few years later, leading to new petitions in 2011. In response to all these protests, the Socialist Party committed during the presidential campaign (2012) to organize a National Conference on Higher Education and Research. François Hollande was elected, and the National Conference was held in November 2012. The previous system of assessment was, among other things, widely criticized during the Conference, and the participants proposed that research centres should no longer be graded and ranked. A working group, comprising representatives of the Ministry and the CPU, was set up in April 2013 to re-examine the Sympa model. At the beginning of 2014, the AERES replaced the grades with a “textual appreciation” of research centres, which made Sympa obsolete. The working group was supposed to produce a new resource allocation model by the end of 2014, but the Ministry and the CPU did not manage to agree, and the working group stopped working in 2015.

From the above, four sticking points can be at least identified. First, the Ministry conceived Sympa as a decision-support tool, providing it with some room for manoeuvre and negotiations. In contrast, the CPU wanted the system to automatically calculate the budget of each university. Second, the Ministry would have liked to include the payroll in Sympa, but the CPU refused. Third, there was disagreement about the indicators themselves. The negotiating bodies seemed to agree on replacing the number of publishing academics by the number of academics, but they disagreed on the performance indicator for research that would replace the grades of research centres. The Ministry proposed two indicators, one measuring the participation in European Framework Programmes, and one based on the number or the percentage of Institut Universitaire de France laureates. The CPU questioned the method of calculation of the first and refused also the second arguing that it measured individual performance but did not evaluate the collective performance of a research centre or a university as a whole. Finally, a controversy about the weighting factors by field led to the conduct of a cost analysis of teaching and research in order to base the factors on objective information. As a result, the decision-making process on the budget allocated by the Ministry to each university became, contrary to what was initially intended, even less transparent.

Conclusion

This chapter has traced the life of performance indicators in French higher education from their birth in the beginning of the 2000s to the end of the 2010s. Analytically, it distinguished between three levels that make up a quantification device, such as performance indicators: (a) the bedrock or the ontology of the device; (b) the intermediate level made up of the conceptions about the “raisons d’être” of the quantified entity; and (c) the upper stratum comprising the micro-conventions of calculation, and it analyse the context in which the indicators were deployed. This study has shown how relevant it is to enter into the analysis of a specific field by decoding the genesis and uses of its quantification devices.18 As Salais has highlighted: “The choice of the indicators, the construction of data and their uses reveal the normative assumption of the policies” (Salais, 2004, p. 298). The three levels and the context are the result of socio-historical processes in which different social actors participate, bearing different philosophies or value systems. These processes can lead to great coherence between levels and context. But they can also lead to a weakly integrated device that exhibits many “gaps”. As Kurunmäki, Mennicken and Miller, drawing on Deleuze and Guattari’s works, put it: “The unity of such assemblages derives only from the co-functioning of their components; the relations that are formed among them” (Kurunmäki et al., 2016, p. 399). We can try now to capture the salient points of this assemblage.

This assemblage is based on the idea that the state is accountable for the results of its action, which is a specific conception of public accountability which emerged from the 1980s and which now prevails. The state chose to assess results by, and only by, quantitative indicators, which reveals a belief in numbers and a “realist epistemology”. From these starting points, this assemblage became economics-based and competition-oriented. Firstly, by choosing the vocabulary of “performance” (rather than the more neutral term “results”). “Performance” carries many connotations and introduces the notion of competition. Furthermore, the various negotiations between different parts of public administration led to a widely economics-based idea of performance focussed on revenues, efficiency, competitiveness, insertion into the labour market, attractiveness and networking, being thus closely akin to “knowledge economy” theories. On the other hand, political and civic dimensions of governing were almost absent. The chosen indicators and the way they are calculated made comparisons, benchmarking and ranking possible; they came to be conceived as signals towards universities. A performance-based financing system was implemented that introduced financial rewards and punishments, which was strengthened by some of the Sympa indicators, such as the number of publishing academics and the grades of the research centres. This system led to the production of “Matthew effects” and it introduced competition between universities. Because of Sympa’s weighting pattern, it encouraged universities to redirect financial and human resources from teaching towards research. Together with the “modulation des services”, it made the development of punitive practices against individuals possible. This was a new way of governing public services in France: the relationships within the state and public services (and between them) became thought of in terms of microeconomic theory, and specifically within the framework of the agency theory; collective and individual actors were thought of in terms of homo economicus. All of these elements are quite coherent.

However, the LOLF did not, at its beginning, link performance to funding, thus there was no performance-based funding system at the national or ministerial levels. The indicators resulted from discussions and negotiations; they resulted from power dynamics and different strategies which introduced different logics. The present analysis has tried “to disentangle such multiplicities” (Kurunmäki et al., 2016, p. 397) of rationales. Aside from the competitive logic, there was also strong support for a transparent and automatic funding system that would be able to counteract nepotism and arbitrariness (and at the same time a refusal from the Ministry of such a transparent and automatic system). Porter (1995) and Supiot (2015) showed that quantification devices are also an essential part of “government by rules” and of democracy. There was a strong demand for a funding system based on the needs of the universities, more than on their results; a well-informed MP, for example, welcomed in 2014 the change of indicator from the number of publishing academics to the number of academics. There was finally a will to take into account the social and cultural inequalities in order to reduce them.

Furthermore, several protest movements within academia have strongly criticized the rationale of competition, the quantitative mode of evaluation, the definition and use (modulation des services) of the status of “publishing academic”, the rankings of academic journals, the grading of research centres and the punishing practices. These movements were victorious in some respect: the modulation des services is now only possible if the non-publishing academic agrees with the ruling; the grades for research centres have been abolished; the publication of books and book chapters can be taken into account for the social sciences and humanities; and journals are no longer ranked in these disciplines. These alterations have reduced the impact of the competitive and punitive dimensions of the device and explain why, together with the tenure and status of “fonctionnaire” for the great majority of academics,19 it has become, at least in the social sciences and humanities, a very limited “engine of anxiety” (Espeland & Sauder, 2016).

In summary, the reforms studied here, and the quantification devices implied, do not form a very integrated assemblage. This explains why the changes that the performance indicators were supposed to help bring about were at the same time significant and limited, in comparison with what may have occurred in the UK higher education system for example (see also Eyraud, 2016).

The three-part analytical distinction introduced in this chapter, combined with an analysis of the context in which a quantification device, such as a performance indicator, is deployed, makes visible the very broad range of possible options. One can agree with the idea that the state and the public services must be accountable to citizens, regarding it as a significant democratic progress. But this does not mean that only numbers are able to report on public action and social reality; being accountable is not just about reporting numbers. As Supiot pointed out: “To confuse measurement and assessment inevitably dooms us to lose our sense of proportion; assessment is not only measurement; assessment requires that the measurement is referred to a value-based judgement which gives it meaning” (Supiot, 2010, p. 82). De Gaulejac went even further: “We should abandon the economist, objectivist and mathematical conception of assessment and adopt a qualitative, democratic and dynamic one” (De Gaulejac, 2012, p. 77). Indeed, the very idea of measuring performance by indicators should be questioned, since there may exist other preferable ways of assessing public services.

However, even if quantitative evaluation should not be the only way, quantitative indicators can be useful in the process. In that case, the starting point, from which the indicators emerge, should be a wide-ranging public debate on what is expected from public policies. As Gadrey (1996)proposed, Boltanski and Thévenot’s (2006) “orders of worth” can help to specify the different expectations placed on indicators. Furthermore, detailed definitions of the indicators should not be a way of surreptitiously encapsulating values and hierarchies. These micro-conventions of calculation should be drawn up with an aim of impartiality, and when a choice is needed it should be made democratically in order to construct what could be called a “shared objectivity”. Finally, the uses of the indicators should also be carefully scrutinized and debated. The decision to base funding on performance indicators has powerful effects, such as an increase in inequality. The analytical grid proposed here provides a blueprint for building “what should be, in our view, a satisfying process of quantification” (Salais, 2016, p. 133).20

Notes

  1. 1.

    I use the term “device” referring to Foucault’s concept of “dispositif”. Foucault defines dispositif as “a thoroughly heterogeneous ensemble consisting of discourses, institutions, architectural forms, regulatory decisions, laws, administrative measures, scientific statements, philosophical, moral and philanthropic propositions—in short, the said as much as the unsaid (…). The dispositif itself is the system of relations that can be established between these elements” (Foucault, 1994, p. 299). The term thus emphasizes the complex and varied nature and the systemic dimension of these “ensembles”. I have chosen to use the notion of “device”, even if the English translation of Foucault’s “dispositive” has given rise to extensive discussions. Some of the published translations retain the term in French; others opt for various solutions such as “apparatus”, “device”, “arrangement”, etc.

  2. 2.

    In 2006, the French state moved from a specific public accounting system to a business accounting system.

  3. 3.

    This concept refers to Espeland and Sauder’s work. They define “reactivity” as “the idea that people change their behaviour in reaction to being evaluated, observed or measured. […] Because people are reflexive beings who continually monitor and interpret the world and adjust their actions accordingly, measures are reactive” (Espeland & Sauder, 2007, pp. 1–2). Desrosières has also insisted on this aspect of quantification throughout his work (see for example, Desrosières, 2008b, p. 12).

  4. 4.

    In the same vein, Miller (1990) demonstrated the interrelation between accounting and the state, and Desrosières showed that there is a degree of congruence between modes of governance, conceptions of the state and statistical tools (Desrosières, 2014, pp. 33–58).

  5. 5.

    Miller and Rose (1990; 1992) used the notion of “programme”.

  6. 6.

    It would be possible to go even deeper into the roots of Western civilization and the way it conceived government (Supiot, 2015).

  7. 7.

    A table which groups the various indicators into the different orders of worth can be found in Eyraud (2014, p. 81).

  8. 8.

    We can note here that even if the two ministries did not agree on the way of calculation, they implicitly agreed on the principle that scientific production had to be measured by, and only by, publications in academic journals.

  9. 9.

    In France, the great majority of PhD vivas take place from October to December. This quotation also shows the absolute need, if one wants to understand statistical figures, to go into the details of definitions, delimitations and methods of calculation (Eyraud, 2008). It is one of the reasons why international comparisons using statistical data are so difficult to handle properly.

  10. 10.

    I have followed the budget debates for higher education on the parliamentary channel each year since 2007, and, on that matter, there is no change even after 2012.

  11. 11.

    The calculation can be made using the universities’ Sympa data sheets, which I managed to get hold of for two universities.

  12. 12.

    Grants are allocated based on means-testing parental incomes. The idea behind is that scholarship students are the ones with low cultural capital, thus, with a lowest probability of success which has to be taken into account in the performance measurement of a specific university.

  13. 13.

    Neither the Ministry of Higher Education nor the CPU are homogeneous organizations. These were decisions and positions that had to be won inside these organizations.

  14. 14.

    This process is reinforced by the different “policies of excellence” launched since 2010 and their competitive funding arrangements.

  15. 15.

    I did not conduct any interviews within natural science research centres.

  16. 16.

    But not in all domains. Dubet showed, for example, that quite a similar way of governing French secondary schools pushed them to produce social data on their pupils to justify their choices and to obtain additional resources (Dubet, 2016, p. 387). Statistics produced for acquiring knowledge about pupils existed before, but only at the national level, not at the school level.

  17. 17.

    To understand more about the law and the protests, one can read in French Vinokur (2008) and in English Briggs (2009). The English page of Wikipedia is also quite informative: https://en.wikipedia.org/wiki/2007%E2%80%9309_university_protests_in_France, accessed 15 September 2019.

  18. 18.

    Even if it does not provide a complete analysis of the field, because some of its characteristics and transformations are beyond the scope of the quantification devices. For a comprehensive understanding of the recent transformations of French higher education, one should integrate at least the severe budgetary constraints with which the universities are confronted (Henry & Sinigaglia, 2014; Sinigaglia, 2018), the different “policies of excellence”, the policy of university grouping and merging, and the change of universities legal status (Eyraud, 2020).

  19. 19.

    The situation is different for casual workers, especially for the young generation. The LRU introduced the possibility to recruit “casual lecturers”, but a lot of academics, academic organizations and trade-unions opposed it, so few governing boards of universities decided to hire people under this new status. Things may change quickly as a result of a new law, the “Loi de Programmation de la Recherche” (LPR) passed in December 2020.

  20. 20.

    I develop this idea further with several concrete examples in Eyraud (2019).