1 Introduction

Since 80’s the introduction of New Public Managementprinciples has promoted the use of performance measurement to drive a more efficient, effective and accountable public sector (Hood 1995; Lapsley 1999). The adoption of a sophisticated and comprehensive multidimensional performance measurement system, which looks beyond traditional financial measures, based on organization strategies, such as the balanced scorecard, has thus been suggested (Linard et al. 2000; Eccles 1991; Jackson 1993; Kloot and Martin 2000; Fottler et al. 2006; Yang and Tung 2006). In particular, within the health care sector, several studies developed conceptual frameworks and models in order to help countries in building effective tools (Arah et al. 2003, 2006; Smith 2002; Chang et al. 2002). At the same time, a large number of authors presented single or comparative papers on the design of the performance evaluation systems to be adopted both at the organizational, regional or national level (e.g. Linard et al. 2000; Aidemark 2001; Zelman et al. 2003; WHO 2003; Ba-Abaad 2009).

While there is a broad range of literature on performance evaluation systems at hospital level, still few papers are written on the regional one, and the majority of them concern the design and implementation process (Hilarion et al. 2009; Pink et al. 2001). Indeed, the analysis of performance evaluation systems at the regional level and the effects of its use on system performance are of particular interest, given also the devolution process which has recently involved most of the European countries such as Italy.

In the last decade, each of the twenty Italian regions developed its own management tools (AAVV 2007; Vainieri and Nuti 2011). Among others, the Performance Evaluation System developed by Tuscany region in 2005, has been valued as a particularly innovative and comprehensive system (Carinci 2011; Censis 2008). Actually, starting from 2008, the Tuscan PES framework has been adopted by other eight Italian regions as an internal evaluation tool and for external benchmarking purposes, and also by the Ministry of Health, in order to monitor the uniform and essential levels of health services provided within the country (Nuti et al. 2012).

Given these premises, this paper reports the novel experience of the Tuscan PES; in particular, it assesses PES effectiveness and discusses the critical factors that could have led to the PES success. The outline of the paper is as follows: the second section describes the methodological aspects; the third section reports the making of the Tuscan PES, both the design and the implementation process; the fourth section sums up the first evidence of the PES effectiveness; the fifth one discusses the critical factors that could have led to success and the sixth one concludes.

2 The methodological approach

The Tuscan PES was designed and implemented throughout a constructive research approach (Kasanen et al. 1993). Starting from the performance measurement framework existing in literature, especially in the healthcare context, researchers developed a specific framework for the Tuscan reality looking for its practical usefulness, simplicity and its ease of operation by the stakeholders. At this regard, the study involved many professionals, healthcare managers both at local and regional level including also the regional health councilor.

The regular tracking of performance measurement over the years allowed analysis of the effects that the constant use of PES had on performance. From the methodological point of view, the effectiveness of performance measurement tools can be verified in various ways. The institutional stream of literature, for instance, measures the effectiveness of the performance measurement systems by analyzing the degree of legitimacy (Modell 2001, 2003; Chang 2007), while other approaches measure effectiveness by analyzing the improvements which have been reached over time (this is the approach adopted in this paper), or by comparing results of organizations adopting performance evaluation tools against “control” organizations (Davis and Albright 2004), or by measuring performance effectiveness trough qualitative surveys (Braam and Nijssen 2004).

With regard to the identification of the critical factors that led to success, authors collected and discussed with the research team issues that were recognized also by local and regional managers as elements of success. In particular, some items reported as critical factors of success came from the comments and observations of researchers during the constructive research and these 5 years of PES implementation. Some frequent quotations of Tuscan employees or local/regional managers are reported in the text in order to support the authors’ opinion. In addition, some observations are supported by empirical evidence and, whether available, also by international external auditors.

3 The making of the Tuscan performance evaluation system

3.1 The design and reporting system

The Tuscan PES was designed and implemented in order to pursue the regional strategic objectives on the basis of a specific willingness of the Regional Health Councillor. The system was intended to measures the quality of services provided and the capacity to meet citizens’ needs in order to achieve better health and quality of life standards on one side and, on the other, to preserve financial equilibrium.

The Tuscan PES was firstly implemented in 2005 and actually consists of 50 composite and more than 130 simple indicators. All the indicators are classified in the following six dimensions (Nuti 2008; Nuti et al. 2009): population health status; capacity to pursue regional strategies (i.e. to guarantee that strategic regional goals are pursued in the indicated time and manner); clinical performance (i.e. quality, appropriateness, effectiveness, clinical risk management and primary care); patient satisfaction; staff satisfaction and finally efficiency and financial performance.

Starting from the regional objectives and strategies and by looking at experiences carried out by other countries (i.e. Ontario Health System and English NHS), performance indicators were selected by a team of Tuscan Health Authorities (HAs) and Regional Administration professionals. This initial selection process lasted 1 year and it is still ongoing in order to add new indicators or to refine the existing ones.

The reporting system is visually represented by a “target” diagram which is divided into five colored evaluation bands. Every year each HA receives its own target and the more the HA is able to reach objectives and high performance levels during the year, the closer the performance indicators are to the center (the dark green area); on the contrary, bad performance results are represented by indicators which are positioned far from the centre, in the red area (Fig. 1).

Fig. 1
figure 1

Two examples of local health authorities target diagrams

Full size image

3.2 The implementation process

Since the beginning, the Tuscan PES has been adopted as a decision support tool at both regional and local strategic management level. Performance results on selected indicators are monitored every 3 months and discussed in systematic meetings between the HAs top managers and the Regional administrators. During these meetings the HAs managers are asked to discuss about the performance of their organizations, to provide feedbacks on the strengths and weaknesses and to discuss about future actions and goals.

Moreover, healthcare top management and professionals are also actively involved in the performance evaluation process. On one side they are involved in the indicator definition and refinement process; on the other, they are called to participate to the organizational climate survey which is carried out about once a year within all Tuscan health organizations. Regarding the climate survey, all the employees are asked to provide their opinion about selected aspect of their organization such as leadership style, team working, budget process, training opportunities. Survey results provide top management with valuable and instructive feedback about the strengths, the barriers and the conflicts in the organization. This yearly process contributed to open a constructive dialogue in the organization that empowers employees and helps them in identifying a connection between their suggestions and behavior and the actions of managers.

Starting from December 2007 all the performance indicators, presented in benchmarking, and the yearly targets are available online for top managers and professionals.

Once the employment of the PES had been integrated within the organizations, had become common practice, and had been understood and embraced at all levels, the regional Councillor decided for a public disclosure. Consequently, since July 2008 most of the indicators are freely accessible to citizens on the web (www.valutazionesanitatoscana.sssup.it).

Finally, with regard to this implementation process, the PES results are linked to the CEOs’ reward system. This implies that specific goals for all Health Authorities are set on the basis of three issues: the performance level reached by each Health Authority, their average performance and the defined gold standards (generally set by the Regional Department of Health). Starting from these parameters and following the approach applied by the Agencia Valenciana de Salud (Gómez and Carrillo 2009) the annual goals are set, separately for each indicator and each HA. Compensation details linked to these goals are then discussed and based on accurate and solid measures in order to be reasonable and reliable.

4 First evidence of PES effectiveness

The potential of the PES process for improving performance of health organizations is firstly measured by taking into account changes in the absolute values of the indicators over a 4-year period. The Tuscan health care system seems to make overall progress over the years: more than 50 % of the indicators significantly improved their yearly performance (Fig. 2).

Fig. 2
figure 2

Percentage of improved indicators (average across HAs) in the years 2006–2009

Full size image

As example, the indicator regarding the percentage of femur fractures operated within 2 days increased from 36 to 52 % (p < 0.001) in the four-year trend. Moreover, Tuscan performance, in comparison with other three regions that have been adopting the same PES since 2008, registered a continuous and significative improvement in most of the common indicators. Figure 3 shows, as an example, the 2008 values relating to the percentage of femur fractures operated within 2 days in Tuscany and in the other three regions.

Fig. 3
figure 3

Percentage of femur fracture operated within 2 days, trend 2007–2009

Full size image

At the same time, the constant use of the PES by Tuscan health organizations seemed to promote initiatives for narrowing disparities in health care quality: 63 % of the performance indicators significantly reduced their variability (calculated in terms of standard deviation over LHAs) over the years (Fig. 4). From the managerial point of view unwarranted variability (that is variability that remain after having standardized for population health needs) negatively affects equity in access or/and treatments, which is one of the main goals of a public health care system (Wennberg 2010; Jacobs et al. 2006). Consequently by addressing this kind of variability might help to develop an efficient system able to preserve the principles of egalitarianism and solidarity.

Fig. 4
figure 4

Percentage of indicators that reduced standard deviation over the years

Full size image

The evidence reported seems to support the hypothesis that the constant measurement of organization performance along with targeted objectives orients individuals and organizations towards responsible behaviours (Bevan and Hood 2006). Therefore regional government must support the widespread use of benchmarking data and involve the whole workforce in a shared commitment for improving quality.

5 The critical factors leading to success

Results reported in the previous paragraph have been achieved with the contribution of several factors that have also made the popularity of the Tuscan PES growing over time.

The principal successful factors can be summed up into five points:

  1. 1.

    The striking visual reporting system (Bevan 2009) which is able to highlight multidimensional aspects that impact on performance. The target diagram and its colors became a common language among managers, politicians, professionals and it also represented a friendly way through which healthcare results are communicated to the public. Thanks to its communication power, the PES allowed top managers and professionals to talk the same language. Top managers started to move their attention from financial and efficiency indicators to other performance aspects such as quality and appropriateness of care. Whereas reporting the results in benchmarking across all the levels of the organizations helps professionals to learn from other experiences and to overcome self-referential behaviours. These aspects are highlighted by the debates that occur during public meetings where professionals speak about indicators in terms of colours and codes of indicators and say sentences such as “I am red in C5 while I am green in F15”, which respectively mean that the HA has a critical situation in the quality indicators while it has a good performance in the work safety prevention services.

  2. 2.

    The linkage between PES and CEO’s reward system, which adds emphasis on performance results and guarantees the coherence of managerial tools used at all levels. Since most indicators are used for the CEO’s reward system, all HAs have introduced these indicators in their budgeting processes. Most of health authorities budget worksheets include standards and indicators of the PES often labelled as “PES indicators”. Aligning regional and local managerial tools is a fundamental factor in order to achieve improvements.

  3. 3.

    The public disclosure of the results via website started in 2007. Since then, all professionals, health managers and also citizens have been able to monitor performance by their HA and to compare it with other HAs’ performance. The publication of data about Tuscan health organizations performance engaged interests and responsibilities of all the stakeholders of the system and at the same time preserved them from the reputational damage (Brown et al. 2012).

  4. 4.

    The level of involvement into the performance measurement of the stakeholders of the system. Professionals and managers are involved in various ways: from the climate survey questionnaire to the training courses. All these professionals were asked to be involved in the whole measurement process, and at the same time to actively participate in organizational climate surveys for achieving a collective and continuous improvement. Empirical results from the climate surveys, for instance, pointed out that managers and employees reporting to be aware of the PES results tend to be more satisfied of the use of budgeting tools within their organizations.

  5. 5.

    The strong political commitment throughout the process (i.e. the use of Tuscan PES during periodical meetings with CEOs and its transparent way of showing data in benchmarking).

Indeed, the strong Health Councillor’s commitment enhances confidence in and attributes power to the Tuscan PES. The Health Councillor who sponsored the birth and development of the Tuscan PES was in office for two legislations (about 10 years). This gave a long period of stability that allowed the Tuscan PES to strengthen and refine the tools in order to face a real change.

The success of the Tuscan PES has to be addressed to the comprehensive strategy that was adopted, which includes all successful factors. Although more and more authors in the healthcare literature raise increasing expectation on the public disclosure of data (Leatherman and McCarty 1999; Marshall et al. 2000; Hibbard et al. 2003; Rogowski et al. 2004), this factor may be even more effective if associated with others. Indeed, Pinnarelli et al. (2012) demonstrated that in Italy only two regions improved the percentage of femur fracture operated within 48 h: Lazio and Tuscany. Both of these two Regions decided to publicly disclose data, however improvement registered by Tuscany was significantly higher than Lazio because of the more comprehensive strategy implemented by Tuscany Region.

6 Conclusions

First evidence on the Tuscan experience seems to prove that the performance evaluation system helps the Regional government to both evaluate its strategic actions and promote a “managed” competition among health organizations. Moreover, the information dealt with and uniformly represented enabled a constructive benchmarking across some Italian Regions and Health Authorities within these regions. The regional administration supported this process by coordinating a benchmarking system, in order to encourage local organizations to learn from other experiences, to overcome self-reference behavior and to improve, through a competitive environment based on reputation.

The Tuscan PES can be considered a successful experience in the field of performance management thanks to a combination of factors that acted together.

The underpinning factors linked to the success can be divided into two groups: factors that can be replicated into other contexts and factors that cannot be easily generalized.

The first group of factors relate both to the design and implementation process of the PES, such as the user-friendly reporting system, the continuous refinement and updating work on indicators. Such factors proved to be effective regardless of the context of application.

Scholars pointed out that the process plays a pivotal role. Traditional management accounting systems, oriented to cybernetic models, fail to work in human organizations because they are perceived by most employees as an attempt by technocrats to impose their implicit goals on the other people (Hofstede 1978). The making of Tuscan PES, instead, seems to be closer to the pragmatic constructivism approach described by Nørreklit et al. (2007). These authors argue that the control system should follow a constructivist perspective in order to catch and translate the “reality” of all actors involved into the measurement tools. In this respect, only socially constructed tools can help to stimulate employees internal commitment and lead them to act instead of reacting. In this direction it is fundamental an interactive control process from the strategy formulation to the implementation step (Nørreklit 2000; Nørreklit et al. 2007). The Tuscan PES seems to have these characteristics of interactivity. Indeed, multiple actors (from health managers to policy makers) have been participating from the design of the model to the continuous maintenance of the system and this helps not only to increase the professionals’ involvement, but also to reduce the risk of an excessive attention only to what is measured (thus reducing the synecdoche phenomena) (Bevan 2006).

The second group of factors of the Tuscan PES, the tricky ones to generalize, are those strictly related to the willingness of the Tuscan policy managers (councillor or the CEO) and their leadership. Policy makers have to be aware that if the performance evaluation system has to act as an important support tool, it requires them to be strongly committed. This means that managers should actively support the public disclosure of data, should link the performance system to other management tools (i.e. the reward system) and, above all, should use the system during business meetings.

Having outlined the critical factors of the Tuscan PES leading to its success, further investigations may concern, in the next years, in-depth analyses on differences regarding the application and use of the Tuscan PES approach in other regions.