Systemic Practice and Action Research

, Volume 22, Issue 1, pp 15–30

Implementing and Evaluating Performance Measurement Initiative in Public Leisure Facilities: An Action Research Project

Authors

    • Department of Leisure and Recreation ManagementKainan University
Original Paper

DOI: 10.1007/s11213-008-9103-y

Cite this article as:
Liu, Y. Syst Pract Action Res (2009) 22: 15. doi:10.1007/s11213-008-9103-y

Abstract

This paper addresses issues when implementing and evaluating performance measurement initiative. Applying action research, the study began with designing an innovative performance measurement system and applied the system in three public leisure centres in England during 2005–2007. It was found that, for practitioners in the public leisure sector, inclusiveness and simplicity are the most important criteria of a good performance measurement system. That is, not only does performance data need to be inclusive, the analytical process also needs to be simple and understandable. In addition, facility managers’ analytical skills and motivations for benchmarking are two factors determining the practicability of the system developed. Finally, while conducting a long-term action research, there is a need to continually communicate the actual and potential benefits of the change with senior managers. If they are committed and enthusiastic, it is easier to gain the support of other levels of the organisation.

Keywords

Action researchPerformance measurement systemBenchmarkingLeisure facilityData envelopment analysis

Introduction

In recent years performance measurement has received considerable attention from academics, practitioners and policy makers. Some authors have focused attention on how organisations can design more appropriate measurement systems (e.g., Bourne et al. 2000, 2002; Neely et al. 1997, 2000 etc.). Also, many systems, such as the Balanced Scorecard (Kaplan and Norton 1992) and the Performance Prism (Kennerley and Neely 2000) have been developed. In the field of leisure management, some frameworks have been designed to help leisure facility managers to assess performance. One example is the National Benchmarking Service for Sports Halls and Swimming Pools (NBS), provided for Sport England by the Sport Industry Research Centre at Sheffield Hallam University, UK. The NBS could be regarded as a performance measurement system, which provides a wide range of performance indicators and benchmarks covering multiple performance dimensions. The performance indicators provided by the NBS fall into four groups: access (12 indicators), utilisation (3 indicators), financial (12 indicators) and customer service quality (20 indicators) (Taylor and Godfrey 2003; Robinson and Taylor 2003). Although widely used, the NBS still has some deficiencies need to be overcome. Up to date, action research, consultancy experience and numerous processes have been developed, which organisations can use in order to improve performance measurement systems (Bourne et al. 2002). Based on action research, this research aims to implement an innovative performance measurement system which has the potential to enhance the current NBS framework.

Research Context

The public services in the UK have traditionally taken a piecemeal approach in measuring performance, relying on a set of partial measures (i.e., performance indicators) that capture particular aspects. The approach adopted by NBS is an example. However, regulators are increasingly attracted to the development of global measures of organisational performance (Smith 1990). For instance, one principle of Comprehensive Performance Assessment—a recent policy aiming to assess the performance of UK local authority and the services that they provide for local people—is to collate a range of existing judgements to provide an assessment of overall performance (Andrews 2004; Broadbent 2003). As argued Young (1992), it is important to consider the overall performance of organisations as well as comparing them on the basis of performance indicators, which capture only one dimension of performance.

Secondly, benchmarking is identified as one of an ever-growing number of management practices aimed at improving performance in public-sector organisations (Magd and Curry 2003). Some studies (e.g., Ball et al. 2000; Ammons 1999) have demonstrated that most UK public organisations tend to focus on league tables as comparators of performance (i.e., data benchmarking), rather than highlighting the causes of differences in performance and generating ideas to improve the processes (i.e., process benchmarking). The NBS is an example of data benchmarking, identifying how performance relates to national benchmarks. However, developing benchmarks is only the first stage in the benchmarking process. Data benchmarking without process benchmarking leads to a much more restricted appreciation of comparisons, because the reasons for relative performance are more difficult to deduce (Taylor and Godfrey 2003; Ogden and Booth 2001). Consequently, data benchmarking and process benchmarking should be ideally combined in a synergistic relationship.

Thirdly, it is necessary to recognise the changing nature of performance measurement. According to Neely and Bourne (2000), managers have become obsessed with measuring performance, so they no longer have time to act on the performance data once it has been gathered. They thus suggested two criteria should be considered while developing a performance measurement system. The first criterion is associated with simplicity and automation. The trick is to measure as little as possible, but to ensure to measure the things that matter. Second, it is also important to extract value from the performance measurement data. Usually people are unaware of the tools and techniques that are available to help them understand the messages implied by the performance data. The three issues above therefore prompted the author to:
  • find a way to ensure the inclusiveness and convergence of performance measurement;

  • facilitate a shift away from data benchmarking to process benchmarking; and

  • make use of the NBS database to extract valuable and concise information for managers.

Literature Review

Partial and Global Measures

Traditionally, two broad approaches have been used to present performance data—‘partial’ and ‘global’ measures (Smith and Street 2005). Partial measure (also known as ratio analysis) has become, over the years, a well-established technique that has found numerous applications in many areas of business (Athanassopoulos and Ballantine 1995). Despite the widespread use of ratio analysis for assessing performance, the univariate nature of the resulting statistics leads to some limitations.

One of the most fundamental limitations is that only two dimensions of activity, represented by numerator and denominator, can be examined in any one indicator. Ratio analysis typically involves the use of a number of performance indicators, i.e. a set of partial measures. In single-input, single-output contexts such a measure is a meaningful, easy to use, measure of performance. However, this is not the case where multiple non-commensurate inputs and/or outputs are involved (Worthington 1998); Thanassoulis et al. (1996) also argued that electing to use only some of the potential performance indicators will bias the assessment and the inclusion of large numbers of variables and lack of an indicator to evaluate unit performance overall often frustrate management efforts to implement strategy. Another problem with using ratio analysis is that conflicting signals may emerge from competing ratios when considering many ratios at once, as is usually the case. An organisation that appears to do well on one indicator may perform less successfully when considered on another. Therefore, drawing conclusions about overall organisational performance from a range of performance indicators is difficult (Al-Shammari and Salimi 1998; Thanassoulis et al. 1996).

According to Chen (2003), a good performance measure system should both have high ‘inclusiveness’ and high ‘convergence’. Inclusiveness means that all aspects of the organisation should be considered and convergence means that consistent and simple information should be provided to facilitate decision-making. A set of partial measures is considered good in inclusiveness but relatively weak in convergence. By contrast, a single global measure, designed to provide an indication of overall organisational performance by aggregating different aspects of performance, is considered good in convergence but poor in inclusiveness, since the information provided is less detailed. Consequently, partial and global measure should ideally be integrated in measuring the performance so that inclusiveness and convergence can be considered simultaneously.

Benchmarking in the Public Sector

Benchmarking in the public sector is analogous to that in the private sector but the motivational forces and obstacles are somewhat different (Kouzmin et al. 1999). According to Bowerman et al. (2002), there are three characteristics that distinguish private sector benchmarking from that which takes place in the public sector. First, in the private sector, benchmarking is often undertaken in order to be the best. In contrast public sector organisations may strive, through benchmarking, to be ‘good enough’ or merely to demonstrate that they are not the worst. Second, information generated through benchmarking in the private sector is confidential, but public agencies have no competitive drawbacks to fear from passing information on to peer organisations. Third, private sector benchmarking is voluntary in nature. However, in the public sector, benchmarking is frequently conducted in response to central government requirements.

There is a wide range of benchmarking activities. For instance Foot (1998) and Ogden and Wilson (2000) suggest the following two types of benchmarking from which public bodies can choose. The first type—‘data benchmarking’ is defined as the numerical comparison of performance in key areas and the identification of performance gaps. Typically, performance indicators are used for measuring performance and monitoring progress against set targets. The second type—‘process benchmarking’ is defined as the comparison and measurement of a specific process against a similar process in own or another organisation. It highlights the causes of differences in performance and generates ideas as to how to improve the processes. Magd and Curry (2003) and Hinton et al. (2000) suggested that the success of benchmarking is based on the desire to change processes as well as outputs and on organisational willingness to search for ideas outside the organisation.

In the UK public sector, the introduction of benchmarking is still in its early stages. Technical problems, scepticism about usefulness and the appropriateness of transferring putative private sector competencies into public administration, and resistance to accepting organisational change as a necessary consequence of benchmarking exercises all prevent the widespread acceptance and use of benchmarking in public sectors (Kouzmin et al. 1999). As a data benchmarking approach, the NBS indicates where performance could be improved. However, it is also important to identify suitable benchmarking partners (i.e. the best practices in the industry) for the facility managers so as to move beyond data benchmarking to process benchmarking.

Characteristics of Data Envelopment Analysis

According to previous research, Data envelopment analysis (DEA)—a well-developed global measure approach—is one technique with the potential to address the aforementioned issues. DEA was developed by Charnes et al. (1978) based on the idea of efficiency measurement first suggested by Farrell (1957). DEA has the following three major functions outweighing traditional ratio analysis.
  • Overall Performance Measurement In DEA, multiple outputs and inputs are integrated to yield a composite indicator presenting the overall performance of a facility. Since all-round performance of a facility is taken into account, it is argued that DEA gives a more balanced approach to performance measurement (Athanassopoulos and Ballantine 1995).

  • Clarify Strengths and Weaknesses In running DEA, outputs and inputs are multiplied by some sets of weights, known as ‘virtual weights’. The inputs and outputs to which higher virtual weights are assigned are those which have higher contributions to the facility’s performance rating, they thus help to clarify the strengths and weaknesses of each facility. Since the comparison of virtual weights need not refer to the mathematical programming concept, they are suitable for the lay-person to understand (Boussofiance et al. 1991; Sarrico and Dyson 2004).

  • Identify Benchmarking Partners Another function of virtual weights is to show an inefficient facility its benchmarking partners, which have similar strengths and weaknesses but appear more efficient (Boussofiance et al. 1991).

The question of how DEA can contribute to the performance measurement of public leisure facilities is the motivation of this research. However, DEA is not a technique without criticism. Several technical problems need to be addressed during the research process. One way is to simplify the use of DEA for practitioners by designing an individual facility report, and then, to implement the developed system on an experimental basis to test if the potential value of global measure can be fully achieved in practice.

Methodology

Methodological Approaches

Action research was adopted as the research strategy because the role of the researcher is to conduct the transfer of an innovative performance measurement system, which was not previously installed the organisations studied. The researcher is therefore required to play a part in the implementation process. Action research allows the researcher to actively participate in some form of change in a system. In this way a change can be triggered by the researcher and then the outcome of that change examined (Greenwood and Levin 1998). Also, adopting action research, the researcher cannot only provide assistance to the organisation in terms of improving benchmarking activities, but also contribute to knowledge in the usefulness and practicability of DEA.

In action research, action and research are combined into a structured process usually referred to as the action research cycle. The cyclic process starts with the recognition of the problem, then plans the action, proceeds to carrying this out and finally evaluates the results obtained Carr and Kemmis (1983). The action research cycle can be passed through once in an action research study and repeated in the same context until satisfactory outcomes have been achieved, or a similar process can be applied at a number of different sites. In this research, a multiple-case design was adopted in order to expand research scope and enhance the rigour of the research. The reasons are twofold. First, this research aims at developing a general global measure system. A degree of generality can be achieved by having several applications and the risk of misjudging of a single event can be overcome (Voss et al. 2002). Second, cross-case analysis can provide the opportunity to search for cross-case patterns (Eisenhardt 1989; West and Oldfather 1995), such as the common barriers centres encounter when they use the proposed system, and thus increase the robustness of analysis.

Given the requirement of longitudinal studies to implement and revise the system developed, this action research were conducted only in three centres, referring to as centres A, B and C. Centres with different management types, i.e. facilities operated directly by local authority (in-house), contracted out to private sector (commercial contractor) and managed by trust, were deliberately chosen. The author states no claim that the centres selected are necessarily a representative sample, but it enables the introduction of diversity into the sample. The second criterion is to select centres that have recently received conventional NBS reports so that the managers are still familiar with the benchmarking results and hence provide a reasonable comparison between the two performance measurement systems. According to Sekaran (1992), this strategy can be defined as ‘typical case’ and ‘judgement sampling’. In terms of data collection, a combination of focus group discussion and semi-structured interviews were conducted in this research. In the beginning of intervention, focus group discussion will be held after the workshop which aims to introduce the system. In the end of the process, semi-structured interviews will be conducted with the management teams to evaluate the usefulness of the proposed system. The characteristics of the three centres and the management teams involved in the research are as follows.
  • Centre A (Trust): Chief Executive and General Manager

  • Centre B (In-house): Project Officer and Deputy Manager

  • Centre C (Commercial contractor): Contract Manager

It is necessary to explain the strategy used to analyse the research findings. Traditionally, either ‘within-case’ and/or ‘cross-case’ analysis are used to analyse the results of case study research (Yin 1994). Within-case analysis entails becoming familiar with each case individually and documenting it thoroughly. In cross-case analysis, similarities and differences across cases are explored. Both approaches will be adopted in the following analyses. The process of introducing the performance measurement system is first presented by adopting within-case analysis. The aim is to provide an in-depth analysis of the managers’ attitudes toward the proposed system and the problems encountered in each centre. Once the data displays for each case were completed, the focus then turns to the usefulness and practicability of the proposed system, and cross-case analysis was adopted.

The usefulness of the proposed system was evaluated by examining the constituent elements of each system. Since the system was evolving and parts of the system’s elements were changing throughout the process, it is more appropriate to assess the usefulness of each constituent element than compare the system’s overall usefulness across cases. In terms of the evaluation of the system’s practicability, during the post process interviews, the managers were asked to discuss the system as a whole rather than on an element-by-element basis.

Research Phases

According to Susman and Evered (1978), this action research comprises the following five phases:
  • ‘Diagnose’ involves the identification by the researcher of an improvement opportunity at a prospective client organisation that is likely to lead to the development of relevant knowledge. In this study, literature review is the first stage of action research, which aims to figure out the deficiencies of traditional performance measurement approach.

  • ‘Action planning’ involves the joint development and consideration of alternative courses of action to attain the improvement identified and knowledge development. In this study, the purpose of this stage is to analyse the performance data and design an individual report for each facility. The information about overall performance, a centre’s strengths and weaknesses and suitable benchmarking partners were provided.

  • ‘Action taking’ involves the selection and implementation of one of the courses of action considered in the previous stage. In this research, this stage aims to communicate the key findings to management is stressed. It is done by having a workshop to explain how the report should be interpreted.

  • ‘Evaluation’ involves the study of the outcomes of the selected course of action. In this research, focus group discussions was held after the workshop, which allow the researcher to investigate the suitability and acceptability of proposed system.

  • ‘Reflection’ involves assessing the outcomes of the evaluating stage and, based on this assessment, knowledge generation in the form of a conceptual or theoretical model describing the situation under study. Reflection leads to a revised plan followed by a new action research circle. In this research, this stage focused on the practicability of each element in the individual facility report so as to generate new knowledge of aggregate performance analysis.

In all three centres, the fact that the system is under development was made explicit to the management team at the outset of visit and members of the teams were continually encouraged to comment on system’s practicability. For Centre B and Centre C, one workshop was held at the beginning of the intervention and semi-structured interviews at the end of the process. For Centre A, two workshops were conducted—in the beginning of the ‘system enhancement’ stage and at the end of ‘system evaluation’ stage. While revisiting Centre A the second time, an interview was also conducted immediately after the workshop to investigate managers’ first impressions of the revised system. Evaluation of the actual use of the system is unlikely at this stage of the research, but the feedback from the three centres allows the author to formalise the system and critically evaluate the practicability of aggregate performance analysis. Each workshop presented the system and each constituent part of the system was critically appraised. After each workshop, the author reflected on the acceptability of system and in turn triggered the modifications of the system’s constituent elements.

Results

The focus here is on the action research process in each centre and how the system was evolved. As shown in Table 1, the system consists of two dimensions (i.e. efficiency and customer service measurement) and five constituent elements. Three of the five elements are suggested to be included in the final system (i.e. the revised report for Centre A).
Table 1

Elements included in the report for each centre

Dimensions and elements

Centre A (Pilot)

Centres B & C

Centre A (Revised)

Efficiency measurement

1. DEA-performance score and target

 

2. DEA-strengths and weaknesses

3. DEA-benchmarking partners

Service quality measurement

4. DEA-service quality measurement

  

5. Grid analysis diagram

 

Note: ▲ signifies the elements included in the report for each centre

Cycle 1—Pilot Research in Centre A

Centre A (managed by a trust) was chosen as the pilot centre as it had just received the NBS report and during a NBS feedback session the Chief Executive criticised several aspects of NBS and presented his willingness to see how the current system can be improved. It therefore stimulated the author to seek a way to improve the current system. According to the literature, a standard DEA report was constructed consisting of four parts of analysis: overall performance score, performance targets, strength and weakness clarification and benchmarking partners’ identification. The pilot report was sent to the manager 2 weeks before the visit.

During the focus group discussion, some merits of the proposed system were appreciated by the managers, such as the function of DEA in identifying strengths and weaknesses. The General Manager stated that: “The headline information of centre’s strength and weakness forces us to look at the issues that are important to the centre”. The Chief Executive also criticised the design of the NBS report and suggested providing a summary table illustrating the centre’s strengths and weaknesses such as the proposed system did. Another useful part of the report is the identification of benchmarking partners. The Chief Executive stated that: “Benchmarking partners with similar background are better than figures (i.e. benchmarks)”.

However, several deficiencies of the proposed system were also identified during the discussion. The major criticisms on DEA are its complexity and inappropriateness in measuring service quality. In terms of the complexity of DEA, the General Manager remarked: “If you are here presenting it, we have opportunities to ask questions. If you just give me the report, I will struggle.” The General Manager also doubted the appropriateness of applying DEA to measure service quality. He emphasised that customers’ comments on individual attributes are more important to the managers, and implied that: “The result is concise but not necessarily to be the right information”. The Chief Executive further suggested that: “The report should be readable and digestible”. He said: “it is not necessary to show all the survey results…a concise report showing the most important information is more likely to be remembered”. From his point of view, in the report, some places are too complicated (e.g. the concept of DEA) but some places are not detailed enough (e.g. service quality measurement). In line with the feedback from Centre A, it was decided that the content of the report should be simplified and the adoption of DEA to measure service quality should be substituted by another technique.

Cycle 2—Centre B

The second action research cycle was started by reviewing the literature to seek out a better approach to present the customer survey data. Importance-performance grid (also known as grid analysis), developed by Martilla and James (1977), was found to be the most common ways used to measure service quality. In its essence, the importance-performance grid combines measures of attribute importance and performance into a two-dimensional grid in an effort to ease data interpretation and derive practical suggestions. Grid analysis has gained popularity over recent years for its simplicity, ease of application and diagnostic value (Oh 2001; O’Neill and Palmer 2004; Mori 2002). This modification was also stimulated by the focus group discussion in Centre A where the managers recommended visualising the text and figures so as to facilitate data interpretation. Grid analysis was included in the report for Centres B and C, and some of the text in the report was also replaced by figures or tables to ease reading. In additional, the reports for Centres B and C were identical.

In Centre B, the managers appreciated especially the grid analysis which provided a snapshot of the centre’s overall service quality performance. The Project Officer highlighted that: “It is a good example where joint use of visual representation and brief text allows managers to know good and bad areas immediately”. The identification of benchmarking partners was another benefit perceived by the manager. The Deputy Manager said: “Similarity is important… we can mimic what they are doing and identify why we are underperforming.” The managers also expressed their willingness to contact other better performers provided in the report.

However, the concept of DEA itself was again criticised as too complicated. The Project Officer stated that the presentation made the system clearer, but it was complicated when she went back to the report alone. From her point of view, “the customer part can be understood, but DEA is difficult… The outcome of DEA is understandable, but how to get that is too complicated”. To facilitate understanding, the Project Officer recommended a meeting with the analysts. She said: “A meeting with you allows us to ask questions and clarify suspicious points.”

Reporting too much information was also identified as a problem. This was particularly the case when the managers hand the report to other staff. The Deputy Manager remarked: “The report is more than twenty pages, but we only used the first four pages… Too many reports, we don’t have enough time to analyse and interpret the report”. He suggested that “the content of the report should be easily accessible to different level of staff and departments … it is better to provide a summary of findings in the first few pages and reduce the quantity of text”.

The post-process evaluation revealed that Centre B provided an example of using the proposed system. The information within the report was used to produce the action plan. According to the Deputy Manager, it was done by listing out the areas capable to do and taking the NBS and the new system simultaneously to address the weaknesses. He said: “We know our strengths and weaknesses. The report reinforced our thoughts and a discussion allowed to double check the evidence… The report shows me an overall picture. Together with the details of NBS, we can have a clearer view of where we are”. The Project Officer further remarked: “The new system provided a holistic overview of the organisation’s performance and the NBS report provides more detailed information. A joint use of two systems has complementary benefits”.

The Deputy Manager also indicated that the cooperation between council and facility manager was the determinant of success in using the system. He remarked: “Effective use of the measurement system is due to her (the Project Officer) promotion of the importance of this report”. Although Centre B provides an example of using the proposed system to produce the action plan, no action was taken regarding contact with the benchmarking partners. This fact was an unexpected result since the manager expressed his intention to conduct process benchmarking during the workshop. The Project Officer confessed that: “We intended to contact them, but time passed by… Other more important issues occupied our time”.

Cycle 3—Centre C

In Centre C, the headline information of the centre’s strengths and weaknesses provided by DEA was once again regarded as an important function. The Contract Manager stated: “It gives me a snapshot view of our performance … A sharp line separating the strength and weakness results in a more focused organisation”. The post-process interview with the Contract Manager also showed that DEA provided a medium that could be used to communicate with the local authority. He remarked: “The weighting system is the most useful part in the system …We are strong in financial performance, but cash is not relevant to the council … A balance can be struck between us when I show them the results”.

However, differing from Centre A and Centre B, the Contract Manager was sceptical of the performance targets suggested by DEA, which were significantly higher than the 75% NBS benchmarks. He was confident of the centre’s financial performance. He expressed his doubt about the DEA results and implied that the DEA targets were unrealistic to him. The Contract Manager also criticised that there was too much jargon in the report. He said: “It is easy for people to stop reading because they will find that the document is not relevant to them … Ensuring that the report reflects the issues that are important to us is important if measurement is to be useful and help management”. He further suggested to simplify the technical concepts and make the report a more friendly and accessible document. He said: “It is important that the report can be skimmed by a manager within a short period of time”. A meeting with the researcher to explain the report was suggested as the manager can question the findings.

In terms of the actual use of the proposed system, the manager hasn’t either used the system in a formal way or contacted with the benchmarking partners. The Contract Manager proposed that: “The report gives us some points… However, this kind of analysis is probably more important to people in a higher level, such as councillor or director”. He also explained the reason why no attempt has been made to contact with the benchmarking partners: “it is not because of unimportance, but because of priority”.

Cycle 4—Revisit Centre A

After testing the system in three centres, it was believed that the system’s general acceptability had increased significantly so it was worth revisiting Centre A to investigate whether the revised system is more acceptable than the pilot system. During interviews after the workshop, the burden of considering simultaneously two systems (i.e. NBS and the pilot report) and the availability of management time were identified by the Chief Executive as the main reason for slow progress. He commented: “There were two real problems with implementing your system. The first was getting the management to feel happy with it. It is easy to feel threatened by them and they needed to be persuaded that it was good for the business and not a threatening initiative… (Secondly), we are bombarded by too many reports, we do not have enough time to put everything into action”. The management team therefore failed to use the pilot system, in favour of the traditional NBS report.

By contrast, the acceptability of the revised system was much higher than the pilot system from the Chief Executive’s point of view. He commented: “Compared with the previous one, the new one is more readable and digestible… There are too many figures and a lot of cross-references are required when reading the previous report”. With respect to the constituent elements of the revised system, similar to Centres B and C, grid analysis was found to be the most useful part in the report. However, the suitability of applying DEA was again criticised by the manager: “Even though the outcome of DEA is succinct, the process is not transparent, which makes the results less convincing” (General Manager). He further recommended that: “Such an aggregate analysis may be more useful at the strategic level but a facility manager needs more detailed information”.

Discussions

After displaying the findings in each case, this section will search for cross-case patterns by discussing the similarities and differences across cases. In terms of the usefulness of system’s constituent elements, four issues have been raised in the aforementioned within-case analysis and can be grouped into the following two dimensions.

Usefulness of System’s Constituent Elements

The Pros of the Proposed System

Strengths and Weaknesses Clarification The summary information of strengths and weaknesses was found, across three centres, as the most useful part of the system, and was the most frequently cited benefit of DEA. The summary information of centre’s strength and weakness forced the managers to look at the issues that are important to the centre (Centre A) and resulted in a more focused organisation (Centre C). For Centre B, the headline information of centre’s strengths and weaknesses provided by DEA had the biggest effect in complementing the NBS. Since contradictory information is often found while examining a series of efficiency ratios, DEA allows clear distinction between the strengths and weaknesses. However, the performance indices provided by the NBS can still give an insight into the gaps in specific performance dimensions. This is an area where the two approaches can complement each other. Furthermore, this function of DEA needs not refer to complicated mathematical concepts, which has been argued by Staat and Maik (2000), and are therefore comparatively easy for the lay person to understand and accept. Similarly, applying grid analysis to measure service quality can also offer a snapshot of overall performance and a clear-cut line demarcating the strengths and weaknesses.

Benchmarking Partners Identification The second initiative appreciated by the managers is the identification of benchmarking partners. For Centre A and Centre B, providing systems with similar backgrounds to follow is a benefit. Although there was a general consensus among the managers interviewed of the importance of process benchmarking, none of them has taken steps to contact the benchmarking partners provided by the system. Although some managers expressed their willingness to contact other centres, all of the interviewees admitted that the priority of process benchmarking is relatively low. It appears that there are substantial barriers to move beyond data benchmarking to process benchmarking. This result also echoes the study of Ogden and Wilson (2000) who found that it was rare to find cases where a full range of benchmarking activities has taken place and data benchmarking is still prevailing in the UK public sector.

The Cons of the Proposed System

Inappropriateness of Aggregate Assessment As argued by Charnes and Cooper (1994) and Athanassopoulos and Ballantine (1995), all-round performance is taken into account while applying DEA, rather than basing the assessment of individual performance indicators. It therefore gives a more balanced approach to performance measurement. However, the research findings showed that a single score provided by DEA representing the centre’s overall efficiency provided no operational meaning for the managers. Both Centre A and Centre C proposed that such an aggregate approach may be more useful at a higher strategic level of decision makers. Similarly, the major pitfall of DEA in service quality measurement is that detailed information is lost while aggregating various attributes into factors. For the practitioners, more detailed information is required to understand customers’ views about each single service attribute because management decisions relate to individual attributes.

Performance Targets Provided of DEA The target suggested by DEA is to become the best, but it may not be the aim of most public service providers. This assumption is reflected in the case of Centre C. As argued by Bowerman et al. (2002), in the private sector, benchmarking is often undertaken in order to be the best. By contrast, public sector organisations may strive, through benchmarking, to be ‘good enough’ or merely to demonstrate that they are not the worst. In this case, benchmarking results become more important than acting on those results in order to close performance gaps.

Practicability of the System as a Whole

The results presented above are of specific interest for the usefulness of each element within the system. Here, the concentration turns to the determinants of success of system’s practicability as a whole. They can be grouped into two different categories. The first set concerns the system design itself, while the second set concerns the implementation process.

System’s Design

“The quality of the performance measurement system is critical in establishing the credibility of the measurement processes, and therefore critical to the confidence managers would have in using the system to assess and evaluate the programs” (Bernstein 2001, p. 99). Indeed, some authors have discussed the design of performance measurement systems and suggested that it should be transparent, simple to understand, have visual impact and visible to all (e.g. Neely et al. 1997). The results basically coincide with these findings and two quality characteristics were cited most frequently by the interviewees.
  • Understandable The lack of appropriate skills to interpret the DEA results was identified as a common barrier. It emerged that there was considerable confusion in the minds of those interviewed regarding their understanding of the terms related to DEA. For instance, there is a lot of jargon (Centre C) and there is a difficulty to understand the report alone (Centre B). To facilitate understanding, some recommendations were made, such as: a meeting with the analysts to ask questions and clarify suspicious points (Centre B) and simplifying the technical concepts within the report (Centre A).

  • Concise Due to time constraints, it is important that the report can be skimmed by a manager within a short period of time (Centre C) and it is not necessary to show all the survey results (Centre A). Some suggestions were made to make the report a more friendly and accessible document, such as: providing an executive summary at the front of the report (Centre B) and simplifying the technical concepts (Centre C).

System’s Implementation

As argued by Neely et al. (2000), the process of designing a measurement system is intellectually challenging, however there is increasing anecdotal evidence to show that the real challenge is the implementation. The following three issues were revealed by the managers which may be the main barriers to implementation.
  • Commitment Top management commitment was found to be critical to the success of the system’s implementation. In Centre B, the Project Officer has emerged to promote the use of the system. It is therefore argued that the leadership must have conviction in the validity of the system in the first place, and then the engagement of senior managers will help to facilitate its use.

  • Time The availability of management time was identified across three centres as the main reason for slow progress. Joining more than one performance measurement scheme, the managers stated that they do not have enough time to read so many reports (Centre B) and to put everything into action (Centre A).

  • Priority Given that the length of time between the intervention (i.e. workshop) and evaluation (i.e. post process interview) lasted more than 8 months, it is highly likely that management found other more pressing issues and the enthusiasm for change declined. As such, the distraction of other events (Centre B) and relatively lower priority to use the new system (Centre C) were identified as the reasons for not contacting with the benchmarking partners.

Conclusion

This research addresses issues when implementing and evaluating performance measurement system in public leisure facilities. Many issues of relevance to the growing literature in the field of performance measurement while providing organisations with an effective performance measurement system were also discussed. The paper will conclude by discussing the usefulness of the framework developed and the implications for performance measurement in the public leisure sector. The implications of this action research are threefold.

The first implication relates to the criteria of good performance measure system. The results showed that grid analysis was highly appreciated by the managers. As discussed earlier, two major benefits of grid analysis are: a snapshot of overall performance and a clear-cut line demarcating the strengths and weaknesses. Actually, these two benefits can also be realised by DEA, but the acceptability of DEA was much lower than that of grid analysis. Since the application of DEA requires levels of knowledge in mathematics which most managers in the public leisure sector probably do not possess. The analytical process of DEA is less understandable, so the results are less transparent and less acceptable. According to the ‘inclusiveness’ and ‘convergence’ criteria proposed by Chen (2003), DEA can be regarded as good in ‘convergence’ (i.e. providing a snapshot of overall performance) but weak in ‘inclusiveness’ (i.e. presenting detailed results). The research demonstrates that ‘simplicity’ is another important criterion of a performance measurement system. By applying grid analysis, ‘inclusiveness’, ‘convergence’ and ‘simplicity’ are ensured simultaneously since it provides detailed information, visualises complicated data, condenses all results into one diagram, and provides simple information to facilitate decision-making.

The second implication is about the willingness of facility managers to apply the proposed system. The research findings demonstrated two factors, which determine the feasibility of introducing the proposed system into the public leisure sector. Firstly, the system’s practicability relies heavily on the communication and interaction between practitioners and analysts. By adopting action research, the author played a proactive role in introducing the new system. However, the support of leading bodies is necessary in the future to stimulate other local authorities to adopt the proposed system. One example to follow was the promotion of the Public Services Productivity Panel to introduce DEA to measure the efficiency of UK police forces (Spottiswoode 2000). Secondly, although the usefulness of DEA in identifying benchmarking partners was appreciated by the facility managers, and the public sector may have greater potential for benchmarking than the private sector due to the availability of a wide choice of benchmarking partners (Bowerman et al. 2002), no real action was taken to contact the benchmarking partners identified by the system. Time constraints were frequently cited as the reason by the managers, but this finding has wider implications about the motivation of benchmarking in the public sector. If data benchmarking is an end in itself and local authorities have little incentive to conduct process benchmarking, the value of the proposed system will be seriously constrained.

Thirdly, this research is mainly related to performance measurement initiatives but has wider implications for management commitment in change management. The literature highlights many of the issues affecting the management of change within organisations. The results provide a structured view of the factors affecting the system’s implementation and basically coincide with some previous researches. For instance, top management commitment was identified as a key driver for successful performance measurement initiatives (e.g. Bourne et al. 2002; Bauer et al. 2004; Bernstein 2001) and implementation of change (e.g. Frizelle 1991; Kotter 1995). The availability of management time was also found to be essential to implement a new performance measurement system (e.g. Bourne et al. 2000). It appears that if a long-term action research is required, there is a need to maintain enthusiasm and momentum for the duration of the research. Thus, there is a need to continually communicate the actual and potential benefits of the change with senior managers. If they are committed and enthusiastic, it is easier to gain the support of other levels of the organisation.

While case study data cannot be generalised to populations or universes, it can be generalised to theoretical propositions (Yin 1994). Thus, this paper will conclude by discussing the following theoretical proposition developed. The scope of this study was confined to the local level of leisure industry. Since the facility managers generally lack sufficient analytical skills, and much more detailed benchmarking results are needed for local managerial purposes, ‘simplicity’ and ‘inclusiveness’ are more important than ‘convergence’ as criteria of a performance measurement system. That is, for practitioners in the public leisure sector, not only does performance data need to be inclusive, but the analytical process also needs to be simple and understandable. This may be the reason why traditional ratio analysis, an inclusive and easily interpretable approach, continues to be the method of choice, at least for the UK government, in reporting performance in the delivery of public services (Thanassoulis et al. 1996). However, as proposed by Smith and Street (2005), global measure (such as DEA) may be more useful from a strategic regulator’s point of view, since it might, for example, identify beacons of good practice. This is a proposition yet to be tested and can be used to frame further research.

Copyright information

© Springer Science+Business Media, LLC 2008