Background

Clinical audit is a quality improvement tool that facilitates the evaluation and improvement of care [1, 2]. In human medicine, it has become a standard of care required by some national governing bodies [3] and its implementation has yielded measurable improvements in clinical care [4–8]. Furthermore, the absence of clinical audit has been identified in allowing persistent poor performance, resulting in failures of care [9]. In veterinary medicine, national level audit with explicit criteria and reporting standards set by governing bodies does not currently exist. The only nationally mandated audit the authors are aware of is the requirement that clinics participating in the Royal College of Veterinary Surgeons Practice Standards Scheme perform audits to evaluate clinical outcomes (with the implication that these are limited to individual clinics) [10], though many veterinary clinics do perform informal audits through practices such as mortality and morbidity rounds. A formal clinical audit, first described in veterinary medicine by Mosedale [2], is centred around a structured audit cycle. In its simplest form, a clinical audit can be described by a 4 step process: Plan-Do-Study-Act (PDSA) [11–14]. This process is cyclical in nature to allow for continuous monitoring of performance and improvement [1, 11–14]. In the “Plan” step, a clear problem is identified [14]. The problem can be an outcome, such as anesthetic mortality rate, or a process associated with achieving an outcome, such as the number of patients receiving a physical examination prior to general anesthesia. When a clinical audit focuses on an outcome or process it may be described as an “outcome [clinical] audit” or “process [clinical] audit”, respectively. The “Do” step is the process of data collection [1, 11–14]. Following data collection, the “Study” step is analysis of the data followed by comparison of results to a predetermined set of standards (“explicit criteria”) [11, 14–16]. Finally, the “Act” step is the development of recommendations based upon the outcome of "Study". These recommendations are then implemented and additional audit cycles performed [1, 11–14]. In summary, clinical audit compares a current standard of care against clearly defined criteria and uses this information to drive change with the goal of improvement in patient care [1, 14]. The process should be cyclical, with regular re-evaluation to achieve the goal of continual improvement [1, 12, 14, 17].

The key principles of clinical audit are the selection of explicit criteria and its cyclical nature. Selected criteria should be easily measured, relevant to the process/outcome of interest, based on available evidence and clearly defined (“explicit”). Where there is insufficient evidence to identify criteria, clinical audit allows for performance comparison with other institutions (“benchmarking”). The tracking of performance through regular audit creates an “audit cycle”, resulting in a continuous cycle of improvement [1, 17]. Absence of a cyclical process, as is common with traditional audit (audit of clinical practice), results in a limited ability to generate continuous improvement [18]. This is an important distinction; due to the absence of explicit criteria (as defined above) and a cyclical evaluation of performance an audit of clinical practice often takes the form of a simple survey of practice and lacks the ability to drive and monitor improvement through a cyclical process.

Clinical audit has been variably defined in veterinary and human medicine [1, 11, 12, 19, 20]. Burgess [11] defines clinical audit as “a quality improvement process that seeks to improve patient care and outcomes through systematic review of care against explicit criteria and the implementation of change” [11]. Since an initial description in 1998 [2], it is unclear to what extent clinical audit is being reported in the veterinary literature and how closely reported studies adhere to the principles of an audit cycle. The purpose of this study was to perform a systematic review of the reporting of clinical audit in the veterinary literature related to companion animals (dogs, cats and horses).

Methods

A systematic review was performed in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines [21]. The Cochrane Collaboration’s Cochrane Handbook of Systematic Reviews of Interventions (version 5.1.0) 2011, which is widely recognized as the gold standard for conducting systematic reviews, was used to guide our processes, along with guidance from the Centre for Reviews and Dissemination [22, 23].

Key search concepts and synonyms were developed by the review team (NR, DP). A health sciences librarian (LT) refined the search strategy by conducting iterative database queries. To ensure a high degree of search sensitivity, a combination of subject headings and text words were used and the search strategy was adapted to the search interface and indexing conventions of each database.

The following electronic databases were searched to identify journal literature and conference papers: OVID CAB Abstracts 1910 to 2014 (searched March 21, 2014 and April 4, 2014), OVID MEDLINE 1946–2014 (searched March 21, 2014), Scopus 1960 to 2014 (searched March 21, 2014) and Web of Science 1899 to 2014 (searched March 21, 2014). Search results for all databases were limited to studies published in English or French. No study design, publication type or publication date limits were applied to the search results.

As veterinary conference papers and proceedings are very selectively indexed in research databases (not all papers may be indexed for a given year, nor is indexing consistent from year to year), and relevant full-text conference proceedings are often not available (library subscriptions are prioritised to peer-reviewed journals), to identify additional unpublished studies we also searched the International Veterinary Information Service (IVIS) website tables of contents for the Annual American Association of Equine Practitioners and the British Equine Veterinary Association conference proceedings, using the terms: “audit”, “governance”, “event” and “incident”. The IVIS website did not provide access to abstracts for any of the relevant small animal conferences. The ProQuest Dissertations and Theses database was searched from 1861 to February 6th, 2014.

The full search strategy is documented in Table 1 and Additional file 1. Search results were downloaded into a commercial reference management software program (EndNote version X6, Thomson Reuters, New York, NY, USA) and duplicates were removed manually. Reference lists of included studies were scanned to identify additional sources.

Table 1 Electronic database search strategy for OVID CAB Abstracts <1910 to 2014 Week 12>
Full size table

Two authors (NR, DP) independently screened all abstracts for inclusion and any disagreement resolved through discussion to reach consensus. For the purposes of this review, clinical audit as defined by Burgess [11] (“clinical audit is a quality improvement process that seeks to improve patient care and outcomes through systematic review of care against explicit criteria and the implementation of change”) was applied as the criterion standard [11]. Inclusion criteria for the final review were: 1. Articles reporting at least two of the four components of the clinical audit cycle (data collection, review of data against defined criteria, implementing change, re-auditing) and 2. Articles that discussed the process or benefits of clinical audit in veterinary medicine. Following identification of included articles, each article was read by two authors (NR, DP) and the following information extracted to facilitate critical appraisal: aim, explicit criteria, study population, collected data, analysis performed, intervention(s) implemented.

Exclusion criteria were 1. Non-veterinary studies and, 2. Clinical audit not related to individual companion animal care.

Results

The initial database literature research yielded 89 potential original articles (Fig. 1). Of these, 19 articles were disqualified because they were limited to the human medical literature. This resulted in 70 veterinary articles, of which 49 were excluded as they did not discuss clinical audit in companion animal care (11 food safety/biosecurity, 11 herd health, 21 business in veterinary medicine, 3 lab animal/wildlife related, 2 incidence of disease, and 1 disaster response). Initial search terms included large animal practice (Table 1 and Additional file 1) in case any clinical audits were performed where these species were treated as companion animals. With the exception of equine practice, none were identified. This resulted in 21 articles included in the analysis. The minority (n = 6) of these described the full audit cycle, and were therefore correctly classified as clinical audit (Table 2). These six articles described 10 separate clinical audits, the majority of which were focused on outcomes (“outcome audit”, n = 8) [2, 14, 20, 24, 25]. There were two clinical audits assessing the process(es) which might contribute to change (“process audit”) [20, 26]. Two articles simply described audits, without documented completion of the clinical audit cycle, such as implementation of change or re-audit [27, 28].

Fig. 1
figure 1

PRISMA diagram summarising literature screening process

Full size image
Table 2 Characteristics of articles reporting clinical audits
Full size table

The majority (n = 13) of the identified articles were reviews of clinical audit and its application to clinical practice (Table 3). The publication dates of clinical audits and reviews are sporadic with no clear pattern of increase in recent years (Fig. 2).

Table 3 Characteristics of articles describing or reviewing the application of clinical audit in veterinary medicine (without reporting clinical audit data)
Full size table
Fig. 2
figure 2

Histogram showing the sporadic publication rate of clinical audits in veterinary medicine (from studies reported in Table 2 [black bars] and 3 [grey bars]). Literature searches conducted between February – April, 2014

Full size image

Sample size calculations

The most prevalent form of data analysis was descriptive, in the form of summary data (e.g. total number of affected cases) or percentages [2, 14, 20, 24, 26]. Where percentages were reported, none of the identified studies applied suitable statistical analyses to evaluate the reported percentage changes, such as Chi squared test or Fisher’s exact test. Analysis (Fisher’s exact test) of published data revealed that the description by Viner [24] of an increased implementation of arterial blood pressure recording was the only clinical audit to show a statistically significant improvement (p < 0.0001; note that an assumption was made that the sample size of the second audit equalled that of the first, as this information was not provided) [24].

In contrast, analysis of reports by Mosedale [2], Dunn and Dunn [26] and Viner [14] did not support the claimed improvements (under the following assumptions: two-tailed test with alpha and beta levels of 0.05 and 0.8, respectively) [2, 14, 26].

A documented reduction in post-operative infection rates was not statistically significant (p = 0.41, Fisher’s exact test); sample sizes of approximately 370 surgeries (sample sizes of 147 and 236 were collected) would be required to identify a 25 % change in infection rates [2]. The sample size during the second audit (n = 6) of Dunn and Dunn [26] was only sufficient to detect a change in antibiotic use of 75 %. The difference between audits had a p value of 0.65 (Fisher’s exact test) and a sample size of 60 would have been required to detect a change of 25 %. Analysis (Fisher’s exact test with Bonferroni’s correction) of an effort to improve peri-anaesthetic mortality rates in rabbits as a result of an unspecified change in anaesthetic protocol revealed p values ranging from 0.51-1.00 for pairwise comparisons between the three audits performed [14].

Discussion

We applied the definition of Burgess [11], that “clinical audit is a quality improvement process that seeks to improve patient care and outcomes through systematic review of care against explicit criteria and the implementation of change” [11], to analyse the identified studies (Table 2). We focused specifically on study design, quality of reporting, evidence of a cyclical process and implementation of change, definition and application of explicit criteria, and systematic review of care including the use of appropriate statistical testing.

Overall, the majority of identified audits suffered from weaknesses in study design and reporting, with resultant limitations on the possibility of identifying real improvement and the repeatability of audits.

Study design

The most common study design employed was a retrospective audit of clinical practice followed by an intervention and prospective audit (Table 2) [14, 20, 24, 26]. The reliance on retrospective data collection is limited by a dependence on record quality; the accuracy and completeness of records specific to the clinical audit subject as well as essential supporting information such as the population of animals presented to a clinic (a necessary denominator for study assessment). Many of the clinical audits identified relied on searches of computerised records to identify cases for audit. This has the considerable advantage of speed but the disadvantage of relying on accuracy in entering the appropriate key word(s) associated with the audit subject. It is likely that accuracy of tagging case records in a clinic and the type of computerised billing or database software employed [20]. Additionally, a key word search could lead to a single animal/case being counted numerous times if the procedure being studied was performed multiple times in the same animal [20, 24]. One approach employed to deal with this issue was to focus the clinical audit on the performance of a procedure, such as number of arterial blood pressure measurements performed, rather than the number of unique cases handled [20, 24]. When working with computerised records, a prospective clinical audit design provides the potential for educating staff in the importance of consistent and accurate records, allowing a more accurate comparison of different audit periods. Where computerised search (retrospective) for the initial audit preceded a prospective audit, it is difficult to identify inaccuracies in computerised data recording without some comparison of case numbers [20]. This problem is compounded when the subject of the clinical audit occurs uncommonly during the study period e.g. chronic kidney disease in cats [20].

Several of the identified studies reported a prospective study design, commonly using a simple handwritten log, with the advantage of focused data collection [2, 20, 25]. Ideally, though often limited by labour costs, data collection would be performed by an independent observer to avoid selective reporting. In prospective studies, a possible confound of participants knowing they are being observed should be considered, as being under observation may change behaviour in a way that influences outcome (Hawthorne effect) [29]. The occurrence and impact of behavioural changes will be easier to track if the clinical audit focuses on the process(es) of care (“process audit”) rather than an outcome measure (“outcome audit”). Where retrospective and prospective approaches are included within the same clinical audit [14, 20, 24, 26], unless processes are clearly described in both retrospective and prospective arms, it is difficult to evaluate the impact of the Hawthorne effect and interpret the results.

The majority of identified clinical audits were outcome-based. While it is tempting to focus on outcome (e.g. return of spontaneous circulation following cardiac arrest), as it is the ultimate measure of failure/success, unless the steps (processes) of clinical care are described and recorded (e.g. frequency of chest compressions) it is impossible to compare clinical audits, replicate studies and maximise learning [20, 25]. Furthermore, where the key process(es) associated with a successful outcome are unknown, limiting focus to outcomes limits evaluation of the clinical audit when an improvement in outcome is not achieved. Conversely, when an outcome audit achieves improvement, unclear documentation of processes inhibits identification of potentially contributing processes [14]. Though clinical audits are not designed to establish a causal relationship, accurate reporting will help identify areas for further improvement and research.

Quality of reporting

In general, reporting quality was highly variable between studies with deviations from expected standards in scientific reports. These limitations make it difficult, if not impossible, to evaluate results, compare findings between clinical audits or replicate clinical audits. Reporting deficits ranged from not identifying the species studied, not describing the criteria/methods of case selection (including omission of the number of eligible cases versus those collected), neglecting a detailed description of the changes implemented during the clinical audit cycle or intervening time between audits [2, 14, 20, 24–26]. Those audits that formed the focus of a paper, rather than being included in a general review, tended to follow a higher standard of reporting [25, 26].

As a result of the limitations described above, as well the limited power of many of the identified audits (see Systematic review of care, below), it is difficult to evaluate the claimed improvements in care. Where failure to achieve improvements were identified common causes included poor owner compliance, selection of an uncommon disease, and limited statistical power [20].

Cyclical process and implementation of change

Of the 15 audits reported in 8 publications, five audits (33 %) failed to demonstrate that a cyclical process had been followed [2, 20, 27, 28]. Furthermore, in 2 (20 %) of the audits that included a cyclical process it was unclear which interventions or changes were implemented between audit cycles [14, 20]. Claimed improvements were made by four of the published audits, though further analysis revealed that only one may have had sufficient power [24]. In contrast to a patient-centred approach, Proot and Corr [25] focused on documenting an improvement in individual performance, using a statistical process control chart approach to track performance (see Systematic review of care, below) [25].

Explicit criteria

The role of identifying explicit criteria as a standard of care allows comparison of performance [1].

Sources of explicit criteria vary depending on the availability of high quality evidence or peer-reviewed consensus guidelines. The identified studies report 3 broad types of criteria: those resulting from consensus discussion within a clinic [2], those based on published guidelines or evidence [2, 25, 26] and criteria based on a combination of these latter approaches [20]. Where explicit criteria were not applied or reported [14, 24, 27, 28], improvements in performance can still be demonstrated but may be misleading if they are below published standards. Conversely, performance exceeding published standards that goes unrecognised limits the opportunity to learn from, or apply, successful practice. Ideally, explicit criteria should be based on available high quality evidence. In the face of the limited evidence available in many areas of veterinary medicine, consensus guidelines combining available evidence with expert interpretation produced by professional bodies can be used [30–33]. In contrast, guidelines developed within a single clinic are likely to be limited by the smaller pool of experience and expertise available.

Systematic review of care

Review of care was limited by over-interpretation of percentage changes without appropriate statistical analyses. In particular, the reporting of percentages in the presence of small sample sizes is misleading as a small change in numbers will have a dramatic effect on the resultant percentage [20, 24, 26]. Subsequent analyses revealed that only one of the identified clinical audits resulted in a statistically significant change [24].

Additionally, while it is tempting to select an easily measurable and emotionally-charged outcome, such as anaesthetic death, as the subject of a clinical audit, where such events are rare it is difficult to achieve an appropriate sample size within a single centre.

Despite the sample size limitations some audits were successful in identifying several potential areas of improvement within each clinic: Mosedale [2] and Dunn and Dunn [26] both recognised the potential for individual performance to have a negative impact on clinical practice through poor aseptic technique, adherence to clinical guidelines and communication (Table 2) [2, 14].

Viner’s [20] doctoral thesis clearly reported the cyclical clinical audit process and iterative approach which was often used to achieve improvements in performance [20]. This body of work is particularly instructive in highlighting factors contributing to unsuccessful clinical audits, including client communication and compliance (clinical audit of a weight loss clinic), case identification and recruitment (clinical audit of feline chronic renal disease management), loosely defined criteria of care (clinical audit of post-operative complications). These failures, as with negative study results in biomedical research, are seldom reported in the literature but have a role to play in learning the pitfalls of conducting clinical audit [34].

One study applied a statistical process control chart to evaluate performance [25]. Such an approach is relatively novel in human and veterinary medicine and originates from quality control procedures for production systems [35]. This, and similar statistical techniques, allows very rapid feedback on performance and can be applied to document the learning curve associated with a new technique, the consistent performance of a skill and as a means to facilitate early identification of poor performance as a result of dramatic failures in care [36, 37].

Failure and success of clinical audits

There were two instances of clinical audit failure recorded in the studies identified; management of chronic kidney disease in cats and performance of a weight loss clinic [20]. These highlighted two different pitfalls: selection of a feasible subject and achieving change in complex systems. In the former, subject selection can be guided by either a pilot study or examination of retrospective data to identify feasibility [1, 11, 38]. When the selected subject is part of a complex system, the risk of failure can be minimised by identifying the processes associated with the outcome of interest [1, 11]. Additionally, an early realisation that changes in human behaviour may be necessary, though hard to achieve, can help design a feasible study. Behavioural barriers, staff communication and record keeping were identified by Viner during evaluation of the clinical audit of the weight loss clinic [20].

Rayment [12] identified the following benefits to implementing clinical audit in veterinary medicine: a reduction of clinical error, the promotion of high standards of care, a gain in the confidence of clients and the promotion of team building amongst staff members [12]. Many of these benefits were described in the identified audits [2, 14, 20, 24, 26]. From these, several important lessons can be learned.

Objective data collection was essential in both identifying sub-standard care and assessing the effect on outcomes when changes were implemented [2, 14, 20, 24, 26]. Without data collection identification of deficits requires marked deviation from normal practice. Such deviation may be difficult to identify when a clinic is assessed as a whole, where the performance of individuals within the group may not be obvious [2, 26]. Furthermore, clinical audit introduces objectivity in recording data, eliminating the inherent risks of relying on memory and confirmation bias [2]. The collected data can form the basis for discussion at staff meetings and allow for input from various parties regarding how care can be improved [26]. Successful clinical audit is predicated on a supportive environment and staff involvement at all levels [39]. This is reflected by the association between a multidisciplinary approach (not limited to veterinarians) and improved practice [2, 14, 20, 24, 40]. Where staff changes occur, audit can confirm that acceptable standards are being maintained, avoiding the assumption that clinic policies will be followed [26]. Where current practice meets acceptable standards, clinical audit can serve as positive reinforcement at a team or individual level [17, 26].

Maintaining the audit cycle over the long term can be difficult. Of the identified studies, a minority were performed for longer than a few months [14, 20, 25]. However, as evident from large-scale audits in production animals, with appropriate support sustained improvement is achievable [41, 42].

Future of clinical audit

Though clinical audit is clearly recognized and implemented in veterinary medicine, the small number of published studies limits firm conclusions regarding the adoption and application of clinical audit from being drawn (Fig. 2). Additionally, the publication dates of identified articles (Fig. 2), do not show a clear pattern of increasing clinical audit publications. However, these data should be evaluated cautiously as it is not possible to know to what extent clinical audit occurs without dissemination in the form of an accessible publication, and in performing this systematic review it was apparent that articles fulfilling the criteria of clinical audit were not identified by the search strategies employed. These potential omissions reflect the well-recognised limitations of article indexing, affecting medicine and veterinary medicine, as well as the poor recognition of clinical audit in veterinary medicine [43]. Accurate article indexing is critical to identification of relevant articles through a structured search strategy. Indexing is based on words included in the title, abstract and key words of an article, and it is likely that many authors are not fully aware of the implications of omitting certain terms (such as “clinical audit”) on the visibility of their article. Related to this, the misuse of the term “clinical audit” adds to confusion regarding its definition [27].

Several recent examples highlight the issue of indexing [44–46]. Hofmeister et al. [46] performed a study fulfilling the criteria of clinical audit [46]. In a prospective, observational pre-post intervention study design they performed an initial audit of anesthesia patient safety incidents measured against explicit criteria (from available literature), implemented changes (related to misinjection, performance of endotracheal intubation and anesthesia machine check), and re-audited, with the outcome of creating a successful decrease in patient safety incidents (from 3.6 % to 1.4 % over the course of over 4000 anesthetic procedures, p < 0.0001). Furthermore, this study exemplified several of the supporting measures discussed above e.g. involvement of several levels of clinic staff (veterinarians and AHTs), prospective data collection and changes in human behavior. This study clearly fulfills the definition of clinical audit, though it was not identified through our search strategy. Broadening the search sufficiently to identify such articles would have prevented a systematic review of the literature as the resultant search would be overwhelmingly broad and unstructured, without guaranteeing that results were complete or reproducible, the cornerstone of systematic review. The solution to this problem is the correct and accurate indexing of articles through education of authors, reviewers and editors [42].

Conclusion

Clinical audits have the potential to generate a measurable improvement in clinical practice and add to evidence-based veterinary medicine. Healthcare, in both human and veterinary medicine, is constantly evolving, indicating an increasing role for clinical audits in the future. By performing audit cycles, the quality of care offered can be monitored, evaluated and continually improved [12]. The number of clearly identifiable clinical audits in the scientific literature remains small, though the number of publications identified is an underestimation due to inaccurate article indexing.

Unfortunately, the majority of published audits failed to generate an improvement in quality as a result of weak study design. Frequent departures from recognised reporting standards limited opportunities to learn from the work performed.