Creating Accountability in Image Quality Analysis. Part 4: Quality Analytics

Introduction

Analytics has been defined as the study of historical data to research potential trends, to analyze the effects of certain decisions or events, or to evaluate the performance of a given tool or scenario, with the goal of improving outcomes through greater knowledge (http://www.businessdictionary.com/definition/analytics/html). While analytics have permeated all aspects of business to date, healthcare analytics represent an area of extremely high and untapped potential. The McKinsley Global Institute (MGI) estimates that big data analysis (i.e., analysis of large datasets) could save the U.S. healthcare system 300 billion dollars annually, with two thirds of that saving in the form of decreasing expenditures by 8 % (http://www.mckinsley.com/insights/mgi/research/technology_and_innovation/big_data).

The data revolution is disrupting established healthcare industries and business models. Information technology (IT) companies such as Google, Microsoft, and IBM have all entered the arena of healthcare data analytics, with the goal of providing increased accessibility and understanding of data to patients and providers. Data is actually becoming a product and service deliverable in itself and is critical in new and burgeoning healthcare applications such as personalized medicine and disease state profiling. The healthcare companies, products, and services which can most creatively innovate through data analysis will triumph in the marketplace and effectively create brand differentiation through data equity. One potential drawback however is that the derived data analytics may not take into account the unique variations and challenges which occur in everyday clinical practice.

To date, the commercialization of healthcare data and the derived analytics have primarily focused on historical data, and serve to improve decision-making through large-scale statistical analysis. In addition to this conventional approach, an additional data strategy would be prospective analytics, which utilize real-time data in combination with historical data to optimize performance. An example in medical imaging practice might consist of a CT scan where real-time quality analytics from the “scout image” is combined with historical data from the patient’s historical imaging database to create an optimal scan protocol, which attempts to simultaneously optimize image quality, radiation dose reduction, and clinical diagnosis based upon the specific attributes of the patient, clinical context, and technology being used. This “combined” approach to data analytics would utilize both prospective and retrospective data analysis at the point of care, taking into account “real world” (i.e., contemporaneous) circumstances. The analyses of these data could also serve as a potential source of collaborative innovation between service providers, technology producers, and IT companies. The ultimate goal is to improve healthcare outcomes by providing service providers with “best practice” data at the point if care, where clinical outcomes can be most critically affected.

Current State of Image Quality Analysis

In its present form, image quality analysis and quality assurance (QA) is largely idiosyncratic, inconsistent, and highly subjective in nature [1]. A number of factors contribute to this relative dysfunctional QA state including changing technology (e.g., analog to digital transition), declining reimbursements, radiology outsourcing (e.g., teleradiology), commoditization pressures, and increasing utilization of imaging services [2–4]. Medical imaging service providers and technology producers have responded to these economic and practice trends by disproportionately focusing on methods to enhance productivity and workflow, which has the potential to adversely affect quality. While everyone shares the common desire to improve quality, the fact that quality in itself is not directly revenue generating has a dampening effect on quality initiatives. In the absence of rigorous data collection and analysis, the desire for quality improvement does not match the reality of everyday practice.

With the exception of breast imaging, which mandates continuous quality analysis through the Mammography Quality Standards Act (MQSA) (http://www.fda.gov/CDRH/MAMMOGRAPHY), the vast majority of medical imaging service QA is left to the discretion of the individual service provider and deficient of rigorous quality analysis. Most medical imaging providers’ QA analytics are focused on measures of operational efficiency (e.g., report turnaround time, scheduling backlog), with limited attention to image quality analysis. If the ultimate goal for healthcare providers and consumers is to improve clinical outcomes, the focus of quality analytics needs to be readjusted to more intensively and objectively evaluate measures of quality and safety. This is beginning to take place with a number of quality-focused initiatives including Pay for Performance [5], Meaningful Use [6], and MIPPA (http://www.govtrack.us/congress/bills/110/hr6331/text). The common denominator to these initiatives is quality-centric data, which can be used for a number of purposes including performance measures, education/training, research, decision support, and establishment of best practice guidelines.

Standardizing Medical Imaging Quality Data

The challenge for any data-driven analysis is creating reproducible data in a standardized format, which in turn can be co-mingled to provide large sample size statistics and generalizable trends. While recording and analyzing data for a single imaging provider has value, the contextual limitations of this data hamper its value. A radiology or hospital administrator who compares quality metrics within his/her own department has the ability to evaluate quality trends over time, compare quality measures between staff, and identify areas of relative quality deficiency, all of which have real value towards the goal of internal quality improvement. The limitations however is that the same administrator cannot accurately determine how these internal quality metrics correlate with those of the outside medical imaging community and national standards, since the data collected is not standardized and uniform in nature. The goal for any data-driven quality initiative should therefore be contingent upon the creation and validation of standardized and objective quality-centric data.

In its present form, most quality data in medical imaging is proprietary in nature and dependent upon the specific medical imaging and information system technologies in use. In the absence of community and industry-wide standards, such as MQSA and BI-RADS in breast imaging, medical imaging providers have no effective means with which to correlate and analyze quality metrics across different technology platforms. As a result, it is imperative for the collective medical imaging community (e.g., service providers, consumers, and technology providers) to create effective and reproducible standardized quality-centric data for longitudinal quality analyses. This standardized data should be consistent across the medical imaging continuum, incorporating all steps, stakeholders, and technologies used in the medical imaging chain, while also taking into account a variety of profiles specific to dataset (Table 1).

Table 1 Individual data profiles for image quality and exam complexity analysis

Methodologies for creating the various quality-centric metrics, infrastructure, and applications have been described [1, 7–9]. One critical and often overlooked aspect of medical imaging quality analysis is the relationship between technical image quality and exam complexity. Clinically and technically challenging medical imaging exams might be expected to have more quality deficiencies than corresponding medical imaging exams which are clinically and technically less complicated. The technical quality of an abdominal CT for an ambulatory and compliant 18-year-old patient in the evaluation of flank pain would in all likelihood be superior to that of an immobile, highly morbid, and less compliant 78-year-old patient being evaluated for suspected mesenteric ischemia. One has to take into account a number of cofounding variables (e.g., patient profile and clinical context) in order to accurately assess image quality and determine best practice guidelines based upon “real-world” limitations.

In order to take exam complexity into account, a quantitative model must be established which take into account a weighted list of variables which collectively contribute to exam complexity (Table 1) [10]. These confounding variables fall into five broad profile groups, each of which contains multiple variables contributing to the overall complexity of that profile group. The five major exam complexity categories include patient, institutional, technology, contextual, and exam profiles. The profile groups and the variables contained within them provide a combined mechanism for determining exam complexity, as well as defining individual metrics for quality analysis. As an example, if one wants to determine the relationship between a specific clinical indication (e.g., right upper quadrant pain) and image quality for all abdominal CT exams performed on patients over the age of 70, they could query the QA database to analyze quality measures at their respective institution for the specific clinical indication of record, exam of interest, and patient age. Now if they want to determine the relative quality performance differences between the three CT devices in use, they can request individual quality analyses for this data fractionated in accordance with the individual CT device of record (as contained in the technology profile data). By doing this, the administrator could determine whether one acquisition device had superior quality measures for the specific exam, patient population, and clinical indication of interest. If the administrator wanted to continue the analysis by evaluating the quality measures of his/her institution relative to a comparable reference group, he/she could request a comparative quality analysis for the specific variables of interest (e.g., clinical indication, exam type, patient population) for all institutional providers of a similar profile (e.g., bed size , geographic location). In this example, the administrator seeks to compare the quality metrics at their facility with other community hospitals located in rural locations with a bed size of less than 100 beds. The value of these profile groups and individual variables is that it provides a mechanism for evaluating quality in the context of exam complexity as well as the individual variables which collectively define the service provider of record. This could potentially become important when defining best practice guidelines, by taking into account a number of confounding variables related to the patient population served, clinical context, institutional demographics, and technology used.

The combined image quality and exam complexity measures could be combined to produce a composite image quality/complexity score, which provides a contextual measure of how image quality measures are reflected by differing levels of exam complexity. It would be categorically unfair to compare image quality scores alone for an outpatient imaging provider catering to relatively healthy and ambulatory patients with quality scores of a hospital imaging provider which provides imaging services to morbid and often immobile patients.

The proposed quality analysis would include the following measures for each exam:

• 1. Composite image quality score (with an optional display of individual quality metrics).

• 2. Calculated exam complexity score.

• 3. Adjusted image quality score (by combining the image quality and exam complexity scores)

The sources of data entry and analysis would also be recorded in the QA database and derived analytics, in order to introduce accountability and reproducibility of the derived analytics. As an example, in the case of subjective image quality analysis, the identification of the individuals performing the quality analysis would be recorded and be subject to its own quality analysis. This is intended to ensure accuracy and consistency of the image quality analysis, provide reviewer education and training tools, and assess the potential for reviewer/institutional bias, while evaluating internal quality control. When computer-generated analysis is being used (e.g., automated image QA or calculation of exam complexity scores), the technical source is also recorded and used for similar internal quality control. This also becomes an important technology assessment tool, in which the derived quality analytics can help guide technology refinement and development efforts.

The combined goals of the proposed innovation is to create a methodology for standardizing quality data which will support subjective and objective means of quality assessment, account for a large number of clinical and technical variables (i.e., profile data), correlate image quality with exam complexity, pool comparable data for large sample size statistical analysis, perform internal quality control analysis to ensure data accuracy and integrity, and utilize the quality database for iterative refinement of the computerized analytical tools.

Quality Analytics and Decision Support

The concept of context-specific quality analysis is important, for it provides insight as to the important relationships between quality and a variety of human, technology, institutional, and clinical factors which can be defined in a series of data profiles (Table 1). While relatively simplistic in nature, this type of analysis is not currently being performed, minimizing the intrinsic value of existing quality analyses. To illustrate how context-specific quality analytics can be used, we will take the example of a chest CT angiogram which is performed in the evaluation of pulmonary embolus. In its current state, image quality analysis would consist of an external measure of image/exam quality performed by a radiologist or supervisory technologist. In the course of this analysis, the reviewer would answer the questions of whether the dataset can answer the clinical question posed (i.e., is it diagnostic?), and whether the overall technical image quality meets acceptable community standards (i.e., is it aesthetically pleasing?). The definition of diagnostic sufficiency and community standards of image quality are highly subjective and often based upon the unique experience and biases of the individual reviewer.

In reality, all chest CT angiograms are not equal. Patient-specific factors such as compliance, body habitus, venous accessibility, and underlying clinical status all play an important role in determining technical image quality, and should be factored into the overall analysis. At the same time, technology-related factors such as the type of CT scanner, image processing software, contrast injector, and contrast agent also play contributing roles in image quality. As a result, image quality assessment should take these contextual factors into account in order to improve the accuracy, reproducibility, and clinical value of the derived analytics. Table 2 outlines a series of context-specific analytics which can be created from the proposed database, taking into account various profile data. This provides the ability to perform a dynamic “apples to apples” image quality analysis, taking into account individual or combined variables of interest, and deriving targeted analytics specific to those variables. An administrator could compare image quality measures at his/her institution with those institutions of a similar profile, or those utilizing the same type of scanner. By doing so, data from institutions or technologies outside the desired purview are effectively removed, providing more specific (and arguably accurate) analytics.

Table 2 Context-specific image quality analytics

In addition to these analytics and derived knowledge, a number of decision support applications can be created which in addition to improved image quality have the potential to improve workflow and operational efficiency, patient safety, cost-efficacy, and clinical outcomes. Unlike traditional quality analytics which are intended to retrospectively assess performance and identify areas of relative quality deficiencies, these decision support applications are designed to intervene at the point of care, which in theory will maximize the derived benefit. In the same manner in which data can be aggregated in accordance with specific profile data to perform targeted quality analytics, the decision support applications can also take these individual data elements into account.

An example of a decision support application which can be derived from the quality database is protocol optimization. Optimal acquisition strategies can be determined by querying the database for examinations which fulfill the desired search criteria and then identifying those exams with the highest image quality scores. By selecting upon the specific exam of interest, the end-user (i.e., technologist) can in turn be presented with specific protocol variables for that specific exam (e.g., acquisition parameters, image processing utilized, contrast administration rate and dosage). If the technology being used is comparable to the exam of interest, the technologist could potentially elect to use the same parameters for the current study (i.e., automated protocol duplication feature). This provides a data-driven methodology for identifying “best practice” in accordance with specific patient, institutional, clinical, or technology search criteria.

Another decision support application is radiation dose optimization. In this application, the desired goal is to maximize the degree of radiation dose reduction while maintaining a specific level of image quality. The end-user can query the database to identify the exam-specific protocol which provides the lowest calculated radiation dose in accordance with the patient profile and technology used, along with the pre-defined image quality score. This provides a data-driven method for optimizing the balance between radiation dose and image quality, while also taking into account available technology and patient-specific attributes.

Decision support can also be used for comparative technology assessment and workflow distribution. For technology assessment, suppose an administrator is tasked with procuring a new image acquisition device (e.g., MRI scanner) and wants to make a decision taking into account institutional-specific and quality variables. One way of doing this is to define the specific search criteria (e.g., institutional demographics and MRI manufacturer) and the database would provide a ranked order of image quality based upon different MRI manufacturers in accordance with the specific institutional profile.

For the workflow distribution decision support application, the end-user could input the exam order of interest and request an image quality profile of the available technologists and technologies. In this example, there may be a complex radiographic study (e.g., scoliosis series) requested and four technologists currently available for three computed radiography (CR) rooms. The supervisory technologist may request an image quality analysis for the specific exam (and corresponding exam complexity score) for the four technologists on duty and three CR units in operation. Based upon these analytics, the technologist may elect to assign the case to the technologist with the highest exam-specific image quality scores, along with a specific room assignment.

Decision support and quality analytics need not be restricted to service providers, but can also be used by consumers of medical imaging services [11]. Through the creation of transparent and accessible QA data, patients and third-party payers may utilize the quality data in the selection of imaging service providers. This may of particular interest in the setting of complex medical imaging exams or patients with unusual or high degrees of morbidity. This profile-specific data-driven image quality selection model could ironically serve as an impetus to reverse ongoing commoditization pressures in radiology, by reprioritizing quality over cost, and return radiology practice to survival of the fittest, as opposed to survival of the cheapest. While this application of the QA database may present many providers with trepidation, in the end it will promote quality and safety, which should remain the highest priority for all healthcare providers.