How might we go about assessing and, where needed, improving the quality of psychotherapy practice delivered in usual care? Given the numbers of mental health specialists practicing in the U.S. (recently estimated at one-half million), the diversity of professional training and licensure that establish credentials to practice therapy, a broad range of practice settings, and variations in the types of individuals treated, the task of answering this question seems daunting (Institute of Medicine 2006a). This special issue includes a range of studies that have made first steps toward tackling this challenging task. Not surprisingly, given the groundbreaking nature of this research, these papers overwhelmingly raise more questions than they answer. A large part of the value of these studies is lessons learned that inform next steps to assess and improve the quality of psychotherapy in usual care.

Quality of Care Framework

Donabedian’s Quality of Care model, commonly used in healthcare quality assessment and improvement research, is useful for putting the issues addressed in this special issue into broad context (Donabedian 1988). In this framework, healthcare can improve the expected outcomes of individuals who access care through two broad domains: (1) structural aspects of care, such as facilities, staff competencies, equipment, organization of care; and (2) processes of care—the specific evaluation and treatment encounters experienced by the individual. In order to improve the quality of healthcare, we need to establish causal links from structural aspects of care to processes of care, and from processes of care to outcomes. With key links established to outcomes, we can assess relevant structural or process components of usual care, target healthcare improvement efforts to those areas where care deviates from what the evidence holds to be best for producing health outcomes, and evaluate whether our quality improvement efforts had the intended effects of closing the gap between usual care and best practices, and improving outcomes. Although it is not an easy matter to establish causal linkages between aspects of healthcare structure, process and health outcomes, the field of health services research, and in particular, clinical effectiveness research, is contributing to a growing body of empirical evidence that has identified some of these important relationships for specific populations and medical, including behavioral health, conditions.

Within the Donabedian framework, the papers in this issue are concerned with the problem of how to assess the processes of mental health care when psychotherapy is a component of treatment delivered in usual care. They are not trying to establish the causal links between processes of care and outcomes. Instead, they are trying to describe usual care psychotherapy practice in all of its diversity, and determine how patterns of usual care relate to what has been learned from clinical literature about effective psychotherapy practices. If researchers are successful in developing methodologic approaches and measurement tools to assess usual care psychotherapy, then these tools can provide a platform upon which to develop feedback systems to support routine quality assessment and continuous quality improvement.

Measurement Challenges

The measurement challenges are considerable, as these papers attest. There are numerous approaches that can be taken, and trade-offs to consider in each decision regarding approach. The investigators have wrestled with these decisions and made choices in their approaches, helping us consider the variety of approaches and strengths and limitations of alternatives. Below, we summarize five questions regarding measurement approach that emerged as prominent themes in this issue.

1) At what level should the unit of observation be set, given the possible range from small and finely grained behavioral interaction units to more encompassing and broadly defined therapeutic orientations? The research examples in this issue all have selected a middle-ground, with the unit of observation being techniques, strategies, or goals that are specific components of broader therapeutic approaches. Garland and colleagues provide an insightful discussion of why this level of observation is preferable over other choices that might be made (Garland et al. 2009). Kelley and colleagues suggest that many therapists can more easily report on the topic of their focus in a therapy session (e.g., a behavioral problem) than on the techniques they use (Kelley et al. 2009).

2) How broadly should assessment cover the range of practice in psychotherapy? Approaches in this issue have ranged from a broad scope that attempts to cover most of what is observed in mental health care for children (Garland et al. 2009; Hurlburt et al. 2009; Kelley et al. 2009) to a more narrow focus on identifying elements of prominent therapeutic approaches in the treatment of depression (Hepner et al. 2009a; Miranda et al. 2009). Differences in breadth of the assessment approach reflects differences in study aims: a broad assessment was suited to the goal of describing natural variation in psychotherapy practice and the associations between that natural variation and outcomes; the narrow focus on specific psychotherapy techniques for depression treatment aimed to understand the extent to which evidence-based practices were delivered in usual care. Both approaches appear to be fruitful ways of generating knowledge about usual care psychotherapy practice.

3) From whose perspective—clinician, client, objective observer—should the assessment of psychotherapy be made? Each of these perspectives has strengths and limitations, with examples of each provided among the studies in this issue. The perspective of the clinician gives insight into the intent of practice that other perspectives cannot capture; the client perspective is unique in reflecting understanding or awareness that clients took from their therapeutic experience, and the observer perspective is unique in capturing observable behavior and verbal interactions that occurred during therapy. There is no gold standard against which to evaluate the validity of alternative perspectives. Each perspective is relevant and it remains to be seen whether one or another, or perhaps a combined approach, will emerge as a more useful and practical way to assess psychotherapy practice in usual care.

4) Can assessment approaches incorporate important aspects of the changing, dynamic nature of psychotherapy over time? While some therapy elements or techniques may be present in every therapy session (e.g., agenda setting or agreeing upon ‘homework’ in a cognitive behavioral therapy session), some therapy elements may be more commonly used during a particular stage of therapy. For example, within cognitive behavioral therapy, educating the client about the cognitive model for depression would typically be a primary focus during the first few sessions of therapy and would receive less emphasis in later stages of therapy. One-time assessments that look back on the full course of therapy to date may need to take into account the number of sessions the client has received at the time of the assessment. Alternatively, approaches that characterize a single session in a client’s therapy may have difficulty capturing key aspects of therapy that change over time.

5) Is it possible to develop low-cost, low-burden assessment tools that could feasibly be used routinely and widely as part of information system support for improving the quality of mental health care? While we want the answer to this question to be “yes”, it is clear that the assessment task is sufficiently complex to warrant further measurement development, evaluation, and refinement before we can hope that acceptable tools will be available. We join others (Bickman 2008) in emphasizing the need for and critical importance of developing information systems that provide feedback on outcomes and processes of care to support mental health care improvement. Further research is needed on methods and measures that are appropriate components of these information systems.

Opening the Black Box

Psychotherapy, as practiced in usual care, has long been considered a black box—a process in which the inner components and processes aren’t easily known. Some studies have reported on the number of psychotherapy visits received by those getting mental health care, as an indicator of the quality of care that was received (Wang et al. 2005). While information about number of visits is often available in administrative data or medical records, it is clearly limited for understanding the content and quality of the therapy received. Researchers can reasonably infer that those receiving very few visits were unlikely to have received even minimally adequate care, but cannot distinguish poor from high quality therapy among those receiving many visits. Clinician credentials, such as educational background and professional licensure, indicate a minimum level of professional competency, but provide little indication of the therapy processes practiced by the clinician. Certification in a specific therapeutic approach is one approach to ensure clinician competency to practice it. For example, the Academy of Cognitive Therapy (www.academyofct.org) certifies clinicians as competent in cognitive therapy following an application process that includes an independent reviewer coding an audio taped session using the Cognitive Therapy Rating Scale (Young and Beck 1980). While certification does not guarantee that a clinician will continue to competently practice cognitive therapy, it does indicate that the clinician has demonstrated an ability to do so. Many clinicians report that they use eclectic (that is, multiple) or integrative (combinational) approaches (Norcross et al. 2005), but it is not clear how techniques from multiple approaches might be selected and combined in any specific episode of treatment. So while clinician training, licensing, and credentialing are integral to the production of therapist workforce skills and capacities, we cannot infer the content or quality of psychotherapy from knowing them.

What do we see when we look into the black box? Several articles in this issue have provided a first empirical glimpse into usual care psychotherapy. Hepner et al. (2009b), examined usual care psychotherapy for adults with depression, finding that clinicians often used some techniques associated with evidence-based therapies—Cognitive Behavioral Therapy (CBT) and Interpersonal Therapy (IPT)—but other core techniques from these therapeutic approaches were also among the least endorsed. The results suggested that many clinicians are using an eclectic approach to treating depression, that is, using some techniques from a variety of approaches. In contrast to pure forms of CBT and IPT, the effectiveness of such eclectic approaches is largely unknown. Brookman-Frazee et al. (2009), similarly found that empirically supported practice elements were used with low intensity in usual care psychotherapy for youth. And these authors found no clear pattern of characteristics associated with delivery of empirically supported practice elements, highlighting the complexity of understanding usual care psychotherapy practice. The work from Hurlburt et al. (2009), suggests that clinicians may find it difficult to accurately report on their own therapy practice in the absence of training in how to do so, based on a study that compared these reports to coded observations. Taken together, these attempts to open the black box suggest that a wide range of techniques are used in usual care psychotherapy, and that clinicians are generally not using empirically supported approaches with the same purity and intensity demonstrated to be effective in clinical trials. Consistent with this view, Landry et al. (2009) found that, among a national sample of adults who reported receiving some form of mental health or substance abuse care, a minority claimed to have received specific counseling consistent with components of evidence-based treatments for the most common disorders.

Future Research

The measurement and description of psychotherapy in usual care will provide a platform for addressing other important research questions. Below, we list several of these questions.

What is the relationship of workforce training and competencies to psychotherapy practice patterns? A professional coalition on mental health workforce issues made a strong case for reform in graduate training to better prepare mental health specialists for practice in today’s healthcare environment (Hoge et al. 2005), and SAMHSA has called for discussion of “An Action plan for Behavioral Health Workforce Development” (Annapolis Coalition 2007), but to our knowledge, no federal efforts are yet underway to implement the plan. Proposals to reform workforce training and development implicitly assume the link between structural aspects of mental health care (workforce composition and competency) and the processes of care (delivery of treatments that are more appropriate and effective to those individuals who seek care). This is a reasonable assumption, but there are important questions to be addressed regarding the kind of workforce training and development that is needed to improve usual care practices. In addition, it is likely that other organizational supports and incentives are needed to facilitate the effectiveness of a well-trained workforce.

What is the relationship between practice patterns and outcomes? Processes of care should also be linked to outcomes. Specifically, we expect that practices based on the best available clinical evidence, such as evidence-based guidelines, will lead to better outcomes in usual care than when practice deviates from evidence-based guidelines. Some studies have demonstrated that specific approaches to improving the delivery of mental health care in primary medical care settings results in better outcomes for patients (Meredith et al. 2006; Wells et al. 2000), but there is still much to be learned about how to improve the outcomes of psychotherapy in usual care, in both specialty behavioral health and general medical care settings. Controlled clinical studies are necessarily limited in their generalizability across broader client populations. Studies of outcomes of psychotherapy in usual care can identify areas in which the evidence base needs to be strengthened to address clinical challenges. Socially and culturally diverse clients, complex health and mental health conditions, and diverse preferences and life circumstances are all realities of usual care that may moderate the effectiveness of evidence-based care. Identifying those for whom existing evidence-based practice is ineffective will point to important ways that clinical practice needs to be improved. Studies of the relationship between psychotherapy practice and outcomes in usual care could also suggest promising new treatment approaches that deserve further evaluation.

How can structural aspects of the healthcare environment improve the practice of psychotherapy? There is growing understanding of the importance of aligning financing incentives, and of putting into place organizational structures and informational systems that support best practices. Producing high quality care requires more than a well-trained workforce; mental health specialists need an environment in which excellent care is rewarded and the environment is organized to support the delivery of excellent care (Institute of Medicine 2006a). In spite of a growing literature on the ways that financing incentives, organizational models, and informational systems can improve the quality of health and mental health care, we know very little about structural aspects of the health care environment that can support the delivery of high quality psychotherapy.

Will development of more informed consumers of psychotherapy improve the practice of psychotherapy in usual care? The importance of consumer-oriented care, where consumers are informed partners in their decisions about and management of healthcare is increasingly recognized in general and mental health care (Katon et al. 1995; Mueser et al. 2002; Wagner et al. 2001). For psychotherapy, it is perhaps obvious that consumer engagement and participation in the therapeutic process is essential, and will influence the unfolding of the psychotherapy process as well as its results. But there are many impediments to the free flow of information about and transparency of mental health care to consumers, especially given the stigma and discrimination associated with mental illness (Institute of Medicine. 2006b). Further research is needed to understand how to promote more informed and actively participating consumers of psychotherapy.

Conclusions

These seminal papers begin to lay the important foundation needed to improve the quality of psychotherapy provided to our nation’s vulnerable populations in need of mental health care. Although much work remains ahead of us, these papers begin to develop a methodology for defining the psychotherapy that is provided to millions of Americans each year. As we understand psychotherapy in practice, we develop the tools needed to routinely monitor the processes and outcomes of mental health care. These tools provide the platform to improve care and provide relief from suffering and opportunities for growth of the many individuals suffering from mental disorders.