Driven by recent policy developments that emphasize accountability and data sharing (e.g., Patient Protection and Affordable Care Act 2010; HITECH Act 2009), health information technologies (HIT) are rapidly becoming ubiquitous within the contemporary healthcare landscape. Specific subtypes of HIT support service quality monitoring and can be classified within the “quality management” set of implementation strategies articulated by Powell et al. (2012, 2015). Despite their potential to function as an implementation strategy, these technologies often require their own strategic implementation supports to be successfully installed and effectively used in new service systems (Cohen 2015; Ruud 2015). Although there is widespread recognition of the potential for HIT to usher in new cost savings in healthcare, issues related to technology design and implementation processes have interfered with the extent to which those savings have been realized (Carroll 2015; Leviss 2011; Ribitzky et al. 2010).

Measurement feedback systems (MFS; Bickman 2008) are one type of HIT that support service quality monitoring and directly inform care decisions through (a) the ongoing collection of intervention process and outcome data and (b) data-based feedback to providers (i.e., routine outcome monitoring; ROM). Because it is a core component of numerous evidence-based psychotherapies (e.g., Cognitive Behavioral Therapy, Interpersonal Psychotherapy) and in light of the burgeoning evidence for its positive impact on service recipient outcomes, ROM has been identified a potential minimal intervention needed for change (MINC; Glasgow et al. 2014; Scott and Lewis 2015). MFS represent a leading strategy to enable ROM in mental health service delivery offering two basic functions: (1) They include, or provide the ability to input into the system, quantitative measures that are administered regularly throughout treatment to collect ongoing information about the process and progress of the intervention; and (2) They provide automated presentation of that information to support timely and clinically-useful feedback to mental health providers about their cases (Bickman 2008). Importantly, these functions do not necessitate a freestanding system. Although MFS technologies often exist as stand-alone products, they may also be integrated into other HIT, such as electronic health records (Douglas et al. 2014; Steinfeld et al. 2015). MFS have seen rapid proliferation, with close to 50 such systems identified (Lyon et al., this issue).

Despite the growth of MFS and their widespread endorsement in the mental health services literature (e.g., Garland et al. 2013; Halford et al. 2012; SAMHSA 2012), little empirical work has examined the strategies through which these systems are developed (e.g., contextual inquiry, software design, system usability testing) and implemented (e.g., training, consultation, use of incentives) in real world mental health service delivery settings. The integration and effective use of MFS is beset with the typical implementation challenges common across behavior change efforts (Cohen 2015). However, technology development and roll out are perhaps uniquely complicated and often unexpectedly protracted tasks in which mental health professionals (e.g., administrators, researchers, practitioners) and resource-poor settings are rarely equipped to engage. Unfortunately, no matter how well intentioned or conceptualized, problematic system design and implementation can impede use and undermine the otherwise effective practices that technologies are intended to support (Littlejohns et al. 2003; Karsh 2004; Maguire 2001). Presently, there are few examples and scarce guidance available to inform design/implementation processes as they relate to the popular technology of MFS—including how to effectively engage key business/technology partners who may not be primarily focused on a system’s clinical utility. As a result, development teams tend to remain highly “siloed,” which limits the extent to which innovative and effective approaches are developed and shared, leading to redundant work and wasted resources. Although some authors have begun advocating for the application of user-centered design principles to MFS development (Bickman et al. 2012; Lyon et al. 2015), scant literature exists to describe the iterative process of system design, testing, and revision. Furthermore, although implementation science frameworks have identified key variables and outcomes related to the introduction of innovations into novel contexts (e.g., Aarons et al. 2011; Damschroder et al. 2009; Proctor et al. 2011), they have only recently begun to be applied to the domain of MFS technologies. These issues have significantly hampered the identification and dissemination of “best practice” models for MFS design and implementation.

Special Section Contents

As a component of the larger special issue on outcome assessment and feedback processes, this special section focuses on the design, development, refinement, and implementation of MFS technologies across mental health service delivery sectors. This collection of articles reflects projects in which new technologies or capabilities were developed (e.g., Bruns et al. 2015; Steinfeld et al. 2015) as well as examples of adapting existing technologies to meet local needs (e.g., Lyon et al. 2015; Nadeem et al. 2015). In addition, the articles present process (e.g., Bruns et al. 2015; Gleacher et al. 2015; Lyon et al. 2015; Nadeem et al. 2015) and outcome (e.g., Bickman et al. 2015; Gleacher et al. 2015; Steinfeld et al. 2015) data from MFS development and implementation efforts that reflect the full range of success; from the decommissioning of the Contextualized Feedback System due to ongoing technological and cost issues (Bickman et al. 2015, 2015), to moderate uptake of the Team Management System WrapLogic (Bruns et al. 2015), and to 90 % penetration of a the mental health progress monitoring tool embedded within the electronic health record at Group Health Cooperative (Steinfeld et al. 2015).

Specifically, Bruns et al. (2015) describe their user-centered design informed process of developing an electronic behavioral health information system (EBHIS) to support Wraparound care coordination for youth. They detail the core functions of electronic health records, acknowledging the critical importance of integrating measurement-feedback and care coordination functions into the existing (required) infrastructure and workflow. Bruns et al. (2015) break down their process into replicable phases (e.g., literature review, input from the field, development, etc.) and put forth a theory of impact as it pertains to Wraparound-specific EBHIS.

Nadeem and colleagues present a school-based, community-partnered method for developing and piloting an MFS to support teacher use of classroom practices and feedback to improve student emotional and behavioral issues. They present an iterative approach to integrating user feedback into the adaptation of a relatively low-tech dashboard MFS (e.g., using Microsoft Excel), designed to fit the implementation context per a baseline needs assessment. Pilot data indicated a dose–response relationship between the amount of feedback provided and its impact on outcomes.

Lyon et al. (2015) introduce the contextualized technology adaptation process (CTAP), a systematic approach to HIT development that draws heavily on human-centered design principles and models of implementation science. The CTAP contends that optimizing innovation-organization fit is vital to MFS implementation and that user-centered design provides methods and techniques for making necessary adaptations. Phase 1 (of 5 phases)—contextual evaluation—is presented in the context of MFS development for use in school-based mental health as an illustration of this mixed methods approach.

Steinfeld et al. (2015) highlight the importance of emphasizing the clinical utility of the measurement tool to be featured in a MFS. This led to their development of the mental health progress monitoring tool (MHPMT)—a combination of established measures (e.g., Patient Health Questionnaire) and new items (e.g., regarding therapeutic alliance) deemed relevant to the majority of their population. After ceasing work with an external vendor (due to cost, time, and privacy issues), their team prioritized integrating the MHPMT into their electronic health record system to streamline workflow and relied upon patient completion of a paper copy to be entered by clinicians. Using lean process improvement tools (Steinfeld et al. 2014) following a pilot implementation at a single site, Steinfeld present quantitative and qualitative data from providers and patients on the utility of the MFS and its impact on patient progress.

Bickman et al. (2015) present data from a randomized experiment aimed at determining the impact of feedback amount and timing on youth outcomes in a rural versus an urban clinic setting. Critical issues regarding software delays, in particular, were noted, ultimately resulting in decommissioning of their longstanding MFS: the contextualized feedback system. Despite the implementation challenges, the trial yielded an implementation index (dose of feedback) that could be used across studies to build what appears to be growing evidence of a strong dose–response relationship between feedback and outcomes, but one that may vary according to the implementation approach (e.g., manual versus electronic data entry).

Finally, Gleacher et al. (2015) describe a qualitative analysis of interviews with clinicians at the clinics included in the above-mentioned randomized trial (Bickman et al., this issue) to reveal explanatory barriers and facilitators to the differential MFS implementation outcomes. Interestingly, the proportion of barriers to facilitators did not seem to explain the differential implementation success, but rather organizational and leadership supports appeared critical in the clinic that demonstrated higher implementation rates. Their study suggests a tipping point may be useful in examining the ratio of barriers to facilitators, but that some categories of contextual factors (e.g., leadership) may carry more weight.

de Jong (2015) offers a critical analysis of the special section’s articles and future directions for MFS implementation research, placing it in the context of the key challenges that emerged from quality improvement efforts in the United Kingdom. Specifically, de Jong (2015) suggests that the major issues that may be especially important to MFS development and implementation are: design planning of the improvement intervention (given the heavy technical emphasis of MFS), organizational contexts (with a focus on leveraging appropriate leadership approaches), as well as sustainability (by ensuring relative advantage of the MFS to existing systems) and unintended consequences (the idea that MFS, or ROM more generally, has not been sufficiently tested across populations and settings and may in some cases have iatrogenic effects).

Chorpita et al. (this issue) encourage a step back from the MFS to shed light on the intersection of technology and decision-making. They contend that four evidence bases should be considered if design and implementation are to achieve their full potential: case-specific historical information, local aggregate evidence, general services research, and causal mechanism research. They present a case example to illustrate the interplay and dependency of these four evidence bases, which MFS have yet to fully address and integrate, thus revealing the complexity of the real world clinical context and the rather large design challenges that remain.

Themes from Special Section Articles

Purpose of MFS Implementation

MFS have the potential to support full intervention or service delivery model implementation (e.g., Bruns et al. 2015; Nadeem et al. 2015). In these cases, MFS capabilities move well beyond collection of patient data and provision of feedback to include functions such as communication among providers, documentation, billing, and tracking of practice(s). Importantly, MFS that are built to facilitate the communication among providers, such as the Mental Health Integrated Tracking System, demonstrate improved patient outcomes via collaborative and coordinated care (Unützer et al. 2002, 2012). Although not yet empirically established as superior to more basic MFS, the addition of tracking practice elements or intervention processes may afford providers greater opportunity for individualizing care, particularly in the face of suboptimal patient outcomes. As alluded to in Chorpita and colleagues’ commentary (this issue), MFS may 1 day have the capacity to inform, support, and enhance real-time decision making about how to best proceed with each unique client given the relevant and existing evidence bases.

Simultaneously, each additional MFS capability has the potential to increase system complexity. Rogers’ (2010) diffusion of Innovations theory conceptualizes innovation complexity as a predictor of adoption, in that the more complex the innovation the less likely adoption will occur. In support of this notion, we suggest that MFS complexity (i.e., number of system capabilities) may be inversely related to system spread (Lyon et al., this issue). Indeed, that the majority of the articles in this special section focus on the relatively “simple” task of collecting patient outcomes and providing feedback to providers. Nevertheless, implementation challenges were frequent, suggesting that parsimony may be a critical starting point for this work going forward.

MFS Technology Platform

Most of the contributions in this special section reflect freestanding MFS technologies, and only the Steinfeld et al. (2015) describes the integration of MFS capabilities into an existing electronic health record (EHR) technology. As noted by Lyon et al. (2015), the lack of MFS integration within the existing electronic health record is likely a key barrier to adoption, implementation, and sustainment. Specifically, there is concern that duplicative data entry may be required for each clinical encounter. Moreover, simply having to navigate two separate systems simultaneously in the context of clear time limitations could undermine MFS potential for impact. There is the possibility of technologically achieving interoperability using products such as middleware (Bernstein 1996), which support communication between freestanding MFS and EHRs. However, these solutions typically retain system independence in that providers must still move between systems/screens. This may be suboptimal, considering that the number of clicks to complete a task is one commonly used metric for “ease of use” in HIT (Clauson et al. 2007). Perhaps for these reasons, Steinfeld et al. (2015) demonstrated great success when they moved away from a separate vendor to support ROM and instead made revisions directly to their EHR. Although perhaps more costly up front, Bruns et al. (2015) also saw value in and pursued the creation of an EHR that would optimize MFS and integrate many additional capabilities.

Further, although the present special section is focused on the development and implementation of digital technologies that support outcome monitoring and feedback, we would like to simultaneously caution against racing to “go digital” when analogue methods may also be appropriate. In addition to the potential for increased complexity that accompanies some digital solutions (e.g., computer/internet access, system compatibility), many accounts of implementing MFS or EHRs reflect a continued need for access to paper measures or charts. In the study by Bickman et al. (this issue), for instance, many practitioners opted to use paper and pencil administration when a digital option was available. Studies of EHR use have similarly found that paper-based workarounds are common (Flanagan et al. 2013).

Stakeholder Input into System Development

Each article in this special section that describes a MFS development process references some degree of stakeholder information and feedback gathering. Nadeem et al. (2015) engaged in an adaptation process that made use of strong community partnerships. Their approach represents rapid MFS tailoring that is especially accessible when working with a relatively simple (and easily altered) MFS. Bruns et al. (2015) sought to generate a more widely applicable EHR that could possibly support Wraparound services in numerous settings and so they carefully sought input via a national stakeholder survey, followed by an interactive webinar. Subsequently, they engaged in formative usability testing, much like what is suggested by Lyon et al. (2015). At its core, user centered design (UCD) emphasizes that direct interactions with (and, frequently, observations of) end users are essential to successful technology development and integration. Important to note is that the process of gathering user input also requires some degree of “pruning” to determine which points of feedback are most in need of attention. Lyon et al. (2015) describe their decision-making process surrounding which components of stakeholder feedback to incorporate into a revision of their technology, which may serve as a useful guide to others new to UCD-informed procedures.

Ultimately, the goal for MFS design and development is to optimize both the innovation-organization and innovation-intervention fit. To some, the costs associated with stakeholder testing during design may appear prohibitive; however, personnel and cost savings are likely realized when this approach is taken. For instance, Johnson et al. (l. 2005) reported that costs exponentially increase depending on the phase in which problems are identified. Specifically, fixing a problem in the initial design phase is 10 times cheaper than fixing one in the development/programming phase. It will be useful for future MFS development research to share information on cost in general, and cost savings with respect to incorporating testing with stakeholders in the design phases, especially given the potential need for adaptation to fit each unique organization or context (Lyon et al. 2015).

Conclusion and Future Directions

The articles contained within this special section reflect a current snapshot of evolving efforts to use digital technologies to support ROM practices in mental and behavioral health. Indeed, MFS technologies are constantly appearing and disappearing from the behavioral health service landscape (Lyon, Lewis et al., this issue), ensuring that the accuracy of any published accounts of the state of this field will rapidly diminish. Therefore, in addition to documenting characteristics of specific existing systems, it may be particularly important to identify the overarching and generalizable mechanisms through which MFS impact provider behavior and client outcomes. Considering the centrality of feedback processes to MFS functioning, initial direction in mechanism identification may most appropriately come from well articulated feedback theories (e.g., Kluger and DeNisi 1996; Riemer et al. 2005). As noted in our review of MFS capabilities (Lyon et al., this issue), mechanistic research of this sort could ultimately support the development of more streamlined, targeted, and effective technologies.

Of more immediate concern is the question of what potential individual- and system-level MFS users might look for when making adoption decisions. It may be unlikely that MFS technologies are available “off the shelf” and ready to implement in a novel context. Similar to most psychosocial interventions (Stirman et al. 2013), MFS implementation in a new setting is likely to require a process of adaptation as well as a period during which an organization receives support surrounding effective system use. The major difficulty, however, is that such a process has the potential to result in considerable system “scope creep,” and ultimately a bloated technology, unless clear parameters are placed on the system from the outset. In contrast, more streamlined and parsimonious innovations are likely to be more readily adopted (Rogers 2010). Simpler MFS also decrease the potential for overlap or redundancy with existing EHR technologies, perhaps improving system acceptability. Clearer articulation of the behaviors and functions that MFS should support (as highlighted by Chorpita et al., this issue) may be one promising pathway toward this type of simplification and specification. In the meantime, potential adopters may be well advised to question the allure of a system that boasts the most functions, measurement tools, or data presentation options, and instead think explicitly about the practices that they are hoping to support.

In sum, we hope readers of this section (and the accompanying section in this special issue; Edbrooke-Childs et al., this issue) will find considerable utility in the content of the papers presented. Whether MFS technologies persist for decades to come, or are ultimately integrated into more fully-functional EHRs and disappear from the HIT landscape, may depend on the ability of system developers and implementers to supply adopters with user-centered, cost-effective, and parsimonious products that provide meaningful information to service providers, recipients, and administrators at the moment when it is most needed.