Background

Health information technologies (HITs) are rapidly proliferating with respect to their sophistication and capabilities, facilitated, in part, by policies and funding priorities that actively promote or mandate their use and dictate key aspects of their functioning (e.g., [1–3]). Increasingly, HIT innovations are being developed to support the implementation of evidence-based practices across a wide range of healthcare domains (e.g., patient/disease registries, measurement feedback systems, clinical-decision support tools). Unfortunately, there are few frameworks for effectively compiling, evaluating, and integrating information about these technologies, limiting their utility for implementation research and practice.

Existing methodologies for HIT evaluation are few in number, may be too narrow in their application, overemphasize academic—rather than commercial—products and information sources, are inconsistently driven by theory, and/or do not adequately evaluate the mechanisms through which HITs exert their effects (e.g., [4–9]). Models such as the risk assessment framework (RAF) [10], for example, indicate growing interest in HIT synthesis. The RAF addresses the critical issue of the risks posed by specific smartphone apps by articulating different types and level of risk, based on the probability and severity of harm, app complexity, and a variety of contextual factors. However, the RAF is atheoretical and the niche focus of this framework (i.e., patient risk posed by smartphone apps) makes it difficult to apply to HIT more broadly. In another example, the multiphase optimism strategy [11] is a flexible and compelling method for the development of effective and streamlined eHealth interventions based on the performance of potential intervention components across a series of randomized tests over three phases (i.e., screening and selection of components; refining and fine tuning the level of selected components; confirming the efficacy of the final intervention). MOST shows great promise for the development of individual interventions for specific applications, but has less relevance to the generation of generalizable, context-independent knowledge about an entire class of technology which can both inform new technology design or the selection of existing technologies for adoption.

Furthermore, such methodologies—and their associated theories and frameworks—rarely consider all relevant aspects of intended functions, technology design, and implementation; and lack specific processes for evaluating the utility of technologies derived from multiple sectors. Steadily increasing growth in HIT products suggests that development and dissemination largely occurs outside of traditional academic pathways. Indeed, there is evidence from other fields (e.g., solar cells; [12]) that a “commercialization gap” often exists in which commercial technologies rapidly proliferate with little influence from relevant empirical findings. Simultaneously, considerable technological innovation occurs in the commercial sector, but is not incorporated into academic research. New methods of compiling, evaluating, and integrating information from academic and commercial sectors are therefore necessary to provide guidance to developers, researchers, administrators, and other stakeholders who are interested in making informed decisions about system adoption or development of pragmatic HIT research agendas. Finally, existing HIT evaluation studies commonly lack a guiding theoretical framework [13], which limits their coherence, generalizability, alignment with existing literature, and the extent to which core components of HIT and associated mechanisms of implementation can be identified. A theory-informed methodology for compiling, evaluating, and synthesizing HITs from both the academic and commercial sectors would be a significant step forward for the field of implementation science.

A case example of HIT in need of evaluation and synthesis

Specific subtypes of HITs that support service quality monitoring have been identified among sixteen “quality management strategies” delineated in a recent compilation of evidence-based implementation strategies [14, 15]. Measurement Feedback Systems (MFS [16])—a class of digital HIT that supports the implementation of routine outcome monitoring (ROM) in health service delivery—are one such subtype, and one that is rapidly expanding both in the USA and worldwide, likely due to a growing emphasis on accountability and purposeful uptake by health services agencies [17, 18]. Simultaneously, ROM is increasingly recognized as an evidence-based practice [19–21], and one that may be considered a minimal intervention necessary for change [22]. Consistent with the broad scope of HIT, extant MFS technologies have emerged from disparate sources, serving a panoply of populations and settings, and reflecting diverse academic, service, and commercial interests. The broad scope of these technologies has led to a stark lack of interoperability or consistent standards for use, which presents a significant problem to potential consumers [23]. As a result, information about HIT development, functionality, and implementation readiness is fractured, siloed, and inconsistent, thereby limiting the extent to which available MFS are accessible to stakeholders seeking to implement ROM and inadvertently reducing their ultimate public health impact. No methodologies are available that comparatively evaluate MFS tools. There is a specific critical unmet need to empirically evaluate existing MFS that support ROM—as well as other HIT designed to support the uptake of innovative practices—in order to advance implementation science and practice.

Methodology overview and aims

This article presents (1) a theory-informed, structured methodology—Health Information Technologies—Academic and Commercial Evaluation (HIT-ACE)—to support compilation, empirical comparative evaluation, and synthesis of available information for HIT, as well as (2) preliminary results from a case example application to MFS. In brief, this methodology integrates information from the scientific literature, as well as publicly available commercial sources, and draws on existing theories and frameworks to conduct a competitive analysis. The competitive analysis method is drawn from marketing and strategic business management [24] to structure and prioritize results. Competitive analysis drives innovation by identifying the strengths and weaknesses of existing products within a particular domain, but has not yet been utilized in implementation science.

The current example application of HIT-ACE is specific to MFS used to support ROM in behavioral health. Consistent with the literature in the USA, where this project was carried out, we use the term “behavioral health” as an overarching term for mental health and substance abuse services [25]. This scope allows for a broad (in that it applies to multiple settings in which MFS have been developed and implemented) yet specific exemplar of the HIT-ACE methodology. The methodology is intended to be generalizable to a broad range of extant HIT (e.g., electronic health records, personal health records, mobile health apps, patient/disease registries, clinical-decision support tools) to advance their theoretical and empirical basis and, ultimately, their capacity to advance implementation science and practice.

Methods

Overview of HIT-ACE methodology

The overall goal of the HIT-ACE methodology is to support the compilation of technologies and their associated capabilities to enable empirical and comparative evaluation and, ultimately, to aid consumers and stakeholders in technology adoption decisions that promote sustainment in service systems. We use the term “consumers” to refer to the individuals who make adoption decisions. In the case of practitioner-facing technologies (e.g., MFS, electronic health records), consumers are likely to be service providers or system administrators. Because HITs are likely to arise from both research and commercial sectors, HIT-ACE is designed to integrate information from both in a single methodological approach with relevance to both the selection of existing technologies or the eventual development of novel technologies. HIT-ACE includes four phases: (1) compilation and coding academic and commercial materials to identify capabilities and characteristics, (2) conducting system developer or purveyor interviews to gather more detailed information about development and implementation processes, (3) a process in which putative implementation mechanisms are linked to HIT capabilities, and (4) experimental testing of HIT capabilities and mechanisms. Earlier phases of the HIT-ACE methodology are designed to be more pragmatic and parsimonious, while later phases address prior phase limitations and are more resource intensive. For instance, phase 2 developer interviews are intended, in part, to evaluate and confirm the findings from phase 1 coding. Figure 1 depicts each phase of the HIT-ACE methodology including inputs, activities, and outputs.

Fig. 1
figure 1

Phases of HIT-ACE methodology

Full size image

Phase 1: coding academic and commercial materials

The first phase of HIT-ACE is focused on developing a theory-driven, objective coding scheme for the capabilities of a given technology. A capability is defined as the ability to perform or achieve certain actions or outcomes through a set of controllable and measurable faculties, features, functions, processes, or services. A characteristic is defined as distinguishing trait, quality or property. Phase 1 is intended to address descriptive research questions surrounding the nature of a particular type of technology (e.g., How many different products have been developed that may be classified as this type of technology? What are the most common capabilities of the identified technologies?). Although Phase 1 identifies the frequency with which different capabilities and characteristics are present within a given class of technology, no specific weighting is given to more or less commonly occurring aspects. This is because the presence of a capability in a system may be influenced by a multitude of factors (e.g., when and by whom it was designed) and is not necessarily reflective of the importance or effectiveness of that capability. Further, while it is practically useful to gather information about system characteristics (e.g., price, languages available), it is critically important to gather capabilities which are then ultimately linked to putative mechanisms of practice change (phases 3 and 4). The HIT-ACE methodology draws capabilities from multiple sources—including theory, expert opinion, and review of existing systems—with the goal of identifying a broad set of capabilities for subsequent empirical evaluation.

The creation of the coding scheme is integral to Phase 1 of HIT-ACE. To ensure its coherence, generalizability, alignment with existing literature, and potential for aiding the identification of core capabilities of HIT and associated implementation mechanisms, code development should be guided by theory representing at least three levels: (1) theory specific to the intended functions of a given class of HIT, namely its intended mechanisms of influence on user behavior in the service of optimizing health outcomes; (2) general frameworks for developing usable, effective, and compelling technology products; and (3) implementation models, focused on the ways in which a technology and its associated supports can facilitate initial adoption and sustained use within service systems. As an example, we present the theoretical foundations for the HIT-ACE methodology applied to MFS below. However, the frameworks identified are not intended to be prescriptive, as specific technologies may be best evaluated using different models.

Phase 2: developer/purveyor interviews

Phase 2 of HIT-ACE is intended to confirm the results of phase 1 coding and gather more detailed information about system development, capabilities, and implementation processes than is often possible from the publicly available sources. Phase 2 also expands on Phase 1 with additional descriptive research questions (e.g., Through what processes—and with what types of stakeholder input—were these technologies developed?). Most consumers are unlikely to move beyond the type of information gathering reflected in phase 1 (albeit in a less systematic way), making phase 2 data especially important to aid stakeholders in making informed adoption decisions. It is likely that developers will have additional information that may contribute to the most accurate depiction of a particular HIT landscape. Phase 2 involves a semi-structured interview—based on identified theories—which gathers specific information about the spread of the identified HIT product. Prior to conducting phase 2 interviews, summaries of phase 1 coding are distributed to respondents for review in order to facilitate clarification and correction of the information collected. Additional interview content in phase 2 aligns with the three domains and frameworks described above (i.e., those specific to the intended functions of the HIT; those focused on design and usability; and those emphasizing successful implementation).

Phase 3: linking HIT capabilities to putative implementation mechanisms

In the third phase of the HIT-ACE methodology, specific system capabilities are linked to implementation mechanisms to support future empirical testing (in phase 4). Identification and evaluation of mechanisms of change has become increasingly common in intervention studies [26], but comparable efforts within implementation science are practically non-existent. Mechanisms explain why or how implementation strategies, such as HIT, exert their effects and are critically important for understanding their heterogeneity and contextual dependence [27]. Phase 3 is designed to address exploratory questions surrounding the putative mechanisms through which aspects of a technology affect implementation outcomes (e.g., What mechanisms might be responsible for the anticipated—or previously measured—impact of the technology? Which system capabilities are likely to influence which mechanisms most directly?). Capabilities that are commonly occurring and those present in only a few systems should both be considered during phase 3.

Recent studies have used expert consensus to improve the specificity with which implementation strategies are articulated [15, 28], but very little empirical evidence about the mechanisms through which they operate is available to drive selection or prioritization [29]. For these reasons, phase 3 reflects a Discovery and Linking Process with the goal of (a) identifying the mechanisms through which a specific class of HIT exerts its effects on service provider behavior to optimize service recipient outcomes and (b) linking those mechanisms to the capabilities identified in phase 1 and confirmed in phase 2. Candidate mechanisms should be drawn from existing, multilevel frameworks for mechanism and implementation research (e.g., [30–32]) and confirmed via structured input from experts in implementation and the specific class of HIT via a systematic Dephi process [33] to link mechanisms to capabilities (e.g., EMR warnings [capability] to aid in memory recall [mechanism]), and achieve consensus rank-ordered capability-mechanism pairings for phase 4 evaluation.

Phase 4: experimental testing of capabilities and mechanisms

Phase 4 is designed to address more causal research questions about the impact of specific capabilities on mechanisms (e.g., does an increase in the intensity or dosage of capability X result in a corresponding increase in mechanism Y?). To accomplish this, Phase 4 involves conducting a series of microtrials—rapid and brief tests of the effects of circumscribed environmental or behavioral manipulations on proximal outcomes or mechanisms of change [34]—to evaluate the connection between HIT capabilities and mechanisms identified in phase 3. Recently, microtrials have been used as a feasible method for testing individual parenting techniques (e.g., praise) outside of the context of full treatment packages to determine their discrete merit [35]. Early in a development process, microtrials have the potential to support the collection of “proof of concept” evidence for specific techniques or HIT capabilities using within-subjects, case study designs. Although microtrials represent a feasible approach to engaging in rapid, small-scale prototyping to evaluate components of complex psychosocial processes [36], they have not yet been applied to the analysis of HIT. In HIT-ACE, each microtrial can test the effects of a single, experimentally manipulated capability of the technology on identified mechanisms and provider practice changes expected to result from the technology (e.g., EMR embedded warnings [capability] to remind [mechanism] physician’s about evidence-based prescribing algorithms). It is hypothesized that each capability will have stronger effects when, based on theory, it targets the putative mechanisms of action per phase 3. Overall, phases 3 and 4 aim to utilize and build upon the preliminary findings of phases 1 and 2 while focusing in an increasingly experimental manner on implementation success. In phases 3 and 4, evaluation of which capabilities impact which mechanisms is intended to allow for the identification of a smaller set of core capabilities and, ultimately, more parsimonious and pragmatic technologies.

Application to MFS: guiding theoretical frameworks

As an example, we provide a description and preliminary results from the application of phase 1 of HIT-ACE to MFS below. We focus only on phase 1 in this paper because a detailed example of all phases would exceed the scope of a single paper and because phase 1 is intended to produce a user-friendly and relatively inexpensive synthesis that addresses the academic and commercial gaps. Phase 2 subsequently addresses phase 1 limitations and expands phase 1 findings, while phases 3 and 4 advance a more extensive, novel research agenda.

The identification of relevant theory in the three major domains described above is critical to phase 1 of the HIT-ACE methodology: (a) intended functions/mechanisms, (b) user-centered design, and (c) implementation science. In the application of HIT-ACE to MFS, we drew from leading feedback theories (i.e., Feedback Intervention Theory [37], Contextualized Feedback Intervention Theory [38]) to inform the review because the provision of feedback is a core MFS function (i.e., the process through which MFS purportedly have an effect). Importantly, in selecting this framework, our research team noted a paucity of potential theoretical frameworks that focused on feedback as a critical component. Among other components, these theories articulate aspects of feedback that are likely to make it maximally effective (e.g., that feedback be provided relative to established standards, given immediately, and include actionable information about how to improve performance).

Second, our approach integrates frameworks and evidence from the growing field of user-centered design. User-centered design is an iterative approach to product development that grounds the process in information about the people and settings that will ultimately use a product [39, 40]. The approach is deeply ingrained in the contemporary discipline of human-computer interaction and the concepts of human-centered design, user experience, and experience design, among others, and is increasingly recognized as an essential component of effective HIT development [41]. Although there are no well-developed theories of user-centered design, there exist a number of frameworks, processes, or compendia of techniques for developing compelling and useful products [42, 43]. Frameworks here were selected for their completeness and breadth with the goal of applicability across systems, given that very little is currently known about design of MFS technologies. Importantly, these frameworks do not necessarily come from any specific health domain, but are intended to be relevant to technology development across sectors.

Third, because the implementation, scalability, and sustainment of technologies is generally a goal of HIT development, the incorporation of implementation science models that articulate (a) aspects of innovations that are likely to enhance their implementability (e.g., relative advantage, acceptability) and (b) outcomes of effective implementation (e.g., adoption, cost-effectiveness) [31, 44, 45] are essential to the review process. Although a large number of implementation frameworks exist [46], they vary in the extent to which they attend to the innovation being implemented. Furthermore, most implementation frameworks attend to a wide variety of constructs at multiple levels of a service system (e.g., inner setting; outer setting [31]). In the current project, implementation frameworks were selected that focused most explicitly on characteristics of the innovation that made it more likely to be adopted and sustained.

Application to MFS: coding academic and commercial materials (phase 1)

Scope

MFS were defined as digital technologies that (1) include, or provide the ability to input into the system, quantitative measures that are administered regularly throughout treatment to collect ongoing information about the process and progress of the intervention, as well as (2) provide an automated presentation of this information and, in doing so, supply timely and clinically useful feedback to mental health providers about their patients and caseloads. As indicated above, we defined the scope of the example application to focus on behavioral health MFS. The behavioral health literature has seen considerable MFS advancement in recent years, including a growing number of MFS publications [47], special issues [48, 49], theories outlining intended use [38], and methods of MFS development [50, 51].

MFS compilation/identification

Given the likelihood of information gaps between the commercial and academic sectors, identification of MFS—and of HIT more generally—requires a multi-method process. We engaged in a systematic process to identify MFS in each sector including: (a) Google searches using the search strings below, (b) database searches (i.e., Web of Science, PsycINFO, PubMed), (c) soliciting systems from known experts (i.e., researchers who have published in the area), and (d) email related professional listservs (e.g., Society for Implementation Research Collaboration and the Association of Behavioral and Cognitive Therapies’ Dissemination and Implementation Science Special Interest Group and Technology and Behavior Change Special Interest Group). Search strings used to uncover relevant MFS included “measurement feedback system,” “measurement-based feedback system,” “clinical measurement feedback system,” “outcome monitoring system,” “routine outcome monitoring”, and “system to track outcomes.” Subsequent to MFS identification, all available materials was collected including websites and relevant literature. Identified MFS had to fit the MFS definition described above and had to report facilitating ROM in behavioral healthcare. Identified systems were excluded if it was not possible to locate a website or literature describing the system. If two systems from the same development team had different names or branding but clearly contained the same set of capabilities, the more recent of the two systems was included. In total, as of December 31, 2014, the final list included 49 systems for review.

Coding scheme development

Although the HIT-ACE coding scheme—composed of capabilities and characteristics—is primarily deductive in nature, the initial coding scheme was created to evaluate MFS via both inductive and deductive processes [52]. First, system capabilities were extracted from the literature to establish an initial framework. This stage of development incorporated aspects of Feedback Intervention Theory [37], including capabilities related to “feedback timing” (i.e., along what schedule does the MFS provide feedback) and “standard-gap feedback” (i.e., does the MFS provide feedback relative to a norm or standard). In addition, relevant literature on electronic health records and other HIT led to the inclusion of additional characteristics, such as the existence of “patient portals” as well as characteristics such as whether the systems were compliant with the US Health Insurance Portability and Accountability Act (HIPAA) [53–55]. Next, two representative MFS were reviewed to ensure comprehensiveness of the coding system and identify additional characteristics or capabilities. This process led to the inclusion of characteristics related to system marketing (e.g., the availability of a promotional demonstration) and specific data elements tracked within the systems (e.g., service recipient critical events). Finally, stakeholders (e.g., agency administrators either using or desiring to use MFS) and experts (e.g., researchers developing or publishing on MFS) reviewed the list and provided additional characteristics or capabilities to be included.

The final step of coding scheme development required the concrete operationalization of each MFS capability or characteristic in order to support their consistent application, including positive and negative examples. For example, the capability, “Service Recipient Portal to View Outcomes,” was defined as, “A treatment view tailored specifically for service recipient. This must be intended for the recipient, rather than a common portal that can be shared, with a separate login from that of the service provider.” Definitions and examples were refined during the pilot phase of the coding process. Complete tables of the capabilities and characteristics, including definitions, are provided in Tables 1 and 2, respectively. When combined, Tables 1 and 2 represent the complete phase 1 coding scheme.

Table 1 List of capabilities
Full size table
Table 2 List of characteristics
Full size table

Pilot and revision

Prior to the formal coding process, the coding system was piloted to test its feasibility and accuracy and to drive refinements. This process was completed by six of the seven authors (A.R.L., C.C.L., M.B., A.M., F.F.L., and N.J.), who independently applied the coding scheme with representative systems and the primary MFS information source (typically a commercial website). After completion of pilot coding for one system, coders met to compare codes, discuss questions, identify possible new codes, and resolve conflicting ratings. Next, revisions were made to the coding scheme to accommodate additional relevant capabilities and to consolidate similar or redundant capabilities. For example, the capability “Alerts to Providers,” defined as “critical information brought to the attention of the mental health provider in a way that differs from how information is usually presented in the system (e.g., flags/highlighting, emails, pop-ups, etc.),” was complemented with a new code, “Alerts to Others,” in order to capture system alerts directed to other recipients (e.g., supervisors).

Coding process

As the next component of phase 1, the coding scheme was applied to evaluate identified technologies. In the case of these MFS, all systems were coded by two independent research assistants using the same information sources, including system websites and empirical articles (see Table 3 for a full list of systems and primary coded source). As needed, the two raters consulted with the investigative team to resolve discrepancies. To facilitate coding feasibility, efficiency, and consistency, a decision was made by the investigative team to code the most information-rich source in an attempt to maximize accuracy in the information accessed. The primary information source was then reviewed in full, including all videos, links, and appendices where relevant, by two independent raters.

Table 3 Representative publications and information sources
Full size table

A consensus coding process was used in which raters reviewed materials independently and then met to arrive at consensus judgments through open dialogue [56, 57]. All capabilities and characteristics were assigned a “0” or a “1” to reflect its absence or presence, respectively. Coding was intended to reflect the viewpoint of a potential consumer in that, if the presence of a capability was unclear or not mentioned, a consumer would likely assume that the system lacked this capability. Therefore, a “0” was given if the information about a capability/characteristic was too ambiguous, the capability was not mentioned at all, or if it was stated explicitly that the system did not have this capability (although few systems explicitly mentioned not having a particular capability). A “1” was given if the MFS clearly had the capability. See Figs. 2 and 3 for a visual representation of this process. The dichotomous ratings for each of the capabilities were analyzed to determine the frequency with which various capabilities were represented across the systems. In addition to dichotomous coding, additional qualitative information was collected to further describe the capabilities and characteristics identified as present or absent. For example, specific information was collected about the assessment measures contained within each MFS, types of standard-gap feedback provided, and the extent to which provider interventions/practices were tracked.

Fig. 2
figure 2

Coding process example

Full size image
Fig. 3
figure 3

Coding process example

Full size image

Bibliometric data

We also examined the number of published articles referring to each system. Relevant articles were obtained by searching the full system name—and acronym if applicable—in quotation marks in Web of Science and PsycInfo. Articles were included if the research described, used, developed, tested or reviewed the system, but not included if the article only cited or made reference to the existence of a system. This search method was intended to provide a snapshot of the frequency with which the system was discussed in the literature, but not necessarily an indicator of the scientific rigor of the system. Additionally, locating these articles made it possible to systematically track the emergence of a subset of these systems over time. Exact data related to the inception of these systems were difficult to locate due to the fact that public information often does not include creation or development history. However, all relevant articles for each system were reviewed through the process described above and the earliest publication date was treated as a proxy for the “inception date,” as it is likely that the publication date of a relevant article is correlated with actual system development.

Results and discussion

Using phase 1 of the HIT-ACE methodology, a total of 49 systems were identified through the systematic review of product websites and scientific literature, as well as solicitation from experts and professional listservs. Although HIT-ACE does not make assumptions about the importance or effectiveness of capabilities or characteristics based on their frequency of occurrence, the application of phase 1 provides a detailed account of the contemporary MFS landscape for subsequent evaluation. The consensus-coded data for all systems revealed substantial variability in system capabilities, with only 12 of the 56 capabilities and characteristics present in at least half of the systems (Table 4). One explanation for the relative low frequency of some capabilities and low overlap across systems is that the coded capabilities were drawn from multiple literatures. Knowledge of these literatures may have varied based on the professional backgrounds of the individuals involved in any given development team. Publication and dissemination of our findings could help spur and focus development of capabilities currently lacking in the majority of MFS.

Table 4 Capability and characteristic frequency
Full size table

However, another explanation is that the wide range of system capabilities and characteristics observed reflects the diversity of needs that often exist across populations and settings and the range of solutions designed to meet those needs within MFS technologies. For example, the two MFS that also function as electronic health records offered fewest additional capabilities (average per EHR MFS = 6.50 versus average per MFS = 9.77). Additionally, MFS that are not available for purchase or acquisition and were therefore likely created solely for the needs of a specific (research) project, possessed fewer capabilities than MFS that are publically available (average per MFS not publically available = 16.27 versus average per MFS publically available = 20.63).

It is also possible this variability in capability representation across MFS is due to the fact that little is known about which capabilities are core/central and which are auxiliary given the dearth of literature focused on mechanisms of MFS. It is unsurprising that “Tracks Standardized Outcomes” and “Outcome Monitoring to Provider is the Prime Function” were the top two capabilities being represented in 93.9 and 91.8 %, respectively, given that these are the defining features of an MFS. However, the next most common capabilities were “Library of Measures to Choose From” (represented in 71.4 % of MFS), followed by provision of “Summary Reports” and “Displays Outcomes as Graphs” (both represented in 67.4 % of MFS). MFS-relevant theory (e.g., FIT, CFIT) would not support this pattern of most common capabilities. Rather, if guided by FIT [34], we might except to see “Compares Treatment Outcomes to User-Defined Goals,” “Corrective Feedback from System,” and Immediate Feedback Timing” as common among MFS. Subsequent phases of HIT-ACE (phases 3 and 4) are designed to address this gap and determine which capabilities drive system performance in terms of improving the implementation of health innovations and patient outcomes. Therefore, at this time, it is premature to conjecture whether the current array of MFS is fit for purpose.

Finally, because capabilities and characteristics were only coded as present if they were explicitly mentioned in the materials reviewed, it is possible that systems actually contain more capabilities and characteristics than were captured and that there is more overlap in capabilities among existing MFS than documented in the current findings from phase 1. It is for this reason that phase 2 is designed to compare and confirm our preliminary phase 1 coding in addition to gathering more detailed information about the systems. However, it is unlikely that validating the phase 1 results would reveal equivalence in MFS capability representation.

Beyond capability presence, our coding revealed the number of relevant academic articles for each system, which ranged from 0 to 231 (median = 1.5; mode = 0) with 31 systems appearing in the literature and 18 systems having no associated published literature. Three or more relevant articles could be located for 25 systems (51 %). This wide range of published articles for each system reflects the considerable variability in the empirical foundation for available MFS, perhaps indicating the disparate goals (i.e., commercial, academic, etc.) that led to the development of each system. Furthermore, despite high representation for some systems the median and modal values suggest that the “average” system has received relatively little attention from the academic community. These findings highlight the importance of reviewing sources of information beyond the academic literature (e.g., websites, promotional materials) to adequately represent the scope of any specific class or type of HIT. These results also suggest that more research is necessary to evaluate the utility of most of the systems identified. Interestingly, although 31 systems were identified in the scientific literature— and far fewer of which were empirically tested to assess system impact—41 MFS described their system as “evidence based.” Clearly, while there is mounting evidence for the effectiveness of using MFS to support ROM in general, individual systems are likely to vary regarding (a) the scientific rigor with which they were developed and (b) the availability of data regarding their impact on patient and provider behavior or actual clinical outcomes.

The published articles also made it possible to approximate the emergence of MFS technologies over time; however, it is important to note that this was only possible for the subset of MFS that were represented in the academic literature (see Fig. 4). Of the 31 systems referenced, the first appeared in 1995. Inception was fairly steady since then, with an average of 1.55 systems emerging in the academic literature every year and relatively linear growth. This apparent steady growth and current volume of MFS technologies demonstrates the need for the development of review methodologies like HIT-ACE, as well as the need for ongoing systematic reviews.

Fig. 4
figure 4

System emergence over time. A graph displaying the chronological emergence of systems that were represented in the literature

Full size image

Conclusions

Given the steady proliferation of HIT—emerging from disparate sources and reflecting diverse academic, service, and commercial interests—a theory-informed, structured methodology to support collation, identification and empirical comparative evaluation for specific classes of HIT would benefit implementation science, health service administrators, and the HIT marketplace. HIT-ACE is one such methodology that can be employed to conduct comprehensive reviews and competitive analyses of any given class of HIT. We described the four phases of the HIT-ACE approach and an example application of phase 1 to available MFS technologies. Preliminary findings clearly demonstrate the utility of HIT-ACE to depict the scope and diversity of MFS beyond what can be identified through a traditional review of the academic literature. Our coding process also revealed substantial vagueness and inconsistency among the publicly available information, highlighting the importance of HIT-ACE phase 2 developer interviews to compare and confirm phase 1 coding and gather additional in-depth information about existing MFS. The variability of capability representativeness from phase 1 (and likely phase 2) highlights the importance of phases 3 and 4, which are designed to facilitate a nuanced—and ultimately experimental—understanding of the mechanisms through which classes of HIT support implementation and impact user behavior and subsequent patient outcomes, reflecting a critical gap in the research and development landscape. This methodology is expected to spur innovation by promoting transparency (e.g., presenting all MFS capabilities currently available on market) in a development space that is historically siloed and fractured.

HIT-ACE is not without limitations. It is a meticulous and labor intensive process that requires significant time and resources to execute, especially for any class of HIT with substantial history, spread, and/or diversity of goals and features. Although the methodology intentionally aims to precede more resource-intensive and potentially expensive methods (e.g., phases 2–4) with those that are less resource-intensive and more feasible (phase 1), it is acknowledged that these methods may be impractical in some contexts, particularly those occurring outside the research setting. Furthermore, by the time all phases of HIT-ACE have been completed, it is possible (if not likely) that new examples of the target technology may have emerged. HIT-ACE is intended to provide a “snapshot” of a given class of technologies at a particular moment in time and to synthesize the available information. Although the HIT landscape tends to evolve rapidly, a detailed “snapshot” is nonetheless valuable given the lack of alternative methodology to gain a comprehensive understanding of the existing marketplace for any given class of HIT.

It is important to note that the nature of the identification process likely did not yield a comprehensive listing of all MFS. However, due to the fact that Google is the most commonly utilized search engine [58] and has been found to produce better results than alternative commercial search engines [59], we believe that it is representative of what a potential consumer or stakeholder may utilize to find systems. Additionally, the English-based databases and search terms utilized—and Google algorithms that prioritize the return of local information [60]—likely resulted in an over-representation of systems based in the USA and potentially excluded other systems originating internationally. Consequently, there is potential for the Phase 1 findings presented herein to be biased toward a US perspective. Nevertheless, we maintain that the HIT-ACE methodology is a potentially powerful tool for systematically evaluating any class of HIT for a variety of research and development purposes as well as health services implementation and dissemination efforts. Related to the latter, HIT-ACE could support the development of rich, regularly updated databases for a given class of HIT that would greatly benefit users trying to select a HIT from among the array of options available or researchers interested in improving technologies through a detailed understanding of their mechanisms of influence.

Future directions

The results presented herein pertain to HIT-ACE Phase 1: Coding Academic and Commercial Materials. Detailed mixed methods evaluation of MFS capabilities and implementation is currently underway in Phase 2: Developer/Purveyor Interviews. As a component of this we will be able to link information about MFS to measurable dependent variables related to implementation outcomes (e.g., system penetration into the marketplace, views from consumers and/or stakeholders about system acceptability, feasibility, or appropriateness, etc.). Subsequent to this, we intend to engage in Phase 3: Linking Putative Implementation Mechanisms to HIT Capabilities, and Phase 4: Experimental Testing of Capabilities and Mechanisms. Phases 3 and 4 of HIT-ACE applied to MFS will be the first attempt, to our knowledge, to isolate core components of HIT, map them to associated mechanisms, and conduct systematic evaluations. These phases will be critical to determine which capabilities maximize MFS outcomes and should be the focus of future development activities. We also envision the development of a living MFS review repository where consumers and developers of all stripes could search for system capabilities and characteristics based on their settings and interests to support adoption-related decision-making and further innovation.