A Consumer Health App Works Well. . . But, Diffusion Is Not So Easy

The number of mobile health apps is exploding, though few of these tools have been evaluated for effectiveness. Furthermore, even apps certified for their impact may not diffuse widely. App outreach efforts may lack “human touches,” when encouraging potential users to adopt. We (SHE and PC) built a nutrition app for low-income clients of community food pantries and confirmed its effectiveness in a randomized field trial. We then collaborated with a variety of community organizations in six diffusion efforts, inviting our sites to design interventions that incorporated human touches that they could sustain beyond our collaborations. We independently monitored the experiments’ impact. Collaborators’ diffusion efforts that deployed several human touches were judged more successful than trials that executed few. But, only half of our collaborating organizations embraced the most essential human touches, practices we judged were part of a User-Friendly Approach. A fundamental irony can stymie projects that create information technologies to help empower people, such as eating more healthily. The modern technology’s ultimate success may rest on ancient skills of nurturing users at the moment of downloading. In a high-tech world, it is the human touch that will make the difference between widespread adoption or not.

Informed sources within the digital health industry estimate that more than 250,000 health-related apps currently flood the marketplace. As many as 200 apps are added daily. Although the number of app builders continues to grow worldwide, actual downloads of health apps by potential users have flattened in recent years (Research2guidance 2016).
Perhaps one reason is that few apps have been tested empirically and found beneficial in terms of clinical outcomes (Chib and Lin 2018;Lan et al. 2019); lack of proof weakens product claims for apps (Greaves et al. 2018). Many food and nutrition apps have been rated for their use of behavior change techniques and other features (Mauch et al. 2018). Sometimes an app, when tested, compares unfavorably against the use of traditional paper-based media for health results (Ledford et al. 2018). In general, half of health app downloaders say they stop using these tools for a variety of reasons (Krebs and Duncan 2015).
Even for apps with tested positive outcomes, however, barriers arise to downloading and sustained use. This paper identifies such barriers in the case of a nutrition app that was built with the help of abundant formative studies among users . The app also demonstrated its effectiveness in a rigorous randomized controlled trial (RCT), the gold standard for evaluating interventions in health care delivery . We have uncovered a number of "human touches" that improve the use and effectiveness of the app. Human touches include the amount of time the app outreach staff devote to downloaders, whether the staff are present on a regular basis so that they can provide familiar support, answer questions, and engage the user in conversations about the app, whether the staff have an emotional connection to their clients, and much more (discussed below).
Human touches, extraneous to an app itself, may be critical for diffusion efforts promoting use, or if absent, may relegate an app to obscurity. Perhaps our experiences should forewarn others who would build digital health tools for the general public: Even when tools earn top grades for usability and effectiveness, your job is only partly finished.

Materials and Methods
VeggieBook, Our Cooking App for Low-income Households Vegetables in the diet contribute to disease prevention (Slavin and Lloyd 2012), yet consumption of these foods remains far below recommended levels (Moore and Thompson 2015). Underconsumption is especially acute among lowincome populations, although recent efforts have improved the supply of fresh produce at food banks and their community pantries that serve economically deprived households . A significant, remaining hurdle to increased vegetables in the diet is that many family cooks do not know how to prepare varied and appealing servings (Clarke and Evans 2015). Many low-income cooks ask for help with simple recipes that use ingredients at hand (Harmon et al. 2007).
We responded by first developing a tablet-based tool that produced on-the-spot recipes and food tips tailored to the expressed needs of each pantry visitor. These individualized, take-away booklets dramatically increased incorporation of fresh vegetables into meals and snacks at home . We discovered, however, that pantry volunteers' staffing of the tablet could not keep pace with the flow of clients. Consequently, we (SHE and PC) turned to building a mobile app that capitalized on experiences with the tablet system. Many formative studies contributed to the design and content of our app ).
We tested the app's effectiveness in a randomized controlled trial with nearly 300 households. Briefly, app-enabled household cooks began preparing 38% more vegetable-based servings than the control group, and showed other and sustained signs of healthier food use across a 10-week study period .
The VeggieBook has also been certified as an effective obesity-prevention intervention by a national consortium of public health agencies representing (1) the USDA's Food and Nutrition Service; (2) the National Collaborative on Childhood Obesity Research (NCCOR), a partnership between the Centers for Disease Control and Prevention, the National Institutes of Health, the Robert Wood Johnson Foundation, and the US Department of Agriculture; and (3) the Association of SNAP Nutrition Education Administrators (ASNNA). The app is included among evidence-based interventions as part of the US Department of Agriculture's SNAP-Ed Toolkit (https://snapedtoolkit.org/), following a publicly transparent review and evaluation process (Federal Register/ FIND 2017).
The app is free and accessible in both iOS and Android formats and can be used in either English or Spanish. Search for "VeggieBooks" in the App Store and "VeggieBook and SecretsBook" in the Play Store; the app's icon is a green steaming pot. The number of screens that a user accesses depends on content preferences that one signals when inside the app. Figure 1 shows two screens from the app.  Figure 2 shows a food pantry client using her VeggieBook. Our successes in field testing VeggieBook notwithstanding, we had questions about the best means for diffusing our app. Others have observed a wide gulf between the building of powerful digital tools that aim to improve people's access to social goods and the sparse use of those tools in practice (Toyama 2015). Accordingly, we reasoned that collaborations with local organizations offered a promising route toward app adoption and sustained use. Local organizations serving lowincome people know how to identify and attract the clientele for whom VeggieBook was built. These organizations provide tangible services and goods, so clients trust and rely on them.
Organizations have volunteers and staffs who can assist with diffusion .

Our Diffusion Efforts
We enlisted four collaborating organizations for piloting ways to diffuse VeggieBook. These were a food bank in the Midwest, a food bank in the Southwest, a school district in Southern California and its affiliated Promotoras (community health workers), and a statewide program of community-based health fairs in a Western state. In an effort to identify sustainable methods of diffusion, we issued an open-ended invitation to each collaborator: Propose to us ways to promote our app to your clients that (a) align with your organization's mission and capabilities; and (b) that are practical, so that you could continue diffusion after assistance from us has ended.
We had previously gained some experience with diffusion. We had created protocols and materials for a Midwest state and its SNAP-Ed outreach (United States Department of Agriculture 2017). Promotions of the app took place at a number of food pantries across the state. Thus, we could advise our four new collaborators about designing effective flyers for potential downloaders, the need for bilingual signage, training of outreach staff for their tasks, equipment needs, and other operational issues.
In addition, we had deep know-how with deploying VeggieBook from our randomized field trial, and also with studying impacts of our earlier tablet-based tool . Altogether, this background allowed us to advise our collaborating organizations about human touches that seemed likely to affect rates of downloading our app and persistence in using it. These touches are listed in Table 1.
We urged our collaborators to consider these human touches when designing their own ways to diffuse the app, but we did not require them to include any of them.
We offered to reimburse our collaborators for expenditures for equipment (e.g., printers, wi-fi hotspots, tables, chairs) or materials (promotional pop-up banners, flyers, etc.) and to help train staff via Skype or in person. Our staff traveled to each organization and we arranged many conference calls. We offered VeggieBook in both print-enabled and non-print versions. We offered to modify the app's content in accordance with the collaborators' wishes, to insert a collaborator's logo, thereby branding our app as a collaborator's creation, and to provide a screen icon for opening the app that was unique to a collaborator. We supported server costs and other centralized budget items. Each of the downloading activities was Table 1 Human touches that improve the diffusion of VeggieBook a. Whether a fresh vegetable that is featured in the app is distributed simultaneously at app downloading stations. Distribution of a relevant vegetable spurs immediate use of VeggieBook, so users quickly gain experience with its benefits. b. Whether the same diffusion staff are present on a regular basis and can provide familiar app support beyond the first download day, as clients revisit locations for social services. c. Whether clients have enough time and attention at downloading, so they can learn the app, practice, and ask questions. d. Whether diffusion staff have an emotional connection to the community of app users they serve (belong to the same groups, come from the same neighborhoods, or share other potential bonds). Such connections may increase the likelihood of "emotionally-intelligent" exchanges while promoting the app (Goleman 1995). e. Whether printing of app output is enabled. In the print-enabled mode, clients can wirelessly print their tailored, four-color booklets of recipes and Secrets on-site, complete with individualized and illustrated covers. Using the non-print version of our app, on the other hand, clients access their content only on their phones. f. Whether a household cook and one of his/her children train together on the app. Our field test disclosed that children like to participate, and even guide adults' use of VeggieBook (Neffa-Creech et al. 2019).
g. Whether diffusion takes place in an "educational atmosphere," such as a cooking class, or in a briefer and less intimate setting, such as the delivery of social services to crowds of clients. h. Whether clients receive follow-up contact (text or voice) from diffusion staff, regarding their experiences with the app. Fig. 2 A pantry client creates her VeggieBook, as others line up waiting for a food distribution to open executed on several days or at several locations, so that some stability of operations could be achieved.
Our four collaborating organizations developed six different diffusion efforts. See Table 2 for community partners and short descriptions of their efforts.
Importantly, as our criterion variable, we independently judged each download effort's success from high to low. To assist us in making these judgments, the staff at each downloading effort asked site visitors or clients whether their household owned an Android smartphone (the only app platform supported at the time), whether they had their phone with them, and whether they wished to download VeggieBook. Gmail account names of downloaders were recorded. The staff also estimated the total visitors at the site during each downloading opportunity. Using these paper records for each trial, the proportion of completed downloads was noted as a percentage of Android phones on-site. And, the proportion of visitors approached, as a percentage of site visitors or clients in attendance, was noted.
For actual, subsequent usage of VeggieBook, the app's backend analytics panel electronically recorded each user's every touch, time of activity, and Gmail account. Usage for each site was indexed by dividing the total of usage occasions (opening the app) during the week following downloading, excluding hours when downloading activity had been taking place, by the number of that week's downloaders at the site. Since trials extended across several weeks, three percentages were compiled for each trial week: (a) completed downloads divided by eligible phones on-site, (b) number approached divided by number in attendance, and (c) usage occasions divided by number of the week's downloading Gmail accounts. The numbers of cases on which these percentages were based varied widely, depending on the traffic of people at downloading sites.
Blinded to the sites' deployment of human touches, the authors examined these data and sought to rank order our collaborators, from 1 through 6, by equally weighing the three percentages. We found that disagreements between rankings could be eliminated by using four categories of results instead of six. We rated two download efforts as "low" in success; one as "low-medium"; two as "medium-high"; and one as "high." Though we calculated these metrics for estimating the trials' successes, we continually emphasized to our collaborating organizations that diffusion required investing time and attention in each potential downloader. We stressed that promoting the app and trusting that visitors would download it later was likely to be a failing strategy. Our mantra was: a few committed adopters of VeggieBook are preferable to many people who were merely informed about the app. A few persistent app users are preferable to a crowd who download, but never open their VeggieBooks.
Finally, in order to discover details about the app usage, we conducted telephone interviews with "positive deviants," downloaders who consulted VeggieBook especially often, according to the backend electronic records. In conversations, 2 to 3 weeks after on-site participation, we asked intensively about food habits before and since getting the app. The study of positive deviants has proved helpful to many social experiments for diagnosing elements of an intervention that work well, or others that appear to impede success (Singhal and Dura 2017).

Results
We examine the relationship between the number of human touches and the success with which we judged each outreach effort. Organizations embraced between none and five of our human touches, out of the eight touches we had shared with them. As far as we could tell, none of our collaborators developed human touches that were not on our list. The median use of human touches was 2.5.

School district in East Los Angeles
Promotoras (lay Hispanic/Latino community members who receive specialized training to provide basic health education in the community without being a professional health care worker) staffed a download station at the entrance of a middle school during parent after-school pick-up hours. Non-profit in the West that presents large-scale health fairs statewide The health fairs include a wide variety of medical testing and information booths. Table 3 shows that the application of human touches correlated very highly with the efforts' success. The incorporation of at least three touches virtually assured that the download effort would later be judged at least "medium-high" in success. Two or fewer touches resulted in "low" or "low-medium" success.
Why did this correlation emerge? Did the incorporation of any key human touches account for results? Table 3 shows three diffusion efforts as standouts. What were these three successes? The Promotoras took great satisfaction in helping potential downloaders, dedicating time to ask each visitor about kitchen experiences or family food needs. The Promotoras made certain that they satisfied each client's questions or uncertainties about how to use the app before turning to the next visitor. The same Promotoras staffed all diffusion sessions. The Promotoras were middleaged residents of the school district where their outreach took place. They resembled clients demographically and were deeply committed to their community's health.
Similarly, the Southwest food bank's two programs-a cooking camp for kids and its nutrition outreach to pantry clients-had limited numerical goals from the outset, yet scored high in effectiveness. In both outreach efforts, staffing was stable and drawn from the same community as potential users. Disseminators lavished time on their clients.
From our close monitoring of these successful efforts, we feel that we can identify four human touches as most helpful. We believe that it was helpful, if not decisive, to (1) distribute an app-related vegetable that invited downloaders to make early and concrete use of the app; (2) carefully and strategically staff the download station; (3) allow time for practicing with the app, and not just downloading it; and (4) provide users with printed output, though this appears not to be a permanent advantage.
All three outreach efforts that scored "greater success" provided paper, and none of the less successful efforts did so. Perhaps the advantage from printed output was due to the reasons that led us at the outset to build the app to provide tangible VeggieBooks and SecretsBooks. Many household cooks, one-third in our RCT experiment testing app effectiveness , preferred using printed recipes compared with the recipes on the phone, a device that can get wet and greasy in the kitchen. Also, in households with just one smartphone, many cooks cannot commandeer the device during times of value to them, such as visiting food pantries, preparing meals, or shopping.
Despite these results, however, we believe printing emerges as a temporary advantage that fades with increasing exposure to our app. Intensive post-download interviews we conducted with "positive deviants" in each diffusion revealed that app users often consulted the app throughout the day, when planning meals and when shopping, for example, and not just in kitchens at dinnertime. Furthermore, even when paper output had been distributed, after a few weeks many "positive deviants" began taking phones into their kitchens. Paper, in other words, seemed to facilitate early app use, but diminished as a significant factor. We might conclude differently, of course, if we had conducted interviews with marginal users of our app.
In sharp contrast were the efforts that incorporated few or no human touches. The health fairs promoted the app to large crowds of attendees, seeking broad exposure, and these promotions reaped meagre results. Volunteers were white and middle-class, whereas many health fair clients were lowerincome Hispanics. Similarly, a Midwest food bank decided to staff the downloading outreach with a young, Caucasian, male, AmeriCorps worker. Many of his clients were middleaged women who were heads of households; most were nonwhite.
And, what about the one outreach effort that had poor performance, despite human touches? This was a produce distribution held in partnership with community health clinics. What defeated this effort? The pace of the produce distribution was quick, following clinic visits, allowing little time for staff to approach clients and hold their attention. This effort had a high rate of downloaders but a low rate of actual users.
All of our partner organizations spurned the human touch of following up with voice calls or texting to clients who visited their sites, offering help in downloading or in using the app. None of our collaborators had an interest or capacity for this follow-through, at least during these trial periods.

Discussion
Our consideration of whether specific human touches were necessary or sufficient conditions for success has prompted deeper thinking about competing diffusion strategies. These thoughts hinge on distinctions between the User-Friendly and the Mass-Approaches to diffusion.
Overseeing the six efforts reminded us that potential collaborators (community organizations in a variety of shapes and sizes) differ according to their aspirations to achieve quantifiable and impressive outcomes. Some organizations, as Muller (2018) has eloquently written, are obsessed with recording large results that they can publicize and use to justify support through grants and other external rewards. In some of our collaborators' diffusion efforts, this preoccupation translated into their urgency to promote the app to many people and secure their promise to download it eventually, instead of sitting patiently with fewer clients while they downloaded, learned the app, and experimented with its features. The former strategy allowed a collaborator to claim statistics showing an impressive reach for its effort, at the expense of intensity of app use by each downloader. Watching our collaborators plan and execute their diffusion efforts suggested that several human touches combine comfortably into a constellation of complementary features. We distinguish between the User-Friendly Approach and the Mass Approach. In the User-Friendly Approach: And, to these, we add a fourth feature that experience showed to be important: & Disseminators are thoroughly knowledgeable about the app and use it frequently when cooking themselves. This broadens conversations and builds credible dialogue between disseminators and clients.
These might seem obvious features of a User-Friendly Approach, self-evident strengths. But on examination, just two of our four organizations were willing to dedicate themselves to at least a majority of these aspects of user friendliness.
We learned that in the world of non-profit outreach where we experimented, a User-Friendly Approach is elusive and may be perceived as unnecessary. In this, social service agencies resemble many consumer retail establishments, where customer service can often be an afterthought instead of a valued component of brand equity (Moon 2010). To remain objective, a caveat is in order: It is conceivable that mass dissemination by merely advertising the app's existence offers a preferable strategy on grounds of cost, more practical than the User-Friendly Approach, which demands an intensive investment. We have not yet collected outreach expenses, Mass vs. User-Friendly, with which to document this trade-off for achieving persistent app usage, per dollar.
We also recognize that our diffusion trials were small. The correlation in Table 3 could be the result of many factors that our collaborators brought to the project, including organizational capabilities, unique characteristics of their clients, and other aspects of the settings where diffusions took place.
Our experiences promoting VeggieBook mirror the cautions voiced by Toyama (2015). Technologies are often scalable, he has observed, but the human skills of patiently nurturing people to use digital tools are not scalable. Current practice in social services often privileges innovation in technologies over the dogged work of outreach and of sustaining successful interventions. There is no shortage of studies of dietary interventions sited in community organizations like food banks or their pantries, but most of these projects are one-shot trials rather than continuous programs (Seligman et al. 2018;An et al. 2019).
Perhaps many causes contribute to the meagre rate of success by health apps and other technologies devoted to the advancement of social goods. If our experience has any lesson to share, however, it is that building information technologies to help meet urgent needs will disappoint unless diffusions show potential users, in a friendly fashion, how they can realize improvements in their lives. Such demonstrations rest for success on ancient, not cutting-edge, skills.