Drawing upon the authors’ leadership and participant observation of two participatory agenda-setting projects, this paper contributes to our understanding of what participatory agenda-setting looks like in practice. The first project included a series of workshops to co-design a potential future model of existing health systems and explore future policies regarding greater continuity of care for people with behavioral health issues in a large southwestern city. These workshops elicited input from representatives in the following service health and social sectors: primary healthcare; behavioral healthcare; mental health consumer-operated services programs; law enforcement and the courts; mental health advocacy organizations; and state and county government. The ensuing model mock-up supported conversations around cross-sectoral data sharing and transparency, as well as prospective healthcare policies and their likely effect on different patient populations.
The second project developed a data-driven model to quantify and forecast the relationship between increased educational attainment, economic outcomes, and workforce alignment in Arizona. The research team conducted over 30 community engagements with educational stakeholders, such as teachers, principals, and superintendents of early childhood through upper secondary education, as well as local city council members, economic development specialists, state government representatives, and non-profit educational organizations. The ensuing interactive model allows users to create, compare, and discuss attainment and economic futures for different populations at state and regional levels. Both projects were facilitated out of the DT. Table 1 summarizes key comparative elements of the two projects, discussed in more detail below.
Project one: cross-sector continuity of care for patients with serious mental illness
In February 2019, the first author joined a research team working out of DT which sought to reduce healthcare data fragmentation in Arizona and create a more holistic system for continuity of care for patients with behavioral health issues, specifically serious mental illness (SMI). The first author’s role in this project was to assist in the evaluation of the workshops’ success in eliciting and collecting participant feedback. The research team had convened a community stakeholder meeting the previous December to engage representatives from various sectors that interact with individuals with serious mental illness, inform them of the project’s goals and timelines, obtain feedback about the projects general goals and objectives, assess levels of interest in engagement, and most importantly, patient-level data capacity and willingness to share those data with the research team (with all appropriate administrative safeguards).
Approximately 50 community and state-level stakeholders were identified through a combination of convenience and snowball sampling and included representatives from the following service health and social sectors: health care; behavioral healthcare; mental health consumer-operated services programs; law enforcement and the courts; mental health advocacy organizations; and state and county government. As a result of past projects, the research team was generally aware of the patient-level information capacities of these various sectors. Attendees provided the project with specific feedback, recommendations, and differential levels of interest and engagement in the project. After the December kickoff meeting, stakeholders were encouraged to sign up for one of four follow-up Data Design Workshops, each addressing a specific topic area such as social determinants of health, mental health in the criminal justice system, and high utilizers. Expertise in the topic area was not required for attendance; stakeholders could select topics which were of tangential interest to learn more about data and approaches among other sectors. The general agenda for each workshop was to orient stakeholders to the overarching objective of combining data sets into a comprehensive, interactive data visualization and then present them with a semi-interactive mockup, eliciting feedback through a series of facilitated conversations both with the research team and with each other.
Workshop structure and evaluation
Each Data Design Workshop lasted approximately 2 h to allow for in-depth discussions about the types of analyses and graphical visualizations of the data they would find informative and actionable in enhancing their understanding of service delivery trends. During these workshops, the research team provided the stakeholders with an update on data acquisition and data curation activities and provided each with a series of documents that detailed the data elements that were contained in the various data files represented on screen. Stakeholder feedback was then integrated into the visuals for the next workshop, and so on. In addition to the research team presenting and observing, each of these Data Design Workshops (approximately 7–10 stakeholders per workshop) were assigned a project staff facilitator and note-taker to capture the ways stakeholders discussed data arrays and data visualizations that could make use of the cross-sector data, as well as the ways in which they contemplated real-world, actionable service delivery issues. Research team members conducted participant observations during each Data Design Workshop. After each Data Design Workshop, the research team met to debrief what they had seen and heard and compared observations about individual stakeholders as well as the ideas and themes which dominated differentially themed workshops. In addition to participant observation during the Data Design Workshops, the research team conducted semi-structured summative interviews with a select group of stakeholders after the final workshop, including 11 men and women from behavioral health, public health, criminal justice, and community liaison programs. All of the stakeholders who consented to be interviewed served at a directorial position or higher within their respective organizations. Each interview lasted between 30 and 90 min, resulting in over 100 pages of interview transcripts. An in-depth analysis of these interviews is explored in a separate report, but they inform the implications for participatory agenda-setting discussed below.
Resultant data visualization
The data visualization which resulted from the Data Design Workshops was semi-interactive; that is, some screens were only conceptual in nature whereas others were connected to real data sets supplied by the research team. Arizona State University’s Center for Health Information and Research maintains one of the most comprehensive Medicaid claims data sets in the state and thus used it as a starting platform for participants to imagine where their data could contribute to a larger, more comprehensive model connecting healthcare to public safety, criminal justice, and so on. In its final form, the model sought to connect these conversations by allowing users to select a population and see the distribution of physical, behavioral, and pharmaceutical claims (Fig. 3), visualize the differences between those with and without serious mental illness within this sub-population, understand the relative cost of different claims for SMI and non-SMI populations (Fig. 4), compare the number of claims by illness, gain a geographic understanding of how the location of illnesses claimed overlap with crime and housing data (Fig. 5), examine social network connectedness of patients across different sectors and governmental programs, and high-level costs for different programs.
In contrast to the Achieve60AZ model described in the next section, the healthcare visualization served primarily as an imagineering mechanism to enable stakeholders to imagine where and how their data could contribute to a higher resolution picture of the current connections between various sectors which all treat patients with serious mental illness. This project was still in the design stages of what a future model could look like, what research questions were of interest to the various stakeholders involved, and what data would be required to support the future creation of such a model.
Whereas the Achieve60AZ model was supported by publicly available data sets from the US Census, Bureau of Labor, and Office of Economic Opportunity, there are a number of additional data privacy and sharing conversations which would need to occur to connect sensitive health and criminal justice data across the state. While stakeholders in the workshops were aware of the privacy demands, the focus within the workshops was on what kind of model would be useful to them and the kinds of questions they would want to ask of it. If and when the project moves beyond this imagineering stage, requirements regarding data sharing and privacy would be essential.
Roles of researchers and level of citizen participation
The Continuity of Care project was funded by an external public health philanthropy program to which the research team had proposed in early 2019. As such, the research question and plan had already been determined and approved for funding, creating constraints for the direction of any future development guided by community members and stakeholders participating in the Data Design Workshops. It is for these reasons that the roles of the researchers were primarily that of facilitator and reflective scientist . That is, the research team selected participants, facilitated and encouraged expressions of diverse viewpoints within the workshops, and sought to balance competing interests within the context of the workshops when, for example, the public health official’s narrative regarding ambulance drop-offs in emergency rooms contradicted that of the emergency medical technician. Throughout the workshops and the debrief process, other members of the research team were reflective scientists, analyzing the outcomes of the ongoing data analysis in light of observations and reflections regarding how the analysis and data visualizations were received by the stakeholders. In this capacity, the reflective scientists sought to connect application-oriented knowledge vocalized by the stakeholders into possible best practices for future cross-sector engagement and data sharing, as well as the provision of continuity of care, and to connect these insights back to existing academic knowledge and policy.
Given the constraints of the research plan set forth by the research team and approved by the funding agency, the level of citizen participation which best describes that of the stakeholders and community representatives in the Continuity of Care project is that of placation . The workshop attendees were recruited specifically for their expertise in matters of serious mental illness and its relation to primary healthcare, behavioral healthcare, mental health consumer-operated services programs, law enforcement and the courts, mental health advocacy organizations, and state and county government. Placation is characterized by citizens “hav[ing] some degree of influence through tokenism is still apparent…[they] advise or plan but retain for powerholders the right to judge the legitimacy or feasibility of the advice.” The strategic selection of organizational representatives from the social sectors listed above could be interpreted as tokenism, though the funding schedule and workshop plan put legitimate constraints on the research team such that the priority was to include as many expert opinions as possible, from as many community organizations as possible which reflected the diversity of social sectors necessary to understand challenges of continuity of care across those sectors for people with serious mental illness. These experts were recruited for their applied, on-the-ground knowledge of serving people with serious mental illness, as well as their insights regarding best practices and existing policy for coordinating across social sectors in regard to data sharing. However, at the end of the workshops, it was at the discretion of the research team which elements of stakeholder knowledge or insight would be included in the data visualizations, as well as ongoing analysis of continuity of care from academic and policy standpoints, a power dynamic characterized by the placation rung of Arnstein’s ladder.
Project two: Achieve60AZ and the impact of postsecondary attainment
Founded in 2016, Achieve60AZ is a community-based initiative to increase educational attainment for all Arizonans, guided by the belief that a more highly trained and educated population will attract more business, boost economic well-being, and increase social gains. As of 2019, the Achieve60AZ Alliance was comprised of 40 local Arizona governments, as well as over 75 community, business, philanthropic, and education organizations. The central goal of this effort is to reach 60 percent adult postsecondary attainment in the state of Arizona by the year 2030. The state is currently at 35 percent postsecondary attainment for adults aged 25–64, according to the most recent U.S. Census. This percentage includes associate’s, bachelor’s, and graduate degrees and does not currently include certificates and licenses. The organization funding the project, the Arizona Board of Regents (ABOR), is a leading member of the Achieve60AZ Alliance. As the governing board of the state’s three public universities, ABOR seeks to provide leadership and a unifying voice on key higher education issues, as well as influence public policy through advocacy and initiatives founded on evidence-based research (www.azregents.edu). The research team was charged with examining education and learning as a dynamic, interdependent, nonlinear process and not just as an outcome variable. The authors led the project and managed model development, data collection and analysis, design of the user interface, and facilitation of stakeholder engagement with the model.
In November 2017, ABOR formed a collaborative partnership with DT to create a data-driven model and interactive visual display to explore and quantify tradeoffs of different pathways to achieving 60 percent adult postsecondary attainment in Arizona. The first opportunity to understand existing conversations regarding postsecondary attainment among education stakeholders came in December 2017, when the Arizona Commission for Postsecondary Education hosted a two-day conference titled “Developing Arizona’s Human Capital: Acceleration to 60.” The authors attended the conference to listen to the ways in which thought leaders conceptualized and described the state of Arizona’s education system, as well as capture perceived barriers and solutions to educational attainment. The conference also provided an important network opportunity for the authors to identify key stakeholders who could provide subject matter expertise regarding the state’s education system and history of existing interventions geared toward increasing postsecondary attainment. Though the first several stakeholder engagements were with attendees of the December conference, the authors engaged in snowball sampling to elicit additional viewpoints and areas of expertise regarding Arizona’s education system, how the education stakeholders currently conceptualized barriers to, as well as benefits of, attainment, and what data existed to support a model which explored alternative pathways to attainment in Arizona.
Stakeholder engagement structure and evaluation
Over the next 14 months, the research team facilitated 32 participatory agenda-setting engagements with over 200 stakeholders from education, economic development, and local government representatives and community groups in the DT Drum (Fig. 1) to model different pathways to achieving 60% adult postsecondary educational attainment by the year 2030. Consistent with the DT Convergence Framework described earlier, ABOR and the research team wanted the model to be informed by stakeholder ideas, questions, and input. Thus, the goals of these engagements were to (1) create a preliminary model which attended to conversations captured during the Arizona Commission conference, (2) facilitate stakeholders' interaction with the evolving model, (3) elicit their feedback on the relevance and usability of the data, display, and analysis, and (4) observe how the model contributed to the depth and complexity of conversations between stakeholders. The iterative process of model development and stakeholder engagement required constant coordination and communication between the research team and the software developers, data scientists, and graphics designers within DT. Thus, in addition to the 32 stakeholder engagements, the authors and DT team met every other week to clarify expectations and make sure stakeholder interests were reflected in ongoing model development and visualization. The participatory agenda-setting process unfolded for over a year between November 2017 and February 2019. The authors each spent over 100 h in the field with stakeholders and collected over 350 pages of meeting minutes and feedback, in addition to field notes and photographs cataloguing, the evolution of the model.
Resultant data model
Throughout the model development process, the research team sought to understand the collaborative process by which researchers and laypeople integrated diverse forms of expertise and information to create a coherent boundary object of use to Arizona’s education communities. In this case, the boundary object was a model of the relationship between Arizona’s attainment percentage, economic outcomes, and workforce alignment (Fig. 3). To do this, we needed to understand how stakeholders in education communities conceptualized education’s public value, framed its implications for communities’ health, prioritized different actors (teachers, students, parents, policymakers) within the education system, discussed challenges related to data collection and transparency, and brought to light any other issues regarding Arizona education about which we were unaware. Thus, we engaged in an iterative process of education data analysis, model development, stakeholder engagement, integration of stakeholder feedback, and updating the user interface.
Throughout the course of our participatory agenda-setting process, two questions dominated stakeholder engagements and ultimately shaped the final version of the model (Figs. 6, 7, 8). The first was, “What is the impact of increased attainment on Arizona’s economy and workforce?” Stakeholders did not want to conceive of education for the sake of education. They wanted to understand the impact of education on economic well-being and the ability of people to find occupations which could provide income stability far into the future. The second question was, “Which 60%?” In the effort to get from 35 to 60 percent educational attainment, did it matter who comprised the 25 percent necessary to reach the state goal? Stakeholders wanted to be able to examine the relative impact of increased attainment for one sub-population or geographic region versus another and explore alternative economic and workforce futures as a result. Figure 3 shows the final version of the Achieve60AZ model, wherein users select sub-populations of interest (Fig. 6), examine each sub-population’s descriptive statistics compared with the state average, compare geographic locations of each sub-population, increase attainment for each sub-population and see the relative impact on the state goal (Fig. 7), compare economic outcomes (Fig. 8) and workforce alignment as a result of the attainment increase, and explore differences in attainment-occupation pathways for each. Though the immediate end-user of the model is ABOR, other educational stakeholders are welcome to engage with the model at DT in a facilitated environment before it is made available to the public online.
Roles of researchers and level of citizen participation
Whereas the Continuity of Care project took a deductive approach characterized by predetermined directives and deliverables approved by an external funding agency, the Achieve60AZ project was an inductive endeavor which began with a much more open-ended line of inquiry to understand phenomena of interest surrounding the Achieve60AZ Alliance goal of increasing postsecondary attainment. For the first 9 months of the project, the research team served a largely facilitative role, initiating conversations with education, industry, and governmental organizations to solicit diverse ideas, perspectives, and viewpoints regarding the role and potential impact of education on society, as well as continuously updating the data visualizations and model interface to reflect the ongoing maturation and focus of conversations. As the data analysis and visualization began to coalesce into a coherent, interactive model, the role of the researchers shifted to that of change agent. In this role, the research team sought to empower ABOR representatives and the Executive Director of Achieve60AZ, with whom they had worked closely for over a year, to own the model, learn to lead community engagements using it as a tool, to present and interpret its findings for lay audiences, and use it within their own organizations to develop information and policy interventions without the need of the research team. The research team identified such transition of ownership of the model as a goal of the project since its inception.
Though both Achieve60AZ and Continuity of Care were externally funded, the research team for Achieve60AZ set forth criteria for ABOR that in addition to financial resources they needed the time and intellectual involvement of ABOR representatives as co-creators of the model which was being developed. The collaborative infrastructure created and maintained by the research team and ABOR reflects more of a partnership level of citizen participation. At this rung of Arnstein’s ladder, power is “redistributed through negotiation between citizens and powerholders. They agree to share planning and decision-making responsibilities… after the ground rules have been established through some form of give-and-take, they are not subject to unilateral change” (1969). ABOR representatives and the Achieve60AZ Executive Director met with the research and model development team every 2 weeks for almost a year to assess progress, provide feedback regarding the possible use and relevance of the model, and make suggestions for community groups whose voices they felt should be included in the model development process. If ABOR and the Executive Director felt that the analysis and visualization had drifted away from questions and areas of interest, the research team course-corrected to make sure the model reflected stakeholder and community concern. As the research team began to develop layperson data analysis briefs, ABOR and the Executive Director had final editing power over what would be shared in any communication material related to data analysis resulting from the model.