Introduction

Culture and climate surveys are frequently used in United States schools to collect information about how students are experiencing school (Roach & Kratochwill, 2004; Schweig & Yuan, 2019). School climate is generally defined as the commonly held values and beliefs that influence the nature and consistency of interpersonal interactions among members within a school community (Bradshaw et al., 2014). Positive school climate has been shown to be associated with a range of positive outcomes including academics (Daily et al., 2019; Davis & Warner, 2018; Kwong & Davis, 2015), student well-being and mental health (Aldridge & McChesney, 2018; Ancheta et al., 2021; Newland et al., 2019; Nguyen et al., 2021), and student motivation (Fan & Williams, 2018; Verner-Filion et al., 2023). Racial-ethnic minoritized students, however, report worse school climate than their White counterparts (Voight, 2013). Black and Latino students express less positive perceptions of safety, sense of belonging, and interactions with adults (Voight et al., 2015). Thus, improving school climate for all students has the potential to promote positive and equitable student outcomes in a number of important ways (Lea et al., 2022). Yet, data derived from school climate surveys are often under-utilized to facilitate improvement, or only used in haphazard or non-systematic ways (Debnam et al., 2022). As such, this paper examines how one school district critically used data from a school climate survey to generate local evidence and make changes to their policies and practices in an effort to promote equitable change.

Data based decision making

Data-based decision making (also known as data-driven decision making; DDDM) encompasses efforts by educators to make decisions informed by local data. Data can be academic (e.g., student test scores) or non-academic (e.g., attendance, school climate surveys; Reeves et al., 2022). Educators can reflect on data to identify problems and frame questions, transform data into contextualized information, and use information to make decisions and evaluate outcomes (Mandinach & Gummer, 2016). Further, educators can respond to local data (e.g. school surveys) with the goal of improving student outcomes (Prenger & Schildkamp, 2018) and influence equity goals (Datnow & Park, 2018). A 2014 systematic review by Sun and colleagues (2016), illuminated five purposes for educator use of student data for decision making, including “assessing,” “planning and goal setting,” “improving instruction,” “identifying students’ behavioral or social problems,” and “communicating with parents” (Sun et al., 2016, p.16–17). Similarly, Datnow and Hubbard (2015) reviewed existing literature on teacher capacity for, and beliefs about, using data for decision making. The authors concluded that educator training largely focuses on how to access data, rather than addressing teacher capacity for using data, which can result in teachers feeling unprepared or lacking confidence in their ability to use data. In addition, research on educator use of data has not substantively emphasized equity, by failing to consider how power relations and positionality influence educators’ choices around the use of data (Datnow & Park, 2018). Dodman and colleagues (2021) suggest that to critically engage with data (i.e., shed light on the institutional mechanisms producing and maintaining inequities; p. 5), researchers, educators, and administrators must challenge the neoliberal framing of data use as an objective process (Dodman et al., 2021).

Use of research evidence

Data use is an interpretive, meaning-making process that is contingent on school and district organizational and political context, as well as educator knowledge, values, and beliefs (Coburn & Turner, 2011; Tseng, 2022). When local data is analyzed systematically, either by educators or external partners, research evidence is generated (Nutley et al., 2007; Supplee et al., 2023; Tseng, 2012). Like data-based decision-making, research evidence can be used (i.e., pulled into practice) in at least five distinct ways: (1) Conceptual: evidence impacts how users think about problems; (2) Instrumental: evidence is applied by users to a decision; (3) Process: research process engagement provides insights – distinct from acquiring insights from evidence; (4) Strategic: evidence is used to advocate for or otherwise justify a pre-existing stance; and (5) Imposed: evidence is used to constrain options through a mandate or a condition of funding (Nutley, 2007; Tseng, 2012; Weiss, 1977). Collectively, these functions form a typology of research evidence use.

Despite a meaningful scholarship on the use of research evidence (URE), some scholars have argued that this literature has not yet sufficiently centered practitioner perspectives (Gleeson et al., 2023). In a review of the URE literature, Tseng (2012) argued that scholarship on research use has not yet substantively explored the “pull factors” in how educational leaders (e.g., administrators in districts and schools) use research evidence. Echoing this critique, a decade later, Farrell and colleagues (2022, p.1) note that URE scholarship still has not substantially examined how school leaders actually use research. Ming and Goldenberg (2021) suggest that studies which seek to understand research use, especially as used to promote equity, must re-center practitioner and educator perspectives, needs, and values. Further, studies of URE should conscientiously consider institutional contexts, including, social hierarchies, educator and researchers power dynamics, and ubiquitous structural racism and systems of oppression (Doucet, 2019, 2020; Kirkland, 2019).

Critical use of research evidence

The practice of data-driven decision-making and the use of research evidence both benefit from integrating critical perspectives. Critical perspectives in education research often draw on Critical Race Theory (CRT), which is predicated on the assumption that racism is the foundation from which all U.S. systems and structures were built (Crenshaw et al., 1995). CRT seeks to promote equity by transforming the relationship between race, racism, and power in schools (Delgado & Stefancic, 2001) by emphasizing the lived experiences of people of color, bringing to light realities that have been obfuscated in White-dominant spaces (Crenshaw et al., 1995), and by situating experiential knowledge of people of color and their communities as valid and essential (Ladson-Billings & Tate, 1995). Building on these tenets, critical data-driven decision-making (cDDDM) is grounded in liberatory practices that seek to disrupt classroom and school-based systemic inequity by using data to reveal opportunity gaps in students’ school experiences, identify problematic structures and practices, and promote educators’ agency to address them (Dodman et al., 2021, 2023). The critical use of research evidence (cURE) aims to deploy research rooted in the perspectives and experiences of people of color and research that intentionally connects to and benefits communities of color (Michener, 2020). Scholars have argued that critical engagement with research evidence is required for high quality URE (Doucet, 2019; Tseng, 2022). As such, cURE seeks to democratize the production and use of research evidence by centering community and “insisting that the intended beneficiaries of interventions and policies be involved in the process of developing interventions and policies” (Doucet, 2019, p.15). The ways that researchers and educators make sense of data is bound by the individual’s social location–their prior knowledge, experiences, and beliefs (Dodman et al., 2021). Thus, “thoughtful engagement” with research evidence (Rickinson et al., 2020, p.25) must incorporate the perspectives of all engaged partners (Ming & Goldberg, 2021), center race (Doucet, 2019), and build “power in and for racially marginalized communities” through both the research process and the use of evidence (Michener, 2020).

Present study

The present study investigates one districts’ use of school climate data for equity-focused change. Our district practice partners define their equity work as having courageous conversations around race, collaboratively addressing disproportionality in their system, strengthening positive relationships, and deepening social and emotional competencies. Building on this definition, we focus on critical conceptualizations of equity that shift the focus from the products of inequity–i.e., student learning outcomes–to the processes of inequity–i.e., the systems and structures that reproduce inequities (Dodman et al., 2021, 2023). This study fills the gaps in the literature by illustrating the ways in which educators apply critical approaches in generating and using local evidence for equity. We join existing conversations around cURE to ask: how does one school district critically use evidence generated from climate data to promote equity?

Methods

Positionality

Reflecting the collaborative nature of this project, our research team consisted of university students, a postdoctoral scholar, and university faculty, in consultation and collaboration with educational practitioners. The lead author, a White ciswoman, led the project in collaboration with district and school staff. Her insider positionality as a former classroom teacher, and longstanding collaborative relationships with practitioners engaged in the project, led to the development of this partnership, and enhanced trust between university researchers and district personnel. The second author, a White transgender person, has experience conducting culture and climate surveys in higher education, using data to engage in strategic planning processes, and organizing initiatives to address student-identified areas of concern. The third author identifies as a Mexican cisgender male who has worked as an educator for more than 18 years. He currently works as a member of the central office district team that led the data collection initiative, and joins the writing of this manuscript, using a pen nameFootnote 1 to preserve district anonymity, on behalf of the practitioners who made this project such a success. The fourth author, a White ciswoman, worked as a school based mental health therapist, has analyzed climate surveys and helped school district leaders to make sense of this data. The fifth author, a White ciswoman, became a school social worker and prevention scientist as a result of observing her well meaning school unintentionally harm young people they were trying to support. Throughout data collection, analysis, and writing, we were cognizant of how our personal and professional experiences influence the research process (Braun & Clarke, 2022; Gough & Madill, 2012) by checking in with each other often, engaging with our practice partners in deep conversation replete with criticality about our roles in this work, and continuously returning to literature that offers a roadmap for conducting anti-racist research (e.g., Braun & Clarke, 2022; Gough & Madill, 2012; Tuck, 2009).

Setting

Here we describe two settings: (1) the district-led initiative with insights from the third author about the practice-facing side of the work, and (2) the research setting connected to the initiative.

District-Led Survey Initiative

The district team engaged in a year-long process to administer culture and climate surveys to students across the district, and, alongside the research team, sought to learn together how to use the survey-generated evidence to support change making efforts. The district conducted two brief climate surveys that were administered to every student in third through twelfth grade at two timepoints (February and May, 2023). Survey questions that were used districtwide were developed by district leadership in a department focused on culture and climate, in consultation with school-level educators and community members. The climate survey asked students to reflect on their perceptions of safety, wellbeing, and racism at school. The decisions around what survey items and constructs to use were largely made via a practice-side consensus building process gaining momentum over several months, where practice-side leaders convened frequently to draft items, review existing surveys, and make collective decisions for the district. The practice-side leadership then asked the research-side leadership for recommendations on additional or alternative survey items to consider that aligned with their practice-side efforts. These researcher recommendations drew on Likert surveys such as the well-established Patient Health Questionnaire-2 (e.g., “I felt down, sad, or hopeless”; Kroenke et al., 2003) and the emerging and highly action-oriented Berkeley Assessment of Social and Emotional Learning - Youth Voice (BASEL-YV; e.g., “there are people like me in our school materials''; Shapiro et al., 2022). Ultimately, understanding districtwide data use practices were prioritized over disrupting the consensus that was being reached around their own survey items – with the exception of incorporating six of the research-side recommended items into the district wide survey. In addition, two school sites (one middle school and one high school) served as district pilot sites for using the additional research-side recommended items. The surveys were administered via Kelvin (https://kelvin.education.com/), a district-licensed survey platform that immediately displays districtwide and schoolwide de-identified survey responses, as well as de-identified disaggregated survey responses by grade, race, language status, disability status, and gender to the district-authorized staff (e.g., district leaders could see data districtwide, classroom teachers could see data pertinent to their classroom) immediately upon survey completion. These categories were selected a priori by the district.

A district Culture and Climate team led a multi-step process to systematically analyze the data. The team of district leaders and training specialists, expanded to include one university researcher (the first author), met regularly guided by broad, descriptive research questions (e.g., “What is the state of wellbeing across the district?”, “What are the racial disparities in feelings of safety at the middle school level?”). The district team leadership was composed of mostly (self-identified) educators of color who have all worked as classroom teachers, some as school leaders, and all who currently hold roles at the district level. Consistent with cDDDM approaches, this team intentionally confronted inequity as the driving reason for disparities (Dodman et al., 2021). They did this by reflecting and challenging each other to interrogate root causes. For example, in conversations, members of the team would ask each other questions like, “Is it possible that anti-Blackness is a factor in these results?” and “What might be the role of racism in how these students perceive safety?” After several meetings to engage with each other and analyze the district-level data, the team then developed a three-step protocol and training materials for district and school-site level educators to engage with and analyze their own data in ways that prioritized equity. Drawing upon a framework called “Recognize, Interrupt, Repair” from Epoch Education (https://epocheducation.com), the team adapted steps to guide critical data use. In the district’s version, step one, “recognize,” prompted educators to observe their thoughts, feelings, and questions about the disaggregated data as related to belonging, well-being, safety, and racism. Educators were invited to pause and “check-in,” by reflecting on questions such as “What feelings are coming up for you as you look at the data?” and “How is your body reacting?” In small groups, they discussed with colleagues what they were noticing about their own reactions to the data and focused on feelings, rather than jumping straight to analysis and solution generation. In step two, “interrupt,” educators identified what information was missing and named subgroup trends– and corresponding inequities– as related to school culture and climate. This step specifically asked educators to look at differences across racial, linguistic, gender, and disability status differences in order to examine (in)equitable student experiences. During this phase, the district communicated the following to all the educators engaged in the training: “Conversations about data often lead to ‘one-size-fits-all’ solutions that obscure biases and ignore differences in environment, identity, and culture. Data reflection should inform decision-making that promotes equitable outcomes for all members of the school community.” Thus, the “interrupt” step provided concrete opportunities to make “the interrogation of inequitable conditions explicit” (Dodman et al., 2021, p. 6) by asking questions such as “what systemic issues might these data trends point to?” In step three, “repair,” educators brainstormed how data might inform classroom, school, and district-level next steps. In the present study, we investigated how these educators described their critical use of the research evidence they generated from survey data to promote equity.

The research context of the present study

This research investigation was enabled through a collaboration between one large (enrollment > 10,000 students) public school district and one large public university, and was approved by the University of California, Berkeley, Institutional Review Board. For the purpose of learning from these educators about how they used climate survey data, we conducted semi-structured interviews via Zoom with a sample of district and school personnel. School personnel that were interviewed came from schools where leaders, in addition to using the district-wide survey items, used university recommended survey items during the study period. District personnel interviewed were part of the leadership team tasked with leading the district-wide data collection efforts.

Procedure

Researchers used purposive sampling to recruit individuals who provided practice-side leadership for the work described above. The district-level leader who initiated the data collection provided researchers with the names of those in the district who served in leadership roles. Researchers then reached out to those staff to describe the study aims and invite them to participate in an interview. We conducted hour-long, semi-structured interviews via Zoom after the first of two survey administrations (May of 2023) in which we asked educators about their experiences engaging with evidence generated from the recent culture and climate survey. We asked participants about what they hoped to learn about their district from the data, how they collected and analyzed data, and how school and district staff have used data so far. Participants received $50 gift cards. Once interviews were completed, a team of undergraduate students de-identified and transcribed the audio recordings. Interviews were transcribed using Otter.ai and transcripts were subsequently cleaned and de-identified by the research team.

Participants

Four district-level (/d) and four school site-level (/s) leaders consented and participated in the research process. Participants described themselves as cisgender women (5) or cisgender men (3). Participants identified as: Asian / Asian American (1), Hispanic / Latin American / Latinx (1); White / Caucasian / European American (4), as well as Multiracial: Native American / Indigenous and White (1); Black / African American, Hispanic / Latin American / Latinx, and White (1). Participants were aged 30–39 (1), 40–49 (6), or 50 + (1), and had been in their roles for less than one year (3), 1–5 years (2), 6–10 years (2), or 10 + years (1). Participants were given pseudonyms for reporting purposes.

Data analysis

To answer our research question, we engaged in reflexive thematic analysis (Braun & Clarke, 2022). Our coding team consisted of three undergraduate students, two doctoral students, and a postdoctoral scholar. After engaging in familiarization, we developed our preliminary codebook based on the URE typology (e.g., “conceptual use,” “process use”) (Nutley et al., 2007; Penuel, 2020; Tseng, 2012). We also coded based on Doucet’s (2019) conceptualization of critical URE, which center issues of power and oppression. With this frame, we coded for the ways in which conceptual URE facilitated recognition of oppression, instrumental URE interrupted and disrupted mechanisms of oppression, strategic URE—also known as political URE, advocated for policy change that dismantles institutional racism, and process URE centered the experiences of racially marginalized community members (Doucet, 2019). After several rounds of coding transcripts to consensus (Hill et al., 2005), we moved from codes to themes. Within reflexive thematic analysis, this phase of the process includes developing “candidate themes” (i.e., draft themes; Braun & Clarke, 2022, p.98), as the first iteration of theme generation. Through iterative dialogue and reflexivity, we developed these candidate themes by elevating significant codes and actively identifying patterns in the coded data (Braun & Clarke, 2022). The final step of our analytic process was writing, which Braun and Clarke (2022, p.118) describe as “a key component of the analytic process.” Through the writing process, we further refined themes, named them, and selected illustrative quotes. Our writing process incorporated a critical lens that considered educator use of evidence as “markedly social, complex, and political” (Honig & Coburn, 2008, p.579).

Findings

Our analysis resulted in six themes (see Table 1) that align with use typology (Tseng et al., 2012; Nutley et al., 2007).

Table 1 Presentation of themes generated
Full size table

Using evidence to provide a common language

Educators conceptually used the data to establish a common language. Across district and staff interviewees, participants explained how the data provided an opportunity to focus discussions around survey dimensions. One participant explained how the common language from the dimensions gave staff “a starting point of how we can use that data-based problem solving process to start coming up with some hypotheses about why barriers exist for some of our students” (Mark/d). Similarly, school staff elaborated how the data provided the structure of discussions among various groups in the community:

The data is valuable, it gives us a place to start a discussion with lots of different groups and something common to look at and question and have noticings and wonderings about…Parents, community, staff, faculty, students, everybody has a common something they can discuss.” (Jill/s)

Several participants discussed how they used the evidence during “data talks” with staff. Participants explained how during this time, school staff had structured conversations about the evidence to examine inequitable disparities across subgroups and plan for action. One school staff member described a portion of the “data talk”: “So when I'm talking about it to our whole staff… [I asked] what are we trying to do? … Who are the subgroups here? Who are the most chronically absent? Who is struggling on our tests? Is it the same groups?” (Kara/s). This same participant then engaged in critical self-reflection, stating “we think we do a good job, but do we really?” Having the same baseline data, drawn from the same dimensions, and the same questions, then allowed the district to support school leaders in creating more profound understanding about current issues of campus culture and climate, as described by the same district staff member:

“We needed this baseline data in order to have [school] sites understand where they want to go … We can take that data, combine it with attendance, and behavior, and anecdotal information from teachers, surveys, and principal surveys. And with a compilation of it, then you can get a good feel of the culture and climate of the campuses. If you have some campuses are just knocking it out of the park, and others aren't, [you can ask] okay why is that? Can we then dig a little deeper and find out the why?” (Wendy/d).

Regardless of if participants worked at the school or district level, the interviewees spoke about using the evidence to create a shared language that framed their understanding of the topic.

Bringing attention to important trends to change the way staff look at problems

Participants also conceptually used the data to identify patterns and trends, with the goal of encouraging a collective shift in educators’ perspectives to frame insights and potential action. One district-level staff person spoke about using the data to bring attention to trends across the district:

“Looking across middle school sites, we're able to see the difference. We had one school site [where] only 64% of students felt safe. As opposed to another school where, like, 90% felt safe. I mean, that's a huge discrepancy” (Wendy/d).

Participants recognized the value of extracting meaningful insights from the evidence, as illustrated by this school-level interviewee who, in reference to a school staff meeting, stated “[we were] able to highlight and discuss the different things that jumped out at us and what we might do to address them” (Patrick/s). Similarly, participants also discussed how understanding trends served as a catalyst for staff to look at needs and potential problems from a student perspective, as illustrated by this interviewee who described their internal response to the tendency that some students did not feel like they could be themselves at school:

“[the evidence is] kind of like, ah man, it kinda it stings, right? You're like, gosh, how does that feel to feel like you can't be yourself? How do you learn if you can't be yourself? How do you feel safe if you can't be yourself?” (Kara/s)

This reflection underscores educators’ willingness to embrace the data-driven insights, working to accept the evidence as reliable representations of students’ reality.

Additionally, participants spoke about using the research evidence to bolster their equity lens by using the evidence to better understand differences in responses across groups (e.g., race, ethnicity, gender). One district-level staff member described a scenario where a site leader could look at these groups and shift their thinking:

“Maybe perhaps [they could say], ‘oh, wow, this is actually some really good data that I need to take action on. And what do I do with it? I have 45 black kids in my school, and they are scoring the lowest in the dimension of safety. And I have like 50 White kids at my school, and they are scoring the highest in safety. Those are comparable student groups and size[s]. What is different?’” (Gabriel/d)

Participants also challenged themselves, and others, to dig deeper around perceived problems or disparities in the data. For example, a district leader described how he encouraged school staff to interrogate potential root causes of what presented as inequities in student data: “You look at that statement, and … instead of just developing a bunch of action steps, let's consider why it's actually happening. … Is the instruction causing the problem? Is it the curriculum?” (Jeff/d). The work of attending to possible systemic roots was important for participants who were careful not to “blame” students for their survey responses. Indeed, school staff took up this critical reflection, and used the evidence to ask more challenging questions of themselves, as illustrated by this participant:

“[As a team we thought] how do we target and help these specific subgroups? Specifically, our Black, our English language learners, our Indigenous, our LGBTQ populations? … And how do we make them feel safe, make them feel welcome–increase their sense of belonging overall on the campus?” (Allison/s)

Many participants, particularly those of color, described having prior awareness of inequities within district schools. The survey validated what many already knew, and provided participants a path to bring that knowledge to other staff. One participant shared “for me, I'm not shocked at the data at all, at their responses. It was more about how to guide site leaders and the system as a whole on what to do. We [intentionally] opened Pandora's box” (Gabriel/d).

Making structural changes

District and school staff also instrumentally used the data to plan for and to make systemic shifts. For example, they updated goals, reconsidered the location of classrooms, questioned the demographics of classes (e.g., gifted education, special education), and completed library and curriculum audits. We provide some information about each of these uses below.

A core aspect of these structural changes was using survey data to inform state-mandated processes for setting improvement goals. Goal setting was a team activity that involved educators at various levels. Administrators developed “a general goal at the principals meeting, then brought that to the leadership team, and the leadership team kind of agreed with that, and then we brought it to the overall staff” (Patrick/s). Participants described these goals as structural changes that “will give us something to monitor our progress” (Jill/s). These goals influenced the long term success because they determined “where [staff] need to focus our money and our time and our effort and our resources” (Kara/s). In this way, goals provided a foundational framework for changes in school facilities, course offerings, and curricular materials.

One school decided to interrogate how temporary classroom facilities, separated from the main school building, were being used primarily for special education. Reflecting on disparities in the data, indicating that some students felt less connected to the school, site leaders reassigned classroom locations. One school staff member described restructuring the entire map of the school: “[Now] there's GATE (gifted and talented education) out there [in the temporary facility], there's elective classes, there's special education classrooms… like anybody you see walking out there, it's not just like, ‘oh, because I'm, you know, or they feel like, I'm stupid. I'm walking out.’ No. It could be any classroom” (Kara/s). This inspired participants to also reconsider the racial ethnic demographics of classes. As Kara said, “there are so many things to always consider when we think about who's in a classroom, right? A leadership class, what's the racial makeup of that? … What are we doing about that?”.

Structural changes also included curricular audits. At a district level, administrators used specific questions such as “there are people like me in my school materials” to enact change; they reflected that “we really need to make sure that we are looking at those possible new [curriculum] materials from a lens of diversity, equality, all of that kind of stuff” (Wendy/d). They elaborated on this point by stating “we cannot purchase curriculum that is not representative of our students… If we have extra money, we need to be purchasing books that we can put in the classrooms of our students, so that they are seeing people that look like them in the stories that they're reading” (Wendy/d). In this way, participants highlighted representation and inclusivity as key aspects of equity.

Planning for professional learning

Across participants’ roles, there was a recurrent theme of instrumentally using the evidence to plan for future professional learning opportunities for staff. One district interviewee explained how the data informed their planning: “So the intent is, from this data, what are the professional learning offerings that [we] can provide [teachers] as it relates to that [survey]?” (Gabriel/d). District-level staff discussed how the data identified a “plethora” of professional learning opportunities, including working “one on one in classrooms with teachers, we can do classroom observations, we can model lessons or circles” (Wendy/d). When planning professional learning, participants focused on the survey would advance the district’s work around equity, as described by this participant:

[We’ve been training teachers on] how to have a structured discussion [about race] with your class, even if you're a White teacher… How to really connect on a deep level with your kids so that they feel like you are a trusted adult… How do we then support the site leaders and teachers with resources and tools on how to do that?” (Gabriel/d)

Another discussed how exciting it was to use district-wide data for equity:

“I'm excited to see where the information from the survey shows up, where we can go and get into those classrooms, get into those sites, and have some real relationship building and conversations on pedagogy and instruction, like, how do we change the classroom?” (Cynthia/d)

When preparing for professional learning specifically for classroom teachers, another participant explained how they used the data to plan tailored instructional support:

“I've been called out to a couple of schools to go do some work [with training teachers]. And before I go, I look at their data because I want to know, ‘how are these students?’ This is my first data point before I get to school...that's how I've used [the data]. … I can have that in mind when I'm going into the classroom or to the school.” (Cynthia/d)

District educators also used the data for their role-specific planning. For example, those involved in the Multi-Tiered Systems of Support (MTSS) conducted an “all day training,” on data-based decision making. Others used it for social and emotional learning:

“We can make a direct professional learning offering. So let's say for instance, if there is a trend [related to survey item], ‘I have adults that I can go to for support at school’. If that's low, then maybe a direct professional learning is ‘how to build relationships with kids.’” (Gabriel/d)

In this way, participants used data to provide targeted professional learning opportunities.

Following up directly with students

Across interviews, participants expressed how data collection can– and should– be accompanied by application to real situations and decision making (i.e. instrumental use). In our analysis, we found that participants emphasized direct follow up with students as another form of critical date use. First, participants reported deliberate follow up related to student wellbeing. Wellbeing questions were used as “actual indicators, or flags, that could mean there's a mental health crisis or just a concern” (Wendy/d). When a student’s responses indicated mental health concerns, the administrators “[notified] those counselors at those school sites, so that they can at least check in and do their own evaluation” (Wendy/d). These questions allowed school staff to identify students whom they “might not otherwise have on [their] radar” (Jill/s). Indicating the importance of this instrumental use of the data, one participant reflected that the survey data “could have saved a life that day, because this could have been a kid, that you really had no indicator. And because they picked that question they at least knew that someone would check in and make sure they were okay” (Wendy/d).

Additionally, participants also discussed the importance of instrumentally using the data in ways that addressed inequities. Participants reported needing to “talk to specific groups of students who said certain things, to find out why they're feeling a certain way. So they can address those barriers intentionally, instead of ‘let's just try some things and hope it works’” (Mark/d). Specifically, administrators followed up with student focus groups, conducted additional interviews and listening sessions, and drafted open-ended survey questions for future surveys. One district-level interviewee stated, “if my Hispanic kids don't feel safe at school, I need to go out and do some empathy interviews” (Gabriel/d). Another echoed this sentiment:

“So if you're going to have [these] discussions with Black kids, and they tell you things to do, then you should listen to those kids and do something about it. And if you do those things, and you build relational trust, then they start saying, ‘Oh, if I do take the survey, something does matter. My voice does count’" (Mark/d).

This same participant provided another example stating,

“At one site you see there's [not many] American Indian students at this site. They've marked themselves very low on many of the items. As far as "I don't feel safe," "I don't have a connected adult," "They don't talk about race positively at my school" –and I'm paraphrasing obviously here. But that's a real opportunity for us to do something … it's pretty criminal in our industry to not do something about that. So the moral imperative is to have someone talk to those kid[s], in some way, right? And so that's how we can start addressing specific underserved students” (Mark/d)

Participants reflected on the importance of these conversations, stating “we can have some follow up because that's where you get so much more rich, actual data. It’s like, what did they say? Not just a yes or no, but like, what did they say, on how they're experiencing things?” (Kara/s). Ultimately, the combination of having direct follow-up with students about their experiences, acting on survey results, and collecting additional data was all about building relationships so that students were able to “trust the adults that something is happening as a result of [the data]” (Kara/s).

Engaging with the community

In addition to using the data, our participants also discussed their process use– and what they learned when facilitating this data collection. In our analysis, we identified that participants engaged with the community in generating evidence from culture and climate data. Participants spoke about seeking guidance on how to interpret and use survey data from students, families, and community members. This community engagement was crucial because “a lot of [decisions] have been done internally, which is unfair, because then we haven't had the voice of students or families… Even with good intentions, [educators] are missing a lens that we need to have” (Gabriel/d). That needed “lens” is one that centers equity. By bringing more stakeholders “to the table,” during community council meetings, participants reflected how this approach could bridge an often present divide between schools and the community. Therefore, staff asked “how do our families feel?” and “do they feel welcome here?” (Kara/s). Notably, staff communicated that families “were very interested” in the use of survey data (Jill/s). One staff member reflected that families were excited and felt like, “wow, the principal's really listening for real” (Kara/s).

One school-level interviewee explained how staff “showed [families and community members] not only the grid of responses, but also some sample questions,” stating, “this is what we were looking at, and then this is what we were thinking are things that we need to follow up on” (Kara/s). In this context, the survey data functioned as “a starting point” for members of the community who attended these council meetings. One participant reflected that: “[the data] has really opened up some dialogue and communication. We have something that we can all refer back to so instead of just people having, like incidental, like this happened in this class. We all have something to focus on to ground our conversation” (Jill/s). The relational aspects of engaging in the evidence-generation process helped build community trust in using survey data for changemaking.

In addition to exploring the process of generating evidence with community members and families during council meetings, participants also described– and engaged in– debrief processes with teachers, staff, and students after data collection. One school-level participant describes the staff meeting, where practitioners had the opportunity to debrief, reflect together, examine student response rates, and consider future survey administrations. A school-level administrator describes one such meeting:

“We are having these different trainings, and I am talking to the whole staff about [the data], right? It's not just going to be from my knowledge, and we have this like secret plan that we're doing. I'm gonna talk to the whole staff about - look at what our data is saying, look at what our kids are saying, this is life here on our campus. And then, [as a school site we discuss] what should we do from there?" (Kara/s)

Although participants reported that collecting, generating, and using the evidence was a time consuming process, they expressed that it was “well worth it when we think of our end goal here, that everyone feels safe, everyone feels included, everyone feels heard, as much as you possibly can” (Kara/s).

Discussion

In the present study, we investigated how district and school-level educators use data from student climate surveys to promote equity. Through our partnership, we learned alongside those generating and using evidence to enact change at the local level. That these educators could do so much from their survey administration is laudable. It was clear that staff we spoke to approached survey administration and action with intentionality, care, and a keen eye on equity. This was evident across our conversations, but most poignantly illustrated by a district leader who spoke about how important it was to use data: “This isn't just like, something that we're just going to do just to do. We're actually going to use this and follow up with it to help inform what we're going to do.” Indeed, the call to “actually… use this” was taken up by all to whom we spoke.

Types of data use

Each of the themes we generated were categorized within the evidence use typology (Nutley et al., 2007; Tseng, 2012) and contribute to existing literature on how educators generate and use evidence for local decision making. We also observed how uses aligned with critically engaged use of research evidence (Doucet, 2019) and critical data-driven decision making for equity (Dodman et al., 2021). Participants conceptually used the climate survey data to provide a “common language” to bring attention to racial inequities, expand their thinking about student concerns, and facilitate an increased awareness of structures of oppression within their institution. School and district staff instrumentally used the climate survey data to plan professional learning opportunities, make changes to policies and practices, and follow up directly with students. By engaging in the pursuit and actualization of structural change-making, participants were able to dedicate resources and codify equity-enhancing evolutions (Dodman et al., 2023). Furthermore, participants used the insights gained throughout this process to inform next steps. In this study, participants reflected on not only the data, but also the process of generating evidence, which could be used to promote equity by centering community member perspectives and experiences throughout data collection, analysis, and application of findings.

In addition to each theme aligning with the aforementioned evidence use typologies, we also found that across our analysis, critical uses were also apparent and cross-cutting. Participants were strategically using evidence to “advance an idea or set of actions meant to disrupt power or interrupt policies and practices that disproportionately harm marginalized groups” (Doucet, 2019, p.7). These educators described using data to raise awareness about known structural inequities, change school and district policies and practices that marginalized students, center the perspectives of racially marginalized community members, and interrogate who holds power in the school community. For example, when participants were conceptually using data to establish a common language or awareness, they were also strategically using the data to interrupt harmful ideologies deeply embedded in the institution of schooling. When educators instrumentally used the data to change the physical layout of their school, they were simultaneously–strategically– using the data to disrupt power imbalances and upend harmful narratives that certain groups (e.g. students who received special education services) should be relegated to a specific section of campus. When convening with community members to process the data, participants were also strategically centering the perspectives of those who have historically been excluded from school-based decision-making processes. Thus, the strategic use of evidence (i.e., advocate for or otherwise justify a pre-existing stance) was a cross-cutting, intersecting theme; our analysis suggests that all critical uses (e.g., all uses that center equity) may rely upon a strategic (also known as “political”) use of information. With this insight, we reflect on how cDDDM and cURE are unlikely to happen in the schools without underlying critical consciousness (Freire, 1973) and the deliberate actions of equity-oriented educators embedded and supported across all levels of the education system.

Practical implications: advancing equity through action

Given how common school climate surveys have become, the findings from this study can offer a roadmap to practitioners who may want to prioritize equity and action, but wonder where to start. First, participants stressed the importance of connecting conceptual understandings with instrumental action–which can be summarized as “embracing the essence of student voice…then doing something with the data” (Gabriel/d). However, the “something” may vary across roles, contexts, flexibility to actually enact change (Datnow, 2011), a climate for data use (Schildkamp et al., 2019), and of course, based on the evidence. In the present study, participants saw disparities across racial equity lines and then followed up with students directly, developed professional learning offerings, and planned structural changes. Strategically employing a critical, equity-oriented lens, while using the data to guide their decision making, allowed educators to act in more targeted and intentional ways. Sharing the data, coupled with tangible action, facilitated trust between students and staff. Staff reflected that they had built trust with students by demonstrating a commitment to action, which aligns with prior work with students who emphasized how, after competing climate surveys, they wanted to be heard and supported, and have adults take action on survey responses (Duane et al., 2009).

This study fills a gap in the literature by elevating school and district staff’s perspectives on how data can be used in race-conscious, equity-centered ways to identify oppressive systems and drive practical equity-centered systemic change in school district policies and practices, and (Doucet, 2019; Farrell et al., 2022; Gleeson et al., 2023). Furthermore, this study responds to critical scholars’ calls to challenge neoliberal conceptualizations of data use as objective, instead framing use of data as a subjective process that constitutes “a system of power” (Kirkland, 2019, p.2; see Coburn & Turner, 2011; Dodman et al., 2021). We focus on participants’ subjective experiences in an effort to illuminate the transformative ways that school and district staff can use data to interrogate deficit frameworks that blame students for systemic inequity (Kirkland, 2019). Reflecting on our findings as situated in this body of literature, we invite scholars and practitioners to consider their role in the critical use of evidence to advance the broader goal of dismantling institutional racism and fostering institutional change.

Furthermore, as this research endeavor sought to learn with and from educators about their critical use of data, we note the importance of making connections directly to practitioners in schools and districts working within current systems, who are using data for equitable transformations. For this reason, we have generated a practice-facing scheme of our findings (see Fig. 1), aligned to various educational routines, with the hope that this figure will communicate, integrate, and sustain practice advances. This may create in-roads for educators in districts and schools who want to better understand how to critically use data for more equitable change.

Fig. 1
figure 1

Critical Use of Evidence in Practice: Depiction of themes as a practice-facing schematic. Image copyright retained by Authors

Full size image

Research implications: embracing critical use

Some critical scholars argue that use of data risks devaluing the experiences and perspectives of racially marginalized community members (Doucet, 2019; Kirkland, 2019) and functions to obscure racism in institutions such as the public education system (Gillborn et al., 2023). Zuberi & Bonilla-Silva (2008) argue that the process of collecting, systematically interpreting, and using data as evidence has historically been aligned with White, Eurocentric epistemologies that work to secure and/or reinforce White privilege and White supremacy. The resulting impact for communities of color has been a history of harm that can reify stereotypes, uphold deficit narratives, and contribute to bias towards people of color (Monroe-White, 2021; Murray-Lichtman & Elkassem, 2021). Thus, our work attending to how data is critically used for equity is necessary but ultimately insufficient; we must also attend to issues of power and oppression in data collection– and in research generally– from start to finish. Future research should more intentionally grapple with questions such as “who participates in the research, who poses the questions, how [are] data gathered, and who conducts the analysis” (Tuck, 2009, p. 423).

Dodman and colleagues (2023) argue that without critical training and professional development, educators’ are underprepared to use data for equity. Illustrating this idea, Sun and colleagues' 2016 systematic review found that one of the most common educator data uses was “identifying students’ behavioral or social problems” (p.16). This type of data use that is individually-oriented, color-evasive, and focused on student deficits (i.e. problems) risks reproducing racial oppression (Dixson & Rosseau, 2005; Ladson-Billings & Tate, 1995). Contrastingly, when data users take up critical approaches to data-based decision making and to using research evidence, they can identify racial inequities, validate experiences of institutional racism, and facilitate action to dismantle structures of oppression. Therefore, we join existing conversations in the cURE and cDDDM fields that call for reflexive engagement that lifts up the systemic, critical use of data and evidence in the work of knowledge production and mobilization.

Limitations & future directions

The conclusions that we were able to draw in this study are limited by our sample. We interviewed a small number of school and district staff who were enthusiastic about using data to promote equity. We did so as a means of lifting up possibilities for critical uses of climate survey data. Other school and district staff members may not have used the survey data as extensively or with as keen a focus on equitable change. Our qualitative design and small sample size, although sufficient for answering our research questions, is not intended for broad generalizations. Instead, we believe these rich and descriptive findings adhere to the qualitative criteria for transferability (i.e. that findings may be applicable to other settings, but application is entirely dependent on the reader’s assessment of fit; Lincoln & Guba, 1985). Additionally, given our study aims to focus on district and schoolwide use of survey data, we elected to speak to district and school-level leaders. Future research is needed to explore how critically engaged use of data is connected to classroom-level practice and change.

Additionally, in this study we did not interview any students or families. Our research question specifically focused on understanding how educational leaders generated and used local evidence to influence processes at the school and district level. Future research that involves youth directly vis-a-vis Student Involved Data Use (Jimerson & Reames, 2015) can be conducted to explore: (1) how youth want educators and administrators to use data (e.g., Duane et al., 2024), and (2) how youth themselves use data in critical ways. Participatory processes that center the voices of students –the most impacted group– will be crucial for advancing both research and practice.

Lastly, understanding the uses of survey data within this particular district is an evolving collaboration. We expect participants' use of the data will continue to change over time. Along these lines, our findings are based on participants’ perceptions of how things have changed in their school and district. Longitudinal and observational research is needed to determine if the uses of data discussed in this study result in sustained practice and policy shifts, improved climate, and more equitable education.