Using GIS to Evaluate a Fire Safety Program in North Carolina
- First Online:
- Cite this article as:
- Dudley, T., Creppage, K., Shanahan, M. et al. J Community Health (2013) 38: 951. doi:10.1007/s10900-013-9705-x
- 280 Views
Evaluating program impact is a critical aspect of public health. Utilizing Geographic Information Systems (GIS) is a novel way to evaluate programs which try to reduce residential fire injuries and deaths. The purpose of this study is to demonstrate the application of GIS within the evaluation of a smoke alarm installation program in North Carolina. This approach incorporates national fire incident data which, when linked with program data, provides a clear depiction of the 10 years impact of the Get Alarmed, NC! program and estimates the number of potential lives saved. We overlapped Get Alarmed, NC! program installation data with national information on fires using GIS to identify homes that experienced a fire after an alarm was installed and calculated potential lives saved based on program documentation and average housing occupancy. We found that using GIS was an efficient and quick way to match addresses from two distinct sources. From this approach we estimated that between 221 and 384 residents were potentially saved due to alarms installed in their homes by Get Alarmed, NC!. Compared with other program evaluations that require intensive and costly participant telephone surveys and/or in-person interviews, the GIS approach is inexpensive, quick, and can easily analyze large disparate datasets. In addition, it can be used to help target the areas most at risk from the onset. These benefits suggest that by incorporating previously unutilized data, the GIS approach has the potential for broader applications within public health program evaluation.
KeywordsGISProgram evaluationFire safetyMethodologyPotential lives savedGeocodingInjury surveillance
Fires are a leading cause of residential injury deaths in the United States . A recent summary from the National Fire Protection Agency states that there were 2,590 deaths resulting from residential fires with nearly two-thirds of these deaths occurring in homes that had either non-functioning or missing smoke alarms . The presence of functioning smoke alarms in homes can help save lives by warning residents and by prompting early detection of fires . In communities that are defined as high-risk, smoke alarm installation programs play an important role in increasing the reception to and use of alarms, ultimately reducing morbidity and mortality rates as the result of residential fires [1, 4].
Between 2001 and 2011, the Get Alarmed, NC! program housed in the North Carolina Injury and Violence Prevention Branch placed 17,085 smoke alarms in 9,085 North Carolina homes. Specific populations were targeted as high-risk for death or injury from residential fires, including residents over 65 years of age, residents under age 5, disabled residents, Hispanic residents, Native Americans, residents living in older, rental, or mobile homes, and residents in low-income neighborhoods . The program has worked with twenty-one local fire departments by providing both alarms and structured training for alarm installation. Compared with alarm giveaways, this installation approach is recommended for ensuring that a higher percentage of alarms are properly installed and functioning after several years [1, 4, 6, 7].
Evaluating Fire Safety Programs
Few smoke alarm and fire safety program evaluations exist to date and they vary by state and methodology. A large multi-state cross-sectional study took place in North Carolina, Minnesota, and Oklahoma in the early 1990s where health departments and fire departments sought out homes with at least one functioning smoke alarm a few years after alarm distribution and installation programs ended . The evaluation required home visitation by persons from each study team, a costly and intensive method. Initial alarm distribution covered 10,000 homes across three states, but only a sampling of program homes were selected to participate in the evaluation. The evaluations of alarm function and coverage by unannounced follow-up visits were completed at 436 of 631 attempted addresses, with 64 % of these homes retaining at least one functioning alarm. Shults points out that installation procedures and target groups varied by state, and therefore evaluation results were not readily comparable .
Prior to 1998, removable smoke alarm batteries may account, in part, for the percentage estimates of alarms that are low-functioning. In 1998, however, the centers for disease control and prevention (CDC) established the smoke alarm installation and fire safety education (SAIFE) program that entailed installing long-term smoke alarms with 10-year lithium batteries. An evaluation of SAIFE alarms was performed across samples of homes in Kentucky, Georgia, Virginia, Washington, and Oklahoma 10 years after the alarms were installed to assess the presence and function of the original alarms. The evaluation required in-home follow-up visits to test for performance, replace batteries, and assess the number of missing alarms. The evaluation results showed that only 33 % of the original alarms still functioned after an 8–10 years period. After further examination, there was evidence that many alarms had been tampered with and the lithium batteries removed or replaced. Where there were alarms with lithium batteries installed, the majority were still functional at the time of evaluation .
The Oklahoma City Smoke Alarm Project involved a large-scale alarm and educational pamphlet giveaway with the ultimate goal of reducing the burden of fire and burn-related morbidity and mortality in the target community [8–10], similar to the desired outcome of Get Alarmed, NC!. The objectives of the evaluation were to measure who received alarms, how the alarms functioned, and the impact the alarms had on the health of the target population. Outcome evaluation measured fatal and non-fatal fire and burn injury rates for both the program population and the larger general population as separate entities. Rates were calculated per 100,000 residents as well as per 100 residential fires for each group . In the 4 years post-intervention, both the target area and the remainder of Oklahoma City saw declining rates, though injury rates in the targeted area were reduced by as much as 80 %. The program had successfully installed additional smoke alarms in a number of program houses with 45 % functional .
Geographic Information Systems (GIS) and Public Health
Geographic Information Systems (GIS) is used in the field of public health for research, program, and policy purposes. Although GIS primarily has been a tool for disease surveillance, it also has significant applications for program planning, risk analysis, and community mapping and assessment [11, 12]. As GIS software continues to evolve and simplify spatial analysis it will continue to be incorporated into public health methods in rather innovative ways, suggesting it still has multiple applications beyond its traditional uses in public health.
Following a California sales ban on alcohol and gasoline, Farmer et al.  conducted a study evaluating alcohol-related driving and car crash-related morbidity and mortality using a regression model incorporating variables on the spatial context of crashes. These variables were derived in GIS and were used to control for spatial heterogeneity by classifying similar municipalities based on commonalities in road infrastructure, population density, and proximity to other population centers. The authors found that these GIS-derived spatial variables were influential predictors (that were in most cases statistically significant) of every crash type. GIS has also been applied to public health practice in past emergency response efforts. In order to document and analyze the needs of affected communities across Texas following Tropical Storm Allison in mid-2001, GIS was critical to quickly and efficiently obtain and produce the information and corresponding maps needed for directing aid to appropriate outlets .
Program Evaluation: The Next Step for GIS
Get Alarmed, NC! and most other fire prevention programs have used telephone follow-up surveys or in-person home visits [1, 7, 15] to evaluate outcomes of smoke alarm programs. Unfortunately, there are limitations to telephone survey, mail survey, and household survey/home visit methods that have demonstrated surprisingly low validity in some cases [16, 17]. In addition to cost, time, potential interviewer bias, potential response bias, and inability to follow-up on large samples, these measures may result in overreporting of the desired outcome by program or study participants [16–18]. Stepnitz et al.  attempted to tease out the causes for overreporting ownership of a functioning smoke alarm by residents and found that residents’ perceptions, education, and overall understanding of smoke alarms played an important role in overreporting.
Interestingly, GIS has been underutilized as an evaluation tool in public health for a number of reasons, but most likely because GIS has not been regarded as “dynamic” or an instrument that can assist in collecting, analyzing, and displaying ongoing process, impact, and outcome evaluation data . However, GIS has been applied as an evaluation tool elsewhere. In an attempt to evaluate the placement of a class of medical extern students in rural practices across Michigan, researchers used GIS to define and produce three different definitions of “rural setting” displayed on thematic maps with the students’ practice addresses plotted on each one. The use of GIS highlighted the program’s misplacement of the extern students since many of them were being placed in less rural areas . The authors acknowledged their creative use of GIS for evaluation of a program even though there is little precedent for it [12, 19]. More recently, in the Songkhla province of Thailand, researchers sought to evaluate the effectiveness of insecticide space spraying at preventing the spread of the virus that causes dengue fever. The study combined local hospital surveillance data and GIS spatial data as a means for tracking and measuring the secondary transmission of the virus, which ultimately proved the campaign to be ineffective .
Ta et al.  emphasize a need for further quality evaluation of smoke alarm and fire safety evaluation programs. The objectives of this paper are to evaluate the Get Alarmed, NC! program, to demonstrate an innovative application for GIS technology within the field of fire safety and smoke alarm program evaluation, and to illustrate the flexibility and accessibility of GIS for other applications in public health program evaluation.
The Current Study
The goals of the Get Alarmed, NC! program were to increase the number of homes with functional alarms and to expand the number of homes with adequate alarm coverage in an attempt to reduce the number of deaths and injuries caused by residential fires. Successful program evaluation was an additional objective and is necessary for understanding the extent of the program’s impact. The initial program evaluation approach was to contact 20 % of participants by telephone surveys and estimate the number of potential lives saved based on this limited sample. This paper introduces an alternative and novel way to use GIS as a tool to evaluate the smoke alarm program and by doing so illustrates its potential broader application to the field of public health program evaluation.
In order to determine the number of program homes that were later affected by fires and calculate the potential lives saved by the program it was necessary to merge two datasets: one from the Get Alarmed, NC! program and another from the national fire incident reporting system (NFIRS).
Get Alarmed, NC!
Get Alarmed, NC! program data are stored in a spreadsheet with an observation for each of the program’s 9,085 installation addresses which correspond to 17,085 installed alarms (many homes had two or more alarms installed as dictated by best-practice). Each observation includes the street address where the alarm was installed, the installation date, the number of alarms installed, home and occupant information, and fire safety questionnaire responses.
National Fire Incident Reporting System
The NFIRS database is the collection of fire incident information reported to the U.S. Fire Administration by participating fire departments. Reporting to NFIRS is voluntary and the average North Carolina coverage over this time period is estimated at 75 % . NFIRS is a national dataset and the final dataset obtained for study was filtered to contain only North Carolina incidents affecting residential structures. Between 2001 and 2010, NFIRS recorded a total of 50,709 residential fire incidents in North Carolina.
The Get Alarmed, NC! and NFIRS datasets were directly imported into ArcGIS 10  for geocoding, mapping, and analysis.
Calculating the Number of Program Homes Affected by Fires
Calculating Potential Lives Saved
Unfortunately, NFIRS data do not record the number of persons in the residence at the time of the fire incident. Program survey data collected during the alarm installation visit did not ask for the number of home occupants or residents. Only limited data were available on youth and older adults living in the home. Therefore, deriving the number of ‘lives potentially saved’ from the install addresses affected by reported fires requires estimating the number of occupants in those residences. We use two distinct methods to estimate residence occupancy starting from either our limited program survey data or US Census data. The two household-level occupancy estimates create separate lives potentially saved estimates.
The first and lower estimate assumes one single resident per home per fire incident, unless a child lived at the home at the time of installation as noted by program documentation. If a child under age five lived in the home, then two people, the child and the implied guardian, were considered to have been in the residence during the reported fire incident.
Number of Program Homes Affected by Fires
Ninety percent of the addresses in NFIRS and 89 % of the addresses in Get Alarmed, NC! were able to be geocoded correctly. There were 232 address matches found between the two datasets, representing locations where smoke alarms were placed by Get Alarmed, NC! and where fire incidents occurred as listed in NFIRS. There were 167 addresses where fire incidents occurred after program smoke alarms were installed.
Number of Potential Lives Saved
Household fires and potential lives saved methods comparison
Participant call back survey (20 % sample)
Traditional straight address merge
Merge using GIS geocoding
Number of household fires
Potential lives saved estimate (program documentation)
Potential lives saved estimate (expected household residency-census)
The current study demonstrates the utility of applying GIS to public health program evaluation and builds upon current evaluation methods. We were able to use GIS to identify 167 household fires that occurred in homes where Get Alarmed, NC! had placed fire alarms. Additionally, GIS allowed us to determine that between 221 and 384 lives were potentially saved as a result of the program. The percentage of alarms placed that later were in fire incident homes varied between fire departments. This indicates that several departments more accurately targeted homes and occupants at high fire risk. Learning from these departments could help future programs target homes more effectively. Additionally, this also suggests that GIS could be an effective process evaluation tool; the project team could have provided feedback to fire departments that were not accurately targeting homes for alarm placement if this were known during program implementation.
GIS alleviates many of the challenges presented by relying solely on collected project data to merge datasets. The strengths of this method include the ability to link large disparate data sources to quickly and effectively assess impact. A simple database match-merge would typically accomplish this by using a variable with unique values as a merge field (e.g. street address). Using this method alone, only 9 % of actual street address matches would be linked correctly because of data inconsistencies in the address fields from each data source. Ideal merge fields would provide unique identifiers with no misspellings and consistent formatting. However, the matching fields in this case are the install and fire incident addresses which are not uniform. Program information from surveys often contains misspellings and inconsistent formatting of street prefixes and suffixes. GIS is a more sophisticated technique which handles these inconsistencies and subsequently allows for more accurate estimates of the number and location of homes and lives potentially saved by the program. This current study was able to match 167 addresses, which would not have been possible using conventional database merging. For this dataset straight match-merging linked only 15 addresses correctly. Compared to the GIS geocoding approach, a larger number of addresses would only be possible from statistical programming methods incorporating probabilistic linkage which involves sophisticated programming skill and experience. Advanced statistical software packages (i.e. SAS, STATA) have this functionality, but the GIS geocoding approach is quicker, easier to learn, more intuitive, and less expensive to implement.
There are several limitations to utilizing this approach. First and foremost, reporting of fires by fire departments into NFIRS is not universal nor is it verified. For example, it is estimated that the NFIRS data for North Carolina are only 75 % complete. The impact of this would be that residential home fires are underreported, thus any additional homes with alarms would be underrepresented. Moreover, if more complete, we would likely see additional fires and additional lives saved. Second, several methodological shortcomings reduce the estimates of the number of lives saved by the program. For instance, the majority of fire events do not involve fire department intervention. Therefore, cases where functioning alarms alert residents and preclude the need for contacting fire departments, are not included in estimates of potential lives saved. Also, the majority of the program’s alarms will still function beyond the time of evaluation, and functional alarms will continue to produce benefits, such as reducing rates of fire damage, as well as injuries and death. Therefore the results presented in this article potentially underestimate the number of program homes affected by fires and the number of potential lives saved. That said, there are also fires where no person or persons are at risk. Either they are not home at the time of the fire or the house is vacant. We have no way of knowing the degree to which this could be happening in these fires but assume a small percentage at most. Using the GIS geocoding method also can exclude homes or fires that did not match. Again, in general we think this is a very small number (in our case less than 1 percent).
To our knowledge, no fatalities occurred within Get Alarmed, NC! homes during this study period.
Using the GIS method, we believe that Get Alarmed, NC! had a greater impact than originally thought and using this method in the future could result in better and more effective programs. GIS is a relatively accessible and easy to use tool which standardizes address information and is perfectly suited for this type of program evaluation. This GIS evaluation approach has several advantages compared with the traditional telephone survey method in that it does not require contacting program participants, which can be time consuming and costly. The scale of the analysis can easily be expanded to larger datasets or geographic regions, and total potential lives saved estimates can be calculated without extrapolating from a smaller sample. Also, this approach generated data that could facilitate program strategic planning and better program implementation. For example, if GIS had been used to identify the areas most at risk from the start based on NFIRS data, the targeting would have been more focused on homes in areas most at risk. Likewise, incorporating GIS into the program design could have influenced the home selection process. This approach has demonstrated a novel way in which GIS can be utilized as a program evaluation tool and its application can be well beyond the fire safety programs.
The authors would like to thank the other members of the NC DPH IVPB including Ingrid Bou-Saada, Stephania Sidberry, and Alan Dellapenna for their guidance, support, and feedback during the writing of this article. In addition, we would like to specifically thank Sherri Troop for her insight, knowledge of the Get Alarmed, NC! Program, and passion for fire prevention and fire safety education. We would also like to thank the “Get Alarmed, North Carolina!” advisory board, specifically, Ernest Grant of the UNC Jaycee Burn Center, Kelly Ransdell and Allan Buchanan of the North Carolina State Fire Marshal’s Office, Mike Bowling of the Injury Prevention Research Center at UNC, and Jeanne Givens formerly of DPH IVPB. Their guidance and assistance made it possible for the project to successfully reach many of the local fire departments throughout the state. We could not have met and reached those most at risk of fire death and injury without the tireless efforts of those firefighters who work to save and educate local residents on fire prevention. Thank you to all those firefighters too numerous to name here!