This paper examines the reliability of the methods used to capture homicide events committed by far-right extremists in a number of open source terrorism data sources. Although the number of research studies that use open source data to examine terrorism has grown dramatically in the last 10 years, there has yet to be a study that examines issues related to selectivity bias. After reviewing limitations of existing terrorism studies and the major sources of data on terrorism and violent extremist criminal activity, we compare the estimates of these homicide events from 10 sources used to create the United States Extremist Crime Database (ECDB). We document incidents that sources either incorrectly exclude or include based upon their inclusion criteria. We use a “catchment-re-catchment” analysis and find that the inclusion of additional sources result in decreasing numbers of target events not identified in previous sources and a steadily increasing number of events that were identified in any of the previous data sources. This finding indicates that collectively the sources are approaching capturing the universe of eligible events. Next, we assess the effects of procedural differences on these estimates. We find considerable variation in the number of events captured by sources. Sources include some events that are contrary to their inclusion criteria and exclude others that meet their criteria. Importantly, though, the attributes of victim, suspect, and incident characteristics are generally similar across data source. This finding supports the notion that scholars using open-source data are using data that is representative of the larger universe they are interested in. The implications for terrorism and open source research are discussed.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Tax calculation will be finalised during checkout.
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
Tax calculation will be finalised during checkout.
The ECDB has received funding to expand its focus to include far-left and Al Qaeda inspired violent criminal activities and financial crimes by far-rightists and Al Qaeda inspired offenders. Since the focus of the analysis is on the far-right homicides, these data will not be discussed.
This study operationalizes the far-right as individuals or groups that subscribe to aspects of the following ideals: They are fiercely nationalistic (as opposed to universal and international in orientation), anti-global, suspicious of centralized federal authority, reverent of individual liberty (especially their right to own guns, be free of taxes), believe in conspiracy theories that involve a grave threat to national sovereignty and/or personal liberty and a belief that one’s personal and/or national “way of life” is under attack and is either already lost or that the threat is imminent (sometimes such beliefs are amorphous and vague, but for some the threat is from a specific ethnic, racial, or religious group), and a belief in the need to be prepared for an attack either by participating in or supporting the need for paramilitary preparations and training or survivalism. Importantly, the mainstream conservative movement and the mainstream Christian right are not included.
We attempted to review an 11th source, the Bureau of Justice Assistance (BJA) and Department of Defense (DOD) funded Institute for the Study of Violent Groups (ISVG) that tracks crimes committed by political extremists since 2002. Our requests for a listing of far-right homicides committed in the U.S. went unanswered.
We excluded the National Counterterrorism Center’s Worldwide Incidents Tracking System (WITS) database that includes terrorist acts committed since 2005 in our analysis because it only contained one far-right related homicide event. This homicide was included in the other sources we examined.
Six of the sources we used only included acts committed to further a “political” or “ideological” objective. These sources assume that ideologically motivated crimes are committed for a higher purpose and are different than routine crimes committed to further personal interests. The ADL, SPLC, the media and our coders focused however, on crimes committed by extremist suspects as opposed to events. Thus, these sources are interested in all homicides committed by far-rightists (both ideological and non-ideological). Our study assumes that the ECDB categorization of a homicide event as either ideologically motivated or non-ideologically motivated is accurate. In this sense, we are privileging the ECDB. For example, if the ECDB categorizes homicide X as ideologically motivated, while the FBI or GTD categorize it as non-ideological we code that event as ideologically motivated. While we appreciate critiques that question why the ECDB is treated as the ground truth, we make this determination for a few reasons. Again, the ECDB’s ongoing data collection efforts focus almost exclusively on homicides committed by far-rightists and other extremists in the United States. Further, the ECDB systematically searched through a series of open sources to identify these events twice. Conversely, most other data collection efforts focus on a much larger geographic universe and have a much larger N.
The FBI and databases like the GTD argue that while hate/bias-motivated crimes are related to terrorism, they are a separate phenomenon. Hate crimes are counted separately (e.g., the FBI’s UCR hate crimes report is distinct from the government’s annual terrorism reports). Non-hate crime ideologically motivated acts are thought to implicate broader political objectives that qualify as terrorist, while hate crimes do not qualify. Other sources disagree and sometimes conclude that ideologically motivated acts- anti- government or anti-minority- qualify and should be labeled terrorist. Here too we assume that the ECDB categorization of a homicide event as either a bias-motivated crime or not is accurate.
The FBI and sources that rely upon its definition (e.g., the ATS and Hewitt) conclude that acts committed by lone wolves usually do not qualify as terrorists. Groups like the IRA or Al Qaeda, 1998- 2001, are organized entities that engage in ongoing criminal activities designed to harm American interests. Conversely, lone actors usually lack the logistical support and infrastructure to conduct repeated attacks and to remain a longstanding threat to government interests. Thus, lone wolves do not implicate the same threat level and do not qualify as terrorist. Other sources conclude that ideologically motivated acts, regardless of the organizational level of the suspects who commit it, should be labeled terrorist.
The F.B.I. is charged with investigating domestic terrorism incidents and these incidents are subsequently prosecuted on the federal-level. Some sources again conclude that the jurisdiction of the prosecution is irrelevant and that what matters is whether the act is ideologically motivated.
The FBI defines terrorism as “the unlawful use of force or violence against persons or property to intimidate or coerce a government, the civilian population, or any segment thereof, in furtherance of political or social objectives” (FBI, 1997).
While Hewitt claimed to rely on the F.B.I.s definition, his precise methodology and validation scheme are unclear. He devotes only four pages, in an appendix, to his data sources and coding procedures. Further, as our analyses demonstrate Hewitt’s chronology includes lone actor, state-level and other attacks that should have been excluded under a strict application of the FBI’s guidelines.
There was the possibility that a homicide that occurred outside the time frame would be included in a source. SLATT reports legal decisions as these homicides progress through the criminal justice system. For example, it reported on the numerous appeals and court decisions related to the Oklahoma City bombing. We treated these cases as being included in the source even though they fell outside the time frame. In addition, Hewitt’s data collection ended 6/30/2004. Thus, we compare this source to the only the 1990–2004 homicides. Finally, the GTD does not include any incidents from 1993 thus we only compared this source for 1990–1992 and 1994–2008.
It is likely that a specific source relied on several of the other sources examined here to identify incidents. That is, these sources rely on other open sources, and thus the number of sources might be indicative of a single source that captured an incident, that was then used by the other sources.
We have excluded the FBI and ATS sources from these analyses because they focus only on federal cases.
All of the suspects for these incidents were male and white.
Aho JA (1990) The politics of righteousness: Idaho Christian patriotism. University of Washington Press, Seattle
Bakker E (2006) Jihadi terrorists in Europe, their characteristics and the circumstances in which they joined the Jihad: an exploratory study. Clingendael Institute, The Hague
Biderman A, Lynch JP (1991) Understanding crime incidence statistics: why the UCR diverges from the NCS. Springer, New York
Chen H, Reid E, Sinai J, Silke A, Ganor B (2008) Terrorism Informatics: knowledge management and data mining for homeland security. Springer, New York
Chermak SM (2002) Searching for a demon. The media construction of the militia movement. Northeastern University Press, Boston
Chermak SM, Gruenewald J (2006) Domestic terrorism and the media. Justice Q 23(4):428–461
Chermak SM, Freilich JD, Simone J Jr (2010) Surveying American state police agencies about lone wolves, far-right criminality, and far-right and Islamic jihadist criminal collaboration. Stud Confl Terrorism 33:1019–1041
Coates J (1995) Armed and dangerous: the rise of the survivalist right. Hill and Wang, New York
Dobratz BA, Shanks-Meile SL (1997) White power, white pride! The white separatist movement in the United States. Twayne Publishers, New York
Damphousse K, Shields C (2007) The morning after, assessing the effect of major terrorism events of prosecution strategies and outcomes. J Contemp Crim Just 23(2):154–174
Dugan L, LaFree G, Piquero AR (2005) Testing a rational choice model of airline hijacking. Criminology 43(4):1031–1065
Dyer J (1997) Harvest of rage: why Oklahoma City is only the beginning. Westview, Boulder
Endler W, Sandler T (2006) The political economy of terrorism. Cambridge University Press, NY
Freilich JD (2003) American militias: state-level variations in militia activities. LFB Scholarly Publishing, New York
Freilich JD, Chermak SM (2009a) United States extremist crime database (ECDB), 1990–2008: preliminary results. Department of Homeland Security University Network Research and Education Summit. Washington, DC
Freilich JD, Chermak SM (2009b) Preventing deadly encounters between law enforcement and American far-rightists. Crime Prev Stud 25:141–172
Freilich JD, Chermak SM (2011) Introducing the United States extremist crime database (ECDB), 1990-2010. Paper under review
Freilich JD, Pridemore WA (2006) Mismeasuring militias: limitations of advocacy group data and of state-level studies of paramilitary groups. Justice Q 23(1):147–162
Freilich JD, Almanzar NAP, Rivera CJ (1999) How social movement organizations explicitly and implicitly promote deviant behavior: the case of the militia movement. Justice Q 16(3):655–683
Freilich JD, Chermak SM, Caspi D (2009a) Critical events in the life trajectories of domestic extremist white supremacist groups: a case study analysis of four violent organizations. Criminol Public Policy 8(3):497–530
Freilich JD, Chermak SM, Simone J Jr (2009b) Surveying American state police agencies about terrorism threats, terrorism sources, and terrorism definitions. Terrorism Polit Violence 21(3):450–475
Green DP, Strolovitch DZ, Wong JS, Bailey R (2001) Measuring gay populations and antigay hate crime. Soc Sci Q 82(2):281–297
Gruenewald J, Freilich JD, Chermak SM (2009) An overview of the domestic far-right and its criminal activities. In: Perry B, Blazak R (eds) Hate crimes: hate crime offenders. Praeger, Westport, pp 1–21
Hamm M (1993) American skinheads: the criminology and control of hate crime. Praeger, Westport
Hamm M (1997) Apocalypse in Oklahoma City: Waco and Ruby Ridge revenged. Northeastern University Press, Boston
Hamm M (2002) In bad company: America’s terrorist underground. Northeastern University Press, Boston
Hamm MS (2005) Crimes committed by terrorist groups: theory, research and prevention. Final Report to the National Institute of Justice, September 2005
Handler J (1990) Socioeconomic profile of an American terrorist. Terrorism 13:195–213
Hewitt C (2005) Political violence and terrorism in modern America: Christopher Hewitt: a chronology. Praeger Security International, New York City
Hewitt C (2003) Understanding terrorism in America: from the Klan to Al Qaeda. Routledge, New York
Horgan J (2008) Interviewing terrorists: a case for primary research. In: Chen H et al (eds) Terrorism informatics: knowledge management and data mining for homeland security. Springer, New York, pp 27–50
Johnson S, Braithewaite A (2009) Spatio-temporal modeling of insurgency in Iraq. Crime Prev Stud 25:9–31
LaFree G, Dugan L (2004) How does studying terrorism compare to studying crime? Sociology of crime. Law Deviance 5:53–74
LaFree G, Dugan L (2007) Introducing the global terrorism database. Terrorism Polit Violence 19:181–204
LaFree G, Dugan L (2009) Research on terrorism and countering terrorism. Crime Justice 38:413–477
LaFree G, Morris N, Dugan L, Fahey S (2006) Identifying global terrorist hot spots. In: Victoroff J (ed) Tangled roots: social and psychological factors in the genesis of terrorism. IOS Press, Amsterdam, pp 98–114
Langer E (2003) A hundred little Hitlers: the death of a black man, the trial of a white racist, and the rise of the neo-Nazi movement in America. Picador, New York
Leiken RS, Brooke S (2006) The quantitative analysis of terrorism and immigration: an initial exploration. Terrorism Polit Violence 18:503–521
Lum C, Kennedy LW, Sherley AJ (2006) The effectiveness of counter-terrorism strategies: a campbell systematic review. [Online]: http://db.c2admin.org/doc-pdf/Lum_Terrorism_Review.pdf
Lynch JP, Addington LA (eds) (2007) Understanding crime statistics: revisting the divergence of the NCVS and the UCR. Cambridge University Press, Cambridge
Maxwell C, Chermak S (2007) Creating a data archive to facilitate research on understanding and responding to terrorism. In Gunes OS, Al-Badayner DM (eds) Understanding terrorism: analysis of sociological and psychological aspects. NATO Science for Peace and Security. IOS Press, Amsterdam, pp 306–331
Merari A (1991) Academic research and government policy on terrorism. Terrorism Polit Violence 3(1):88–102
Naday A, Freilich JD, Mellow J (2008) The elusive data on supermax confinement. Prison J 88(1):69–93
Neiwert DavidA (1999) In god’s country: the patriot movement in the Pacific Northwest. Washington State University Press, Pullman
Ross JI (1992) Research note: contemporary radical right-wing violence in Canada: a quantitative analysis. Terrorism Polit Violence 4(3):72–101
Ross JI (1993) Research on contemporary oppositional political terrorism in the United States: merits, drawbacks and suggestions for improvement. In: Tunnel KD (ed) Political crime in contemporary America: a critical approach. Garland Publishing, New York, pp 101–116
Sageman M (2004) Understanding terror networks. University of Pennsylvania Press, Philadelphia
Schmid A (2004) Terrorism on trial: the definitional problem. Case West Reserve Univ Sch Law 36:375–419
Silber MA, Bhatt A (2006) Radicalization in the West: the Homegrown threat. New York City Police Department Intelligence Division. Available from the WWW at http://www.nypdshield.org/public/SiteFiles/documents/NYPD_Report-Radicalization_in_the_West.pdf
Silke A (2001) The devil you know: continuing problems with research on Terrorism. Terrorism Polit Violence 13(4):1–14
Silke A (2008) Research on terrorism: a review of the impact of 9/11 and the global QWar on terrorism. In: Chen H et al (eds) Terrorism informatics: knowledge management and data mining for homeland security. Springer, New York, pp 27–50
Smith BL (1994) Terrorism in America: pipe bombs pipe dreams. State University of New York Press, Albany
Smith BL, Damphousse KR (1998) Terrorism, politics and punishment: a test of structural-contextual theory and the ‘liberation hypothesis. Criminology 36(1):67–92
Surette R (1999) Media echoes: systematic effects of news coverage. Justice Q 16(3):601–631
Townsley M, Johnson SD, Ratcliffe JH (2008) Space time dynamics of insurgent activity in Iraq. Secur J 21:139–146
Turchie T, Puckett K (2007) Hunting the American terrorist: the FBI’s war on homegrown terror. History publishing Company, Palisades
Victoroff J (2005) The mind of the terrorist: a review and critique of psychological approaches. J Conflict Resolut 49(1):3–42
The authors would like to thank the editors, and participants in the research seminar on methodological issues in terrorism that was convened at John Jay College of Criminal Justice in October, 2010 for helpful insights. The authors would also like to thank the reviewers and Erin Miller for their thoughtful suggestions for revision. In addition, we would like to thank Nick Corsaro and Jeremy Wilson for helping us think through the data analysis plan.
This research was supported by the United States Department of Homeland Security Science and Technology Directorate’s University Program’s Division as well as its Human Factors and Behavioral Science Division both directly and through the National Consortium for the Study of Terrorism and Responses to Terrorism (START) through both research and education grants. However, any opinions, findings, and conclusions or recommendations in this document are those of the authors and do not necessarily reflect views of the U.S. Department of Homeland Security or START.
About this article
Cite this article
Chermak, S.M., Freilich, J.D., Parkin, W.S. et al. American Terrorism and Extremist Crime Data Sources and Selectivity Bias: An Investigation Focusing on Homicide Events Committed by Far-Right Extremists. J Quant Criminol 28, 191–218 (2012). https://doi.org/10.1007/s10940-011-9156-4
- Far-right violence
- Selectivity bias