Introduction

Hyperspectral imaging, also known as imaging spectroscopy, is a technique through which images are captured of geologic or outdoor environments using space-based, airborne, or unmanned aerial systems or lab-based imaging systems where each pixel of images obtained contains a reflective/emisive spectrum of the material present. Provided there is context and a reference library, each pixel can then be interpreted to identify objects in the acquired image or scene. The application of hyperspectral imaging within the context of characterizing materials over a range of scales and environments has been extensively used in other areas of science, such as petroleum exploration, mineral mapping, mine waste, invasive species mapping, radiological material detection, and climate change (Allen and Krekeler 2011; Aumann et al. 2005; Cracknell and Reading 2014; Crouvi et al. 2006; Hörig et al. 2001; Krekeler and Allen 2008; Kruse et al. 2003; Mars and Crowley 2003; Swayze et al. 2000; Ustin et al. 2002, 2009. Hyperspectral imaging technology is being increasingly explored for forensic investigations (Brito et al. 2019; Cadd et al. 2018; De Carvalho et al. 2018; Devassy and George 2020, Devassy et al. 2020, 2021; Edelman and Aalders 2018; Glomb et al. 2018; Khan et al. 2018; Książek et al. 2020; Lim and Murukeshan 2017; Murray et al. 2018; Qureshi et al. 2019; Silva et al. 2017; Xu et al. 2019). However, one current limitation is the lack of detailed, comprehensively characterized libraries optimized specifically for forensic use, especially in outdoor or geologic terranes. As hyperspectral remote sensing technology expands from airborne into unmanned aerial vehicles/systems (UAV/UAS), cameras and video, the potential application for utilizing this technology in the realm of disaster management and law enforcement is rapidly expanding. The potential for hyperspectral imaging to be integrated into investigations which aim to identify and evaluate environments involving humans, clothing, blood, and associated items is therefore significant. Relevant contexts include finding abduction victims of environmental disasters such as hurricanes, earthquakes, and landslides as well as violent crime(s), locating lost personnel or workers, or lost children or adults. This technology can also be used to document large disaster, crime or conflict scenes in essentially one image but also serially over time intervals allowing documentation of aging or changes in the image or scene.

In order to further support forensic investigations within the context of hyperspectral imaging, a library with extensive meta data supporting material identification (e.g., powder X-ray diffraction, scanning electron microscopy) has been created for the ENVI remote sensing platform for use in search and rescue operations, and emergency response operations. A summary of the methods, the content, basic features of the tool, and a discussion of intended applications are provided here.

Materials and methods

Reflective spectra were collected using an ASD (PANalytical) FieldSpec 4 Hi-Res spectroradiometer. The ASD (PANalytical) FieldSpec 4 Hi-Res spectroradiometer is a field portable instrument that can be used in indoor and outdoor environments and thus could be integrated into investigations of crime scenes in a wide range of settings. The ASD FieldSpec 4 Hi-Res spectroradiometer has a spectral range of 350–2500 nm with spectral resolutions of 3 nm at 700 nm and 8 nm at 1400 nm and 2100 nm. The instrument is equipped with a post-dispersive system for extremely low stray light which is at < 0.02% for 350–1000 nm and < 0.1% for 1000–2500 nm. The instrument uses a modular silicon array as well as a Peltier cooled InGaAs detector spectrometer platform. The values for low noise equivalent delta radiance (NeDL) are 1.1 × 10–9 W/cm2/sr/nm at 700 nm for UV/VNIR, 2.8 × 10–9 W/cm2/sr/nm at 1400 nm for NIR and 5.6 × 10–8 W/cm2/sr/nm at 2100 nm. Several previous versions of the ASD (PANalytical) FieldSpec 4 Hi-Res have been produced and there is an extensive record of using these instruments (Allen and Krekeler 2010; Barnes et al. 2020; Burke et al. 2019; Krekeler and Allen 2008; Krekeler et al. 2010; Kuester et al. 2001; Oglesbee et al. 2020).

For reflective spectroscopy targets, or samples that involved granular substrates (e.g., geomaterials, blood experiments), flat black painted plastic petri dishes were used as an experimental substrate for measurements. Reflective spectroscopy measurements were acquired under open path illumination using an ASD (PANalytical) FieldSpec 4 Hi-Res spectroradiometer with a fiber to sample distance of approximately 50 cm. Spectra were collected in 1-s intervals and multiple spectra (usually 5–10 spectra depending on sample) were averaged for each measurement using the RS3 software. Other targets such as clothing, human skin, human hair were measured using a contact probe with the probe directing on the target item. Human subject methodology and IRB materials may be found in supplemental materials.

As was appropriate for each material type, additional characterization data (e.g., powder X-ray diffraction, scanning electron microcopy, grain size analysis) is available for materials analyzed. Such an approach validates the spectrum identity, assures it is representative, and provides additional data for comparison in future forensic studies and tool applications. Powder X-ray diffraction (XRD) data was collected using a Scintag X-1 powder diffractometer. The diffractometer was equipped with a Peltier detector, using Cu Kα2 (0.1548 nm) radiation operating at 40 kV and 35 mA. Samples were typically scanned from 5° to 65° 2θ at 0.02° steps using a count time of 1 s per step for basic bulk powder patterns. The detection limit for most minerals using powder diffraction is a few weight percent.

Scanning electron microscopy (SEM) characterization was completed using a variable pressure Zeiss Supra 35VP FEG with nitrogen (N2) as the compensating gas. The instrument is equipped with an energy-dispersive spectrometry (EDS) detector EDAX2000 with a detection limit of approximately 0.1 wt% for most elements and a backscatter detector (BSD).

Other descriptive information for samples is provided as relevant. For example, grain size analysis for samples was determined using a set of standard 8″ brass sieves; compositions of clothing were recorded from information provided on tags and data from the United States Geological Survey (USGS) library was retained and incorporated.

Summary of the tool

Content

Extensive geological and environmental data is included in the content of the tool. The USGS hyperspectral library was integrated into the tool directly to provide broad natural, and built, environment data. Over 60 additional geomaterial samples are included such as specific sediment, e.g., street sediment from an urban area, sediment samples from the southern border of the US, and glacial till samples representative of the US Midwest. Papers related to further details of glacial till and urban sediment samples are available (Barnes et al. 2020; Dietrich et al. 2019).

For clothing in both wet and dry conditions, over 300 samples were analyzed. Wet conditions simulate sweat and precipitation/post-precipitation conditions. Compositional information for each item is provided as metadata. Discussion of the variability observed throughout the clothing data is available in a recent publication (Burke et al 2019).

Reflective spectra data has been collected from over 100 human subject volunteers who represented a diverse range of ages and skin tones. Spectra were collected from the top and bottom of the left and right forearms, right and left shin and calf, stomach, back, hair, clothing, shoes, and body art if applicable. Images were taken of each area that spectra were collected from as meta data. Additional information such as self-reported height, weight, personal care product use, and eating habits was provided voluntarily by each subject in the form of a questionnaire.

Reflective spectra data on blood, and aging experiments on blood, were also performed and completed using human whole blood purchased from a verified online supplier which was screened for infectious diseases prior to shipping. A volume of 5 mL of blood was pipetted onto on various substrates, such as clothing, rock, sediment, and wood. Spectral data was collected on a total of approximately 80 samples and analyzed within an hour of the blood being added. Additional spectral measurements were collected at 3, and 6, and 8 h after the blood had been added. Additional measurements were collected at more dispersed intervals out to 50 days and in some cases several weeks longer. The most significant changes were observed to occur in the first week and are interpreted to be associated with dehydration and oxidation of organic material.

Spectral data was also collected on fuel spills for gasoline, diesel and jet fuel under warm laboratory (~ 22 °C) and in cold weather outdoor conditions (− 10 to 0 °C). Experiments used sand-rich substates and involved gasoline, jet fuel A, diesel, and kerosene. An accompanying publication details the results of this work (Brum et al. 2020).

Description of the tool and access

The Spectral Search and Discovery Tool is a customized ENVI extension written using the IDL programming language designed to help users find custom targets within hyperspectral imagery. Currently, the tool is only available for Windows. The tool provides a Graphical User Interface (GUI) that runs in conjunction with ENVI, and can be launched either through ENVI’s toolbox or through a shortcut in the Windows Start menu (this will automatically launch ENVI as well). Once launched, a user can use the tool to browse spectral data from a variety of sources including human tissue samples, clothing, and earth materials which are both natural and synthetic in origin. The tool indexes this data and allows users to search, filter, and combine these spectra. The tool can then be used to create libraries to search for, and determine, possible locations of these spectra within a single hyperspectral image.

Upon install of the software, all of the spectral data is placed into a user specified location on the system. Following installation of the Spectral Search Discovery Tool, this data will be automatically located and loaded into the tool when launched. The first time the Spectral Search and Discovery tool is launched, the locations on the filesystem and the structure of the data will be saved to a binary data file on the system. The Spectral Search and Discovery tool references this binary data file to quickly search and filter through the spectral data included within the tool. The organization and names of these custom spectra will appear in a data tree within the tool’s GUI. When a user selects spectral data in the tree, the spectral signatures and metadata images from the spectral data samples are read on demand. The spectra provided within the tool can be combined into custom spectra (multiple spectra combined together to form a new spectrum) or added to a spectral library (multiple spectra that will be individually analyzed while searching through imagery). When creating a new custom spectrum, the user specifies which spectra will be included and the percent weights that will be applied to each spectrum when combining them. New custom spectra can include other custom spectra that have been previously generated.

The spectral data provided with Spectral Search and Discovery Tool, as well as custom spectra created by the user, can be used to create spectral libraries. Once a spectral library has been selected, the user can execute a search on a hyperspectral image for the spectra included in that library. The searchable spectra are first resampled to match the wavelengths of the hyperspectral image, and then the Adaptive Coherence Estimator (ACE) algorithm searches the image for the spectra within the spectral library and assigns a probability of how likely it is that each material is located each pixel of the hyperspectral image.

ACE is a common algorithm used for target identification and works especially well for atypical materials in a scene. The algorithm is derived from the generalized likelihood ratio approach and does not need to know the endmembers for a scene before executing the algorithm. The output is an image that is the same spatial size as the search image and has as many bands as spectra that were searched. Each band has values ranging from 0.0 to 1.0, indicating the probability that a material is located in a certain pixel of the image. A 0.0 value means no probability and 1.0 means a 100% match. A slider in the Spectral Search and Discovery tool’s GUI allows the user to specify a probability threshold for display. If the probability of a material is above this threshold, then the tool will highlight the pixel in red.

The tool can be accessed here.

Discussion

Comments on potential use

The tool has numerous potential applications. Specifically, within the context of disaster, emergency or law enforcement investigations, the tool is intended to be used to interpret hyperspectral images captured in outdoor environments. However, the data are likely easily transferable to indoor environments provided effects for indoor lighting can be compensated for by simply providing an illumination source and taking background lighting measurements. In outdoor contexts, the data and tool would be useful for detailed documentation of larger-scale events of hurricanes, earthquakes and landslides. Additionally the data and tool would be useful for investigating violence such as mass shooting events (Brockell 2021; Densley and Peterson 2019; Dolliver and Kearns 2019; Fox and DeLateur 2014; Hawkins et al. 2021) as well as large-scale violence such as that in Myanmar, Syria, South Sudan, and Congo (BBC 2016; Specia and Pianigiani 2021; Regan 2021; Regan and Yeung 2021; UN News 2020, 2021) among many other types of events. The data would also be useful for investigations of broader conflict such as recent events in Afghanistan (UN Report 2021), Armenia (Treisman 2020), Iraq (Green and Ward 2014), Ukraine (Nygren et al. 2018), and Yemen (Serr 2017). The tool may also be useful for monitoring border areas associated with human migration such as the U.S.–Mexico border (Androff and Tavassoli 2012; Bowen and Marshall 2008; Sapkota et al. 2006) where persons may be injured, in distress, or die. Furthermore, the tool may provide a means to investigate or document acts of atrocity, crimes against humanity, or genocide in areas that may be physically denied but accessible by UAV or other hyperspectral platforms.

Another area where this tool could be applied is the hostage-taking where events occur in remote regions in a variety of geologic environments. Although other methods exist, kidnapping is predominantly the primary method of contemporary or recent hostage-taking (Wukki et al. 2021). Hostage-taking is a global problem and represents a major security concern in several areas of the world (Aduse 2015; Albert et al. 2020; Forest 2012; Loertscher and Milton 2015, 2018; Morewitz 2019). Terrorist organizations, criminal groups, pirates, and militant groups use hostage-taking as a means to leverage policy changes and secure prisoner releases from their advisories, as well as a means of financial gain (Loertscher and Milton 2015; Koseli et al. 2020; O’Brien 2012). While the use of the tactic by any group is difficult to resolve, hostage-takings involving terrorist groups are particularly complex leaving few viable options to secure the release of hostages such as U.S. nationals (Dyrud 2021; Loertscher 2019, 2020, 2021; Loertscher and Milton 2015; Smith 1985; Strobel and Hosenball 2015). The limited options for release increase the importance of the employment of hostage rescue operations, making hostage-taking a particular concern for some groups such as U.S. and allied military forces (Dyrud 2021; Smith 1985; Loertscher 2021; Strobel and Hosenball 2015).

Hostage rescue operations in general, however, are difficult to conduct and require significant investments of intelligence collection assets to provide the precise information required to identify the location of a hostage (Loertscher 2021). Given these challenges, the tool described in this paper may be useful in the investigation and identification of hostages in international settings as the tool would contain direct exemplars (e.g., clothing, skin, hair, blood) or appropriate analogs (e.g., geologic materials, fuels).

Comments on future potential improvement

Efforts were made to capture a range of properties of human materials (skin tone and hair color); however, it is acknowledged that future efforts should be made to capture a wider range of human materials. Owing to human subject protocols, children (< 18 years of age) were excluded from the sample population and thus a detailed data set on skin and hair of children should be acquired in the future. Furthermore, the number of subjects with an age > 50 was also very low, hence a detailed data set on older persons is warranted.

No decomposition data, aside from those on the aging of human blood, was acquired. Thus, an opportunity to investigate human and animal decomposition in controlled settings exists. This would be advantageous not only for disaster and crime scenes, and zones of conflict, but also for search and recovery operations for lost hikers, hunters, and other persons.

The substrates used in the acquisition of spectral data were geologic in nature. However, great potential also exists for these experiments to be expanded to include built environment surfaces, vehicle surfaces, and road or pavement surfaces. Many of these surfaces have mineralogical or synthetic mineralogical components in either the major constituents, or in the pigments used in paint. Systematic studies of the transfer of body fluids, analogs, blood, fuel and other liquids on these substrates are therefore warranted.

The content and features of the developed tool are also broadly applicable to the investigation of many potential environmental issues as well. Examples may include fuel spills, detecting the presence or effects of water, studies of erosion, sedimentation, and specific geologic environments including (but not limited to) riparian settings, deserts, mountainous regions, and ice and snow dominated regions. The content of the tool should be particularly useful for investigations of geologic and environmental disasters in urban environments as many human related items are included.

Summary

A search and discovery tool has been developed for forensic investigations and emergency response operations that can be accessed by the forensic community and other interested scientists at no cost. The provision of this tool allows for a broad basis of extensive academic applications, applied hyperspectral imaging, and reflective spectroscopy investigations. An extensive set of human, geologic, and environmental materials was assembled and will enable extensive fundamental and applied research, as well as numerous operations by law enforcement and other organizations. These collections can be, and should be, augmented to help ensure all persons can be detected to aid in their assistance in emergency situations. The tool should be useful for law enforcement, emergency management and numerous other agencies and organizations.