Optical Remote Sensing pp 49-64 | Cite as

# Integrated Sensing and Processing for Hyperspectral Imagery

## Abstract

In this chapter, we present an information sensing system which integrates sensing and processing resulting in the direct collection of data which is relevant to the application. Broadly, integrated sensing and processing (ISP) considers algorithms that are integrated with the collection of data. That is, traditional sensor development tries to come up with the “best” sensor in terms of SNR, resolution, data rates, integration time, and so forth, while traditional algorithm development tasks might wish to optimize probability of detection, false alarm rate, and class separability. For a typical automatic target recognition (ATR) problem, the goal of ISP is to field algorithms which “tell” the sensor what kind of data to collect next and the sensor alters its parameters to collect the “best” information in order that the algorithm performs optimally. We illustrate the concept of ISP using a near Infrared (NIR) hyperspectral imaging sensor. This prototype sensor incorporates a digital mirror array (DMA) device in order to realize a Hadamard multiplexed imaging system. Specific Hadamard codes can be sent to the sensor to realize inner products of the underlying scene rather than the scene itself. The developed ISP algorithms utilize these codes to overcome issues traditionally associated with hyperspectral imaging (i.e. Data Glut and SNR issues) while also performing a object detection task. The underlying integration of the sensing and processing results in algorithms which have better overall performance while collecting less data.

### Keywords

Hyperspectral imaging Adaptive imaging Compressive imaging Hadamard multiplexing## 1 Introduction

This chapter presents the development of algorithms for Integrated Sensing and Processing (ISP) utilizing a hyperspectral imaging sensor. The ISP paradigm seeks to determine the best sensing parameters for achieving the performance objectives of a given algorithm. The exploitation algorithm may also have components which adapt to the imagery being sensed. In this context, ISP is a coupling between adaptive algorithms and adaptive sensing. Considering the problem of object detection/classification in hyperspectral imagery, ISP can increase sensing and algorithm performance in several ways. Firstly, hyperspectral exploitation usually suffers from a data glut problem. That is, a hyperspectral sensor generates a cube of data where each spatial pixel is represented as a spectral vector. The first step in most exploitation algorithms is some type of data reduction, or spectral band selection. A question which should naturally arise is: Why sense particular information which is going to be immediately eliminated through a data reduction algorithm? If one can design a data collection system that integrates the sensor with the data reduction algorithm, then only information which is pertinent to the exploitation task need be sensed. Secondly, traditional hypserspectral imagers can suffer SNR degradation as compared with broadband imagers. When one is attempting high spatial resolution imaging and the sensing system separates the light into a large number of spectral components, then there is a significant loss of photons being sensed by the detector array. Thus, to get enough light to make a meaningful image, one must increase the detector integration time. If one is sensing a dynamic scene, longer integration time cannot be tolerated; which leads to significant loss of SNR in the final hyperspectral image. In Sect. 2, we show a solution to this SNR issue using spatial/spectral multiplexing.

In order to investigate algorithms for integrated sensing and processing of imagery, we use a near infrared (NIR) Hadamard multiplexing imaging sensor. This prototype sensor was developed by PlainSight Systems (PSS) and incorporates a digital mirror array (DMA) device in order to realize a Hadamard multiplexed imaging system. The known Signal-to-Noise (SNR) advantage in Hadamard spectroscopy [1] extended to imaging systems [2, 3] allows for the collection of a hyperspectral cube of data with more efficient light collection over standard “Pushbroom” hyperspectral imagers.

**.**

The PlainSight NSTIS sensor implements the process from Fig. 3 where the detector array is a standard Indigo Phoenix large-area InGaAs camera operating in the Near Infrared wavelengths.

During standard operation of the system, the sensor collects 512 raw frames of data. Each frame is 522 × 256 pixels and represents superposition of spectra vs. spatial row as shown in Fig. 3. The 512 frames are collected using the 256 Walsh (0 and 1 s) patterns that determine the columns of the DMA to be opened or closed. In other words, each column of the DMA is controlled by a bit of the Walsh code. If the bit is 0, the column is closed whereas if the bit is 1, the column is open. Since the theory of optimal SNR is based upon Hadamard (1 and −1 s) patterns, one needs to collect two Walsh patterns to generate a single Hadamard pattern. Thus, the 512 collected frames represent the required Walsh patterns to form a full set of 256 Hadamard patterns. Since each column in the DMA array will hit the Diffraction grating at a different location, the spectra will hit the detector array at a different location per column. We refer to this as a *Skewness* in spectra which spreads the information across 522 pixels in the spectral dimension but represents only 266 actual spectral bins. Of course, this spatial/spectral mixing and skewness is invertable once all 256 Hadamard patterns have been collected. The resultant hyperspectral scene is dimension 256 × 256 with 266 spectral bands from 900 to 1,700 nm.

Given a sensor that accommodates adaptation while imaging, the ISP concepts we will discuss can be viewed as within the realm of compressive sensing (as presented by Donoho [4] and Candes et al. [5]) in that we will collect far fewer image samples than what would normally be required to exploit the entire scene of interest. Neifeld and Shankar [6] have done similar work on concepts for feature-specific imaging while Mahalanobis and Daniel [7] have looked at exploitation driven compression algorithms (another form of ISP).

The outline of this chapter is as follows. Section 2 presents an algorithm for variable resolution sensing where high resolution imagery is driven by an ATR metric. Section 3 presents the results of an experiment which demonstrates the developed algorithms implemented in a prototype ISP hyperspectral sensor, while Sect. 4 presents concluding remarks and future work.

## 2 Variable Resolution Hyperspectral Sensing

### 2.1 Mathematical Representation

*S*(λ,

*r*,

*c*), we will consider only a particular column of data

*S*(λ,

*r*). We wish to establish a correspondence between the sampling of the hyperspectral row,

*S*(λ,

*r*), as a digital image and a particular mirror of the DMA device. As described in the description of Fig. 3, each row of the scene hits the diffraction grating at a different place, and thus the entire spectrum is shifted on the focal plane as a function of the row. This is referred to as spectral “skewness”. Thus, as a particular row enters the system, the underlying scene actually becomes

*S*(λ(

*r*),

*r*), where the spectrum is now a function of row. We now make the substitution ω = λ(

*r*) and ignore this dependency for the moment. So we are concerned with sensing the hyperspectral row image

*S*(ω,

*r*). The sampling of this function brought about from the DMA generates a matrix

*S*of dimension 522 × 256. We are thus interested in sensing this array with Hadamard vectors of length 256. An example scene matrix, S, is given in Fig. 4. Recall, this is a spectral x spatial data matrix, so there is no intrinsic interpretability.

^{256}as shown in the example below which is of dimension 8.

*h*

_{i}. This ith frame of collected data is the 522 × 1 vector

**S**, from the sensed frames,

**F**, we use Eq. 1 to get

This implies that if all 256 Hadamard vectors are sequentially encoded into the mirror array and sensed through the camera, then we can fully recover S from the actual collected data F. Performing this recovery on all spatial columns of this data will recover the full hyperspetral data cube.

### 2.2 Reduced Resolution Imaging

_{64}be the 64 × 64 matrix of all 1 s, then it can be shown that

So the underlying scene is approximated by averaging. (i.e. the first 64 columns of S are averaged and establish the first 64 columns of the approximation). Again, the dimensions of the matrix *S*(ω, *r*), are wavelength, ω, and spatial row, *r*, implying that the spatial row information in \( \hat{S} \) is the average of 64 spatial rows of the scene: A low pass filtering in the spatial row dimension. In the wavelength dimension it is somewhat more complicated. It would appear that the data in the wavelength dimension is not smoothed along the wavelength axis, but simply averaged over 64 spatial rows. However, we recall that ω = λ(*r*) is a function of the real spectral parameter which has an index which is a function of the spatial row. Thus, \( \hat{S} \), the coarse scale approximation to S, smoothes S in both the wavelength and spatial row dimensions.

At this point, one can apply a metric to the reduced resolution imagery which defines which spatial areas are to be sensed at a finer resolution. The process can then continue until the highest resolution possible is achieved over the spatial areas desired by the controlling metric.

*H*

_{64,4}. Thus, we will sense the scene as

*k*

_{1},

*k*

_{2}, …,

*k*

_{M}}defining the local resolution and are determined iteratively by the controlling criteria metric. For example, a spectral MACH [8] filter could be inserted at this stage as a controlling criteria for finer resolution sampling.

## 3 Experimental Results

### 3.1 Improving SNR Using Hadamard Multiplexing

### 3.2 Variable Resolution Hyperspectral Sensing

^{1}norm is used for comparison and if this norm is smaller than a defined threshold, then that resolution cell is identified as requiring more resolution. The sensing continues in this manner until the highest possible resolution is attained. Figure 14 shows band 210 of the hyperspectral scene used for training. With the training signature calculated, the sequence of collected frames is shown in Fig. 15.

Essentially, the image cube is further resolved by applying additional Hadamard vectors to sense only in regions where there is a potential match with the object of interest as determined by the response of the filter to low-resolution data. Large portions of the scene in the background and fore-ground are discontinued early in the sensing process, whereas resolution is progressively added only to the region that exhibit peaks that are potentially due to the car. This approach improves the sensing process by greatly reducing the overall volume of data and the time required to collect it. In the end, only the spatial information that is salient for the object recognition algorithm to recognize the car is gathered in detail.

## 4 Summary

The concept of Integrated Sensing and Processing (ISP) is a unique way to address the issue of large amounts of data associated with hyperspectral imaging. Typically, much of the data collected by a conventional sensor is often not of interest and discarded during analysis. In this chapter, we discussed a coded aperture hyperspectral imager that allows data to be collected at variable resolution by dynamically controlling the aperture. In an ISP framework, the sensor can collect relevant information only in areas where features (or objects) of interest may be present, and thereby greatly reduce the amount of raw data that needs to be sensed.

Specifically, we first described the conceptual design of the coded aperture hyperspectral imager developed by Plain Sight Systems [9]. It is noteworthy that the raw data sensed by this instrument is not a hyperspectral image, but a mix of coded spatial and spectral information which must be digitally processed to recover the hyperspectral data cube. We presented the algebraic framework for reconstructing the hyperspectral data cube using the Hadamard transform matrix, and described a method for varying resolution in the reconstructed scene.

The coded aperture imager’s ability to collect less data than a conventional sensor was shown by means of illustrative examples. The essence of the experiments shows that raw data can be collected sparsely across the scene, driven by performance metrics such as pattern match criteria and therefore only a fraction of the underlying pixel need to be sensed. Fundamentally, it becomes possible to retain the salient information in the scene while avoiding the need to measure irrelevant information. This has the potential to significantly reduce the requirements for data links and on-board storage in future generation of sensors that are based on the ISP paradigm.

### References

- 1.Harwit, M., Sloane, N.J.A.: Hadamard Transform Optics. Academic Press, New York (1979)MATHGoogle Scholar
- 2.DeVerse, R.A., Hammaker, R.M., Fately, W.G.: Realization of the Hadamard multiplex advantage using a programmable optical mask in a dispersive flat-field near-infrared spectrometer. Appl. Spectrosc.
**54**(12), 1751–1758 (2000)CrossRefGoogle Scholar - 3.Wuttig, A., Riesenberg, R.: Sensitive Hadamard transform imaging spectrometer with a simple MEMS. In: Proceedings of SPIE 4881, pp. 167–178 (2003)Google Scholar
- 4.Donoho, D.L.: Compressed sensing, Stanford University: Department of Statistics report 2004-25, October, 2004Google Scholar
- 5.Candès, E., Romberg, J., Tao, T.: Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information. IEEE Trans. Inf. Theory
**52**, 489–509 (2006)CrossRefGoogle Scholar - 6.Neifeld, M.A., Shankar, P.: Feature-specific imaging. Appl. Opt.
**42**, 3379–3389 (2003)CrossRefGoogle Scholar - 7.Mahalanobis, A., Daniell, C.: Data compression and correlation filtering: a seamless approach to pattern recognition. In: Javidi, B. (ed.) Smart Imaging Systems. SPIE Press, Bellingham (2001)Google Scholar
- 8.Mahalanobis, A., Vijaya Kumar, B.V.K., Song, S., Sims, S.R.F., Epperson, J.F.: Unconstrained correlation filters. App. Opt.
**33**(17), 3751–3759 (1994)CrossRefGoogle Scholar - 9.Fateley, W.G., Coifman, R.R., Geshwind, F., DeVerse, R.A.: System and method for encoded spatio-spectral information processing. US Patent # 6,859,275, February 2005Google Scholar