A System Approach to Adaptive Multimodal Sensor Designs

  • Tao Wang
  • Zhigang Zhu
  • Robert S. Krzaczek
  • Harvey E. Rhody
Part of the Augmented Vision and Reality book series (Augment Vis Real, volume 1)


We propose a system approach to adaptive multimodal sensor designs. This approach is based on the integration of tools for the physics-based simulation of complex scenes and targets, sensor modeling, and multimodal data exploitation. The goal is to reduce development time and system cost while achieving optimal results through an iterative process that incorporates simulation, sensing, processing and evaluation. A Data Process Management Architecture (DPMA) is designed, which is a software system that provides a team development environment and a structured operational platform for systems that require many interrelated and coordinated steps. As a case study, we use an effective peripheral–fovea design as an example. This design is inspired by the biological vision systems for achieving real-time target detection and recognition with a hyperspectral/range fovea and panoramic peripheral view. This design will be simulated and evaluated by realistic scene and target simulations, and the related data exploitation algorithms will be discussed.


Hyperspectral imaging Panoramic vision System architecture Multimodal sensing Target tracking 



This work is supported by AFOSR under Award #FA9550-08-1-0199, and in part by AFRL/SN under Award No. FA8650-05-1-1853 and by NSF under Grant No. CNS-0551598.


  1. 1.
    Becklin, J.A.D.E.E., Casey, S.C., Savage, M.L.: Stratospheric observatory for infrared astronomy (sofia). In: Fowler, A. (ed.) Proceedings of SPIE, vol. 3356, p. 492. The International Society for Optical Engineering (1998)Google Scholar
  2. 2.
    Boardman, J.W., Druse, F.A., Green, R.O.: Mapping target signatures via partial unmixing of AVIRIS data. In: Fifth Annual JPL Airborne Earth Science Workshop, vol. 1, AVIRIS Workshop, pp. 23–26 (1995)Google Scholar
  3. 3.
    Diplaros, A., Geves, T., Patras, I.: Combining color and shape information for illumination-viewpoint invariant object recognition. Image Process 1(1), 1–11 (2006)CrossRefGoogle Scholar
  4. 4.
    DIRSIG Last visited October (2008)
  5. 5.
    Fibreoptic Systems Inc. 60 Moreland Rd, Unit A, Simi Valley, CA 93065, USA
  6. 6.
    Fletcher-Holmes, D.W., Harvey, A.R.: Real-time imaging with a hyperspectral fovea. J. Opt. A Pure Appl. Opt. 7, S298–S302 (2005)CrossRefGoogle Scholar
  7. 7.
    Goldberg, A.C., Stann, B., Gupta, N.: Multispectral, hyperspectral, and three-dimensional imaging research at the U.S. Army research laboratory. In: Proceedings of the Sixth International Conference of Information Fusion, 2003, vol. 1, pp. 499–506 (2003)Google Scholar
  8. 8.
    Gruninger, J., Ratkowski, A.J., Hoke, M.L.: The sequential maximum angle convex cone (SMACC) endmember model. In: Proceedings of SPIE, Algorithms for Multispectral and Hyper-spectral and Ultraspectral Imagery, vol. 5425(1), Orlando, FL (2004)Google Scholar
  9. 9.
    Harvey, A.R., Fletcher-Holmes, D.W.: Imaging apparatus. GB Patent Application (0215)248.6 (2002)Google Scholar
  10. 10.
    Horn, S.: DARPA’s Adaptive Focal Plane Array (AFPA) Program. Available:
  11. 11.
    Headwall Photonics Inc., 601 River Street, Fitchburg, MA 01420, USA
  12. 12.
    Horn, S.: DARPA’s Adaptive Focal Plane Array (AFPA) Program. Available: Last visited (2007)
  13. 13.
    Kincade, K.: MIRTHE center aims to take mid-IR sensors to new heights. (Optical Sensing). Laser Focus World (2006)Google Scholar
  14. 14.
    Kruse, F.A., et al.: The spectral image processing system (SIPS)—interactive visualization and analysis of imaging spectrometer data. Rem. Sens. Environ. 44, 145–163 (1993)CrossRefGoogle Scholar
  15. 15.
    Land, M.F., Nilsson, D.-E.: Animal Eyes. Oxford University Press, Oxford (2002)Google Scholar
  16. 16.
    Schott, J.R., Brown, S.D., Raqueño, R.V., Gross, H.N., Robinson, G.: An advanced synthetic image generation model and its application to multi/hyperspectral algorithm development. Can. J. Rem. Sens. 25(2), 99–111 (1999)Google Scholar
  17. 17.
    Schott, J.R.: Remote Sensing: The Image Chain Approach, 2nd edn. Oxford University Press, Oxford (2007)Google Scholar
  18. 18.
    Soille, P.: Morphological Image Analysis: Principle and Applications. Springer, Berlin (1999)Google Scholar
  19. 19.
    Start, T.M., Fischler, M.A.: Context-based vision: recognizing objects using information from both 2D and 3D imagery. IEEE Trans. Pattern Anal. Mach. Intell. 13(10), 1050–1065 (1991)CrossRefGoogle Scholar
  20. 20.
    Winter, M.F.: N-FINDR: an algorithm for fast autonomous spectral end-member determination in hyperspectral data. In: Proceedings of SPIE, vol. 3753, pp. 266–275 (1999)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Tao Wang
    • 1
  • Zhigang Zhu
    • 1
  • Robert S. Krzaczek
    • 2
  • Harvey E. Rhody
    • 2
  1. 1.Department of Computer ScienceCity College of New YorkNYUSA
  2. 2.Chester F. Carlson Center for Imaging ScienceRochester Institute of TechnologyNYUSA

Personalised recommendations