Skip to main content
SpringerLink
Log in

Development of On-Board Data Compression Technology at Canadian Space Agency

  • Chapter
  • First Online:
Book cover Satellite Data Compression

Abstract

This chapter reviews and summarizes the researches and developments on data compression techniques for satellite sensor data at the Canadian Space Agency in collaboration with its partners in other government departments, academia and Canadian industry. This chapter describes the subject matters in the order of the following sections.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 54.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. S.-E. Qian, A. Hollinger, D. Williams and D. Manak, “Fast 3D data compression of hyperspectral imagery using vector quantization with spectral-feature-based binary coding,” Opt. Eng. 35, 3242–3249 (1996) [doi:10.1117/1.601062]

    Article  Google Scholar 

  2. S.-E. Qian, A. Hollinger, D. Williams and D. Manak, “A near lossless 3-dimensional data compression system for hyperspectral imagery using correlation vector quantization,” Proc. 47th Inter. Astron. Congress, Beijing, China (1996).

    Google Scholar 

  3. S.-E. Qian, A. Hollinger, D. Williams and D. Manak, “3D data compression system based on vector quantization for reducing the data rate of hyperspectral imagery,” in Applications of Photonic Technology II, G. Lampropoulos, Ed., pp. 641–654, Plenum Press, New York (1997)

    Google Scholar 

  4. D. Manak, S.-E. Qian, A. Hollinger and D. Williams, “Efficient Hyperspectral Data Compression using vector Quantization and Scene Segmentation,” Canadian J. Remote Sens., 24, 133–143 (1998).

    Google Scholar 

  5. S.-E. Qian, A. Hollinger, D. Williams and D. Manak, “3D data compression of hyperspectral imagery using vector quantization with NDVI-based multiple codebooks,” Proc. IEEE Geosci. Remote Sens. Symp., 3, 2680–2684 (1998).

    Google Scholar 

  6. S.-E. Qian, A. Hollinger, D. Williams and D. Manak, “Vector quantization using spectral index based multiple sub-codebooks for hyperspectral data compression,” IEEE Trans. Geosci. Remote Sens., 38(3), 1183–1190 (2000) [doi:10.1109/36.843010]

    Article  Google Scholar 

  7. S.-E. Qian, “Hyperspectral data compression using a fast vector quantization algorithm,” IEEE Trans. Geosci. Remote Sens., 42(8), 1791–1798 (2004) [doi: 10.1109/TGRS.2004.830126]

    Article  Google Scholar 

  8. S.-E. Qian, “Fast vector quantization algorithms based on nearest partition set search,” IEEE Trans. Image Processing, 15(8), 2422–2430 (2006) [doi:10.1109/TIP.2006.875217]

    Article  Google Scholar 

  9. S.-E. Qian and A. Hollinger, “Current Status of Satellite Data Compression at Canadian Space Agency,” Invited chapter in Proc. SPIE 6683, 04.01-12 (2007).

    Google Scholar 

  10. S.-E. Qian, and A. Hollinger, “System and method for encoding/decoding multi-dimensional data using Successive Approximation Multi-stage Vector Quantization (SAMVQ),” U. S. Patent No. 6,701,021 B1, issued on March 2, 2004.

    Google Scholar 

  11. S.-E. Qian and A. Hollinger, “Method and System for Compressing a Continuous Data Flow in Real-Time Using Cluster Successive Approximation Multi-stage Vector Quantization (SAMVQ),” U.S. Patent No. 7,551,785 B2, issued on June 23, 2009.

    Google Scholar 

  12. S.-E. Qian, and A. Hollinger, “System and method for encoding multi-dimensional data using Hierarchical Self-Organizing Cluster Vector Quantization (HSOCVQ),” U. S. Patent No. 6,724,940 B1, issued on April 20, 2004.

    Google Scholar 

  13. S.-E. Qian and A. Hollinger, “Method and System for Compressing a Continuous Data Flow in Real-Time Using Recursive Hierarchical Self-Organizing Cluster Vector Quantization (HSOCVQ),” U.S. Patent No. 6,798,360 B1 issued on September 28, 2004.

    Google Scholar 

  14. S.-E. Qian, M. Bergeron, I. Cunningham, L. Gagnon and A. Hollinger, "Near Lossless Data Compression On-board a Hyperspectral Satellite," IEEE Trans. Aerosp. Electron. Syst., 42(3), 851-866 (2006) [doi:10.1109/TAES.2006.248183]

    Article  Google Scholar 

  15. S.-E. Qian, A. Hollinger and Yann Hamiaux, “Study of real-time lossless data compression for hyperspectral imagery,” Proc. IEEE Int. Geosci. Remote Sens. Symp., 4, 2038–2042 (1999)

    Google Scholar 

  16. S-E Qian and A Hollinger, “Applications of wavelet data compression using modified zerotrees in remotely sensed data,” Proc. IEEE Geosci. Remote Sens. Symp., 6, 2654–2656 (2000)

    Google Scholar 

  17. CCSDS, “Lossless Data Compression,” CCSDS Recommendation for Space Data System Standards, Blue Book 120.0-B-1, May 1997.

    Google Scholar 

  18. S.-E. Qian, “Difference Base-bit Plus Overflow-bit Coding,” Journal of Infrared & Millimeter Waves, 11(1), 59–64 (1992).

    Google Scholar 

  19. J. M. Shapiro, "Embedded image coding using zerotrees of wavelet coefficients," IEEE Trans. Signal Processing, 41, 3445–3462 (1993).

    Article  MATH  Google Scholar 

  20. A. Said and W.A. Pearlman, “A new, fast and efficient image codec based on set partitioning in hierarchical trees,” IEEE Trans. on Circuits and Systems for Video Technology, 6, 243–250 (1996).

    Article  Google Scholar 

  21. A. Gersho and R.M. Gray, Vector Quantization and Signal Compression, Boston, MA: Kluwer, 1992.

    Book  MATH  Google Scholar 

  22. K. Chen and T.V. Ramabadran, “Near-lossless compression of medical images through entropy-coded DPCM,” IEEE Trans. Med. Imaging 13, 538–548 (1994) [doi:10.1109/42.310885]

    Article  Google Scholar 

  23. X. Wu and P. Bao, “Constrained high-fidelity image compression via adaptive context modeling,” IEEE Trans. Image Processing, 9, 536–542 (2000) [doi:10.1109/83.841931]

    Article  MATH  Google Scholar 

  24. B Aiazzi, L. Alparone and S. Baronti, “Near-lossless compression of 3-D optical data,” IEEE Trans. Geosci. Remote Sens., 39(11), 2547–2557 (2001) [doi:10.1109/36.964993]

    Article  Google Scholar 

  25. E. Magli, G. Olmo and E. Quacchio, “Optimized onboard loeelsess and near-lossless compression of hyperspectral data using CALIC,” IEEE Geosci. Remote Sens. Lett., 1, 21–25 (2004) [doi:10.1109/LGRS.2003.822312]

    Article  Google Scholar 

  26. Low altitude AVIRIS data of the Greater Victoria Watershed District, http://aviris.jpl.nasa.gov/ql/list02.html

  27. A. Hollinger, M. Bergeron, M. Maszkiewicz, S.-E. Qian, K. Staenz, R.A. Neville and D.G. Goodenough, “Recent Developments in the Hyperspectral Environment and Resource Observer (HERO) Mission” Proc. IEEE Geosci. Remote Sens. Symp., 3, 1620–1623 (2006).

    Article  Google Scholar 

  28. S.-E. Qian, M. Bergeron, A. Hollinger and J. Lévesque, “Effect of Anomalies on Data Compression Onboard a Hyperspectral Satellite”, Proc. SPIE 5889, (02)1–11 (2005)

    Google Scholar 

  29. S.-E. Qian, M. Bergeron, J. Lévesque and A. Hollinger, “Impact of Pre-processing and Radiometric Conversion on Data Compression Onboard a Hyperspectral Satellite,” Proc. IEEE Geosci. Remote Sens. Symp., 2, 700–703 (2005).

    Google Scholar 

  30. S.-E. Qian, B. Hu, M. Bergeron, A. Hollinger and P. Oswald, “Quantitative evaluation of hyperspectral data compressed by near lossless onboard compression techniques,” Proc. IEEE Geosci. Remote Sens. Symp., 2, 1425–1427 (2002).

    Google Scholar 

  31. S.-E. Qian, A. Hollinger, M. Bergeron, I. Cunningham, C. Nadeau, G. Jolly and H. Zwick, “A Multi-disciplinary User Acceptability Study of Hyperspectral Data Compressed Using Onboard Near Lossless Vector Quantization Algorithm,” Inter. J. Remote Sens., 26(10), 2163–2195 (2005) [doi:10.1080/01431160500033500]

    Article  Google Scholar 

  32. C. Nadeau, G. Jolly and H. Zwick, “Evaluation of user acceptance of compression of hyperspectral data cubes (Phase I),” final report of Canadian Government Contract No. 9F028-013456/001MTB, Feb. 6, 2003.

    Google Scholar 

  33. C. Nadeau, G. Jolly and H. Zwick, “Evaluation of user acceptance of compression of hyperspectral data cubes (Phase II),” final report of Canadian Government Contract No. 9F028-013456/001MTB, July 24, 2003.

    Google Scholar 

  34. S.-E. Qian, A. B. Hollinger, M. Dutkiewicz and H. Zwick, “Effect of Lossy Vector Quantization Hyperspectral Data Compression on Retrieval of Red Edge Indices,” IEEE Trans. Geosc. Remote Sens., 39, 1459–1470 (2001) [doi:10.1109/36.934077]

    Article  Google Scholar 

  35. S.-E. Qian, M. Bergeron, C. Serele and A. Hollinger, “Evaluation and comparison of JPEG 2000 and VQ based on-board data compression algorithm for hyperspectral imagery,” Proc. IEEE Geosci. Remote Sens. Symp., 3, 1820–1822 (2003).

    Google Scholar 

  36. B. Hu, S.-E. Qian and A. Hollinger, “Impact of lossy data compression using vector quantization on retrieval of surface reflectance from CASI imaging spectrometry data,” Canadian J. Remote Sens., 27, 1–19 (2001).

    MATH  Google Scholar 

  37. B. Hu, S.-E. Qian, D. Haboudane, J.R. Miller, A. Hollinger and N. Tremblay, “Impact of Vector Quantization Compression on Hyperspectral Data in the Retrieval Accuracies of Crop Chlorophyll Content for Precision Agriculture,” Proc. IEEE Geosci. Remote Sens. Symp., 3, 1655–1657 (2002).

    Google Scholar 

  38. B. Hu, S.-E. Qian, D. Haboudane, J. R. Miller, A. Hollinger and N. Tremblay, “Retrieval of crop chlorophyll content and leaf area index from decompressed hyperspectral data: the effects of data compression,” Remote Sens. Environ., 92(2), 139–152 (2004) [doi:10.1016/j.rse.2004.05.009]

    Article  Google Scholar 

  39. K. Staenz, R. Hitchcock, S.-E. Qian and R.A. Neville, “Impact of on-board hyperspectral data compression on mineral mapping products,” Int. ISPRS Conf. 2002, India (2002).

    Google Scholar 

  40. K. Staenz, R. Hitchcock, S.-E. Qian, C. Champagne and R.A. Neville, “Impact of On-Board Hyperspectral Data Compression on Atmospheric Water Vapour and Canopy Liquid Water Retrieval,” Int. ISSSR Conf. (2003).

    Google Scholar 

  41. C. Serele, S.-E. Qian, M. Bergeron, P. Treitz, A. Hollinger and F. Cavayas, “A Comparative Analysis of two Compression Algorithms for Retaining the Spatial Information in Hyperspectral Data,” Proc. 25 th Canadian Remote Sens. Symp., Montreal, Canada, 14–16 (2003).

    Google Scholar 

  42. A. Dyk, D.G. Goodenough, S. Thompson, C. Nadeau, A. Hollinger and S.-E. Qian, “Compressed Hyperspectral Imagery for Forestry,” Proc. IEEE Geosci. Remote Sens. Symp. 1, 294–296 (2003).

    Google Scholar 

  43. P. Zarrinkhat and S.-E. Qian, “Enhancement of Resilience to Bit-Errors of Compressed Data On-board a Hyperspectral Satellite using Forward Error Correction” Proc. SPIE 7084, 07.1–9 (2008)

    Google Scholar 

  44. S.-E. Qian, A. Hollinger and L. Gagnon, “Data Compression Engines and Real-Time Wideband Compressor for Multi-Dimensional Data” U.S. Patent No. 7,251,376 B2, issued on July 31, 2007.

    Google Scholar 

  45. Consultative Committee for Space Data System (CCSDS), http://public.ccsds.org/default.aspx.

  46. S.-E. Qian, “Study of hyperspectral and multispectral images compression using vector quantization in development of CCSDS international standards” Proc. SPIE 7477A, 23.1–11 (2009)

    Google Scholar 

  47. M. Bergeron, K. Kolmaga and S.-E. Qian “Assessment of Keystone Impact on VQ Compression Fidelity,” Canadian Space Agency internal technical report on May 20th, 2003.

    Google Scholar 

  48. M. Bergeron, K. Kolmaga and S.-E. Qian “Assessment of Spectral Curvature Impact on VQ Compression Fidelity,” Canadian Space Agency internal technical report on May 20th, 2003.

    Google Scholar 

Download references

Acknowledgments

The author would like to thank his colleagues A. Hollinger, M. Bergeron, I. Cunningham and M. Maszkiewicz; his post-doctor visiting fellows C. Serele, H. Othman and P. Zarrinkhat, and over 30 internship students, for their contributions to the work summarized in this chapter. The author thanks D. Goodenough at the Pacific Forestry Centre, Natural Resources Canada, K. Staenz (now at University of Lethbridge), L. Sun and R. Neville at the Canada Center for Remote Sensing, Natural Resources Canada, J. Levesque and J.-P. Ardouin at the Defence Research and Development Canada, J. Miller and B. Hu at York University, for providing data sets and for actively collaborating on the user acceptability study and the impact assessments. The author thanks the following users for their participation and contribution to the multi-disciplinary user acceptability study: A. Dyk at the Pacific Forestry Centre, B. Ricketts and N. Countway at Satlantic Inc., J. Chen at University of Toronto, H. Zwick, C. Nadeau, G. Jolly, M. Davenport and J. Busler at MacDonald Dettwiler Associates (MDA), M. Peshko at Noranda/Falconbridge, B. Rivard and J. Feng at the University of Alberta, J. Walls and R. McGregor at RJ Burnside International Ltd., M. Carignan and P. Hebert at Tecsult, J. Huntington and M. Quigley at the Commonwealth Scientific and Industrial Research Organization in Australia, R. Hitchcock at the Canada Centre for Remote Sensing. The author thanks L. Gagnon, W. Harvey, B. Barrette, and C. Black at former EMS Technologies Canada Ltd. (Now MDA Space Missions) and the technical teams for the development and fabrication of the hardware compression prototypes. The author thanks V. Szwarc and M. Caron at Communication Research Centre, Canada for discussion on enhancement of the resilience to bit-errors of the compression techniques, and P. Oswald and R. Buckingham for discussion on on-board data compression. The author also thanks the CCSDS MHDC Working Group for providing the test data sets and the members of the working group for providing the compression results. The Work in the chapter was created by a public servant acting in the course of his employment for the Government of Canada and within the scope of his duties in writing the Work, whereas the copyright in the Work vests in Her Majesty the Queen in right of Canada for all intents and purposes under the Copyright Act of Canada© Government of Canada 2011.

Author information

Authors and Affiliations

  1. Canadian Space Agency, St-Hubert, QC, Canada

    Shen-En Qian

Authors
  1. Shen-En Qian
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shen-En Qian .

Editor information

Editors and Affiliations

  1. , Space Science and Engineering Center, University of Wisconsin - Madison, Madison, 53706, Wisconsin, USA

    Bormin Huang

Rights and permissions

Reprints and permissions

Copyright information

© 2012 Springer Science+Business Media, LLC

About this chapter

Cite this chapter

Qian, SE. (2012). Development of On-Board Data Compression Technology at Canadian Space Agency. In: Huang, B. (eds) Satellite Data Compression. Springer, New York, NY. https://doi.org/10.1007/978-1-4614-1183-3_1

Download citation

  • DOI: https://doi.org/10.1007/978-1-4614-1183-3_1

  • Published:

  • Publisher Name: Springer, New York, NY

  • Print ISBN: 978-1-4614-1182-6

  • Online ISBN: 978-1-4614-1183-3

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation