Outlier-Resilient Entropy Coding

  • Jordi Portell
  • Alberto G. Villafranca
  • Enrique García-Berro


Many data compression systems rely on a final stage based on an entropy coder, generating short codes for the most probable symbols. Images, multispectroscopy or hyperspectroscopy are just some examples, but the space mission concept covers many other fields. In some cases, especially when the on-board processing power available is very limited, a generic data compression system with a very simple pre-processing stage could suffice. The Consultative Committee for Space Data Systems made a recommendation on lossless data compression in the early 1990s, which has been successfully used in several missions so far owing to its low computational cost and acceptable compression ratios. Nevertheless, its simple entropy coder cannot perform optimally when large amounts of outliers appear in the data, which can be caused by noise, prompt particle events, or artifacts in the data or in the pre-processing stage. Here we discuss the effect of outliers on the compression ratio and we present efficient solutions to this problem. These solutions are not only alternatives to the CCSDS recommendation, but can also be used as the entropy coding stage of more complex systems such as image or spectroscopy compression.


Compression Ratio Data Block Compression Efficiency Entropy Coder Prefix Code 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    J. Portell, E. García-Berro, X. Luri, and A. G. Villafranca, “Tailored data compression using stream partitioning and prediction: application to Gaia,” Experimental Astronomy 21, 125–149 (2006).CrossRefGoogle Scholar
  2. 2.
    CCSDS-101.0-B-5 Blue Book, Telemetry channel coding, 2001.Google Scholar
  3. 3.
    D. Solomon, Data Compression. The complete reference, Springer, 2004.Google Scholar
  4. 4.
    CCSDS-121.0-B-1 Blue Book, Lossless data compression, 1993.Google Scholar
  5. 5.
    CCSDS-120.0-G-2 Informational Report, Lossless data compression, 2006.Google Scholar
  6. 6.
    R. F. Rice, “Some practical universal noiseless coding techniques,” JPL Technical Report 79–22 (1979).Google Scholar
  7. 7.
    S. W. Golomb, “Run-lengths encodings,” IEEE Transactions on Information Theory 12, 399–401 (1966).MathSciNetzbMATHCrossRefGoogle Scholar
  8. 8.
    P.-S. Yeh, “Implementation of CCSDS lossless data compression for space and data archive applications,” Proc. CCSDS Space Operations Conf., 60–69, 2002.Google Scholar
  9. 9.
    P.-S. Yeh, P. Armbruster, A. Kiely, B. Masschelein, G. Moury, C. Schaefer, and C. Thiebaut, “The new CCSDS image compression recommendation,” IEEE Aerospace Conf., 4138–4145, 2005.Google Scholar
  10. 10.
    M. Clotet, J. Portell, A. G. Villafranca, and E. García-Berro, “Simple resiliency improvement of the CCSDS standard for lossless data compression,” Proc. SPIE 7810, 2010.Google Scholar
  11. 11.
    J. Portell, A. G. Villafranca, and E. García–Berro, “Designing optimum solutions for lossless data compression in space,” Proc. ESA On-Board Payload Data Compression Workshop, 35–44, 2008.Google Scholar
  12. 12.
    J. Portell, A. G. Villafranca, and E. García-Berro, “A resilient and quick data compression method of prediction errors for space missions,” Proc. SPIE 7455, 2009.Google Scholar
  13. 13.
    P.-S. Yeh, R. Rice, and W. Miller, “On the optimality of code options for a universal noiseless coder,” JPL Technical Report 91–2 (1991).Google Scholar
  14. 14.
    D. Huffman, “A method for the construction of minimum redundancy codes,” Proc. IRE 40, 1098–1101 (1952).CrossRefGoogle Scholar
  15. 15.
    I. H. Witten, R. M. Neal, and J. G. Cleary, “Arithmetic coding for data compression,” Communicat. ACM 30, 520–540 (1987).CrossRefGoogle Scholar
  16. 16.
    Teuhola, J., “A compression method for clustered bit-vectors,” Information Processing Letters 7(6), 308–311 (1978).zbMATHCrossRefGoogle Scholar
  17. 17.
    Howard, P. and Vitter, J., “Fast progressive lossless image compression,” in Image and Video Compression Conference, SPIE, 98–109 (1994).Google Scholar
  18. 18.
    M. A. C. Perryman, K. S. de Boer, G. Gilmore, E. Hoeg, M. G. Lattanzi, L. Lindegren, X. Luri, F. Mignard, O. Pace, and P. T. Zeeuw, “Gaia: Composition, formation and evolution of the Galaxy,” Astronomy & Astrophysics 369, 339–363 (2001).CrossRefGoogle Scholar
  19. 19.
    Nieto-Santisteban, M. A., Fixsen, D. J., Offenberg, J. D., Hanisch, R. J. & Stockman, H. S., “Data Compression for NGST”, in Astronomical Data Analysis Software and Systems VIII, vol. 172 of Astronomical Society of the Pacific Conference Series, 137–140 (1999).Google Scholar
  20. 20.
    C. E. Shannon, “A mathematical theory of communication,” Bell system technical journal, vol. 27, 1948.Google Scholar
  21. 21.
    A. Kiely and M. Klimesh, “Generalized Golomb codes and adaptive coding of wavelettransformed image subbands,” JPL Technical Report, IPN 42–154 (2003).Google Scholar
  22. 22.
    C. Babusiaux, “The Gaia Instrument and Basic Image Simulator,” in The Three-Dimensional Universe with Gaia, ESA SP-576, 125–149 (2005).Google Scholar
  23. 23.
    F. Murtagh and R. H.Warmels, “Test image descriptions,” in Proc. 1st ESO/ST-ECF Data Analysis Workshop, 17(6), 8–19 (1989).Google Scholar
  24. 24.
    Portell, J., Villafranca, A. G., and García-Berro, E., “Quick outlier-resilient entropy coder for space missions,” Journal of Applied Remote Sensing 4 (2010).Google Scholar
  25. 25.
    A. G. Villafranca, I. Mora, P. Ruiz-Rodríguez, J. Portell, and E. García-Berro, “Optimizing GPS data transmission using entropy coding compression”, Proc. SPIE 7810, 2010.Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2012

Authors and Affiliations

  • Jordi Portell
    • 1
    • 2
  • Alberto G. Villafranca
    • 2
    • 3
  • Enrique García-Berro
    • 2
    • 3
  1. 1.Departament d’Astronomia i Meteorologia/ICCUBUniversitat de BarcelonaBarcelonaSpain
  2. 2.Institut d’Estudis Espacials de CatalunyaBarcelonaSpain
  3. 3.Departament de Física AplicadaUniversitat Politècnica de CatalunyaCastelldefelsSpain

Personalised recommendations