Skip to main content
SpringerLink
Log in

Big Sensor Data Acquisition and Archiving with Compression

  • Conference paper
  • First Online:
Big Data and Visual Analytics

  • 1981 Accesses

Abstract

Machine-generated data such as sensor data now comprise major portion of available information, which raises two important problems: efficient acquisition of sensor data and the storage of massive sensor data collection. These data sources generate so much data quickly that data compression is essential to reduce storage requirement or transmission capacity of devices. This work first discusses a low complexity sensing framework which enables to reduce computation and communication overheads of devices without much compromising the accuracy of sensor readings. Then a new class of compression algorithm based on statistical similarity is presented that can be effectively used in many applications where an order of data sequence could be relaxed. Next, a quality-adjustable sensor data archiving is discussed, which compresses an entire collection of sensor data efficiently without compromising key features. Considering data aging aspect, this archiving scheme is capable of decreasing data fidelity gradually to secure more storage space.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Institutional subscriptions

Notes

  1. 1.

    This can also be seen as inner product operations.

  2. 2.

    The two bases Φ and Ψ are (maximally) incoherent when the largest correlation between any two elements of Φ and Ψ is \(1/\sqrt {N}\) where N is the order of two square matrices.

  3. 3.

    This can be generated by a random permutation.

  4. 4.

    When the random indices for the spatial sampling are not explicitly synchronized between encoder and decoder, the spatio-temporal measurement no longer has a matrix form since the number of spatial sampling can vary between time instants. However, the decoding process does not impose the matrix form on the spatio-temporal measurement.

  5. 5.

    This value is also called the significance level.

  6. 6.

    These analytical models are convex by virtue of the trade-off relationship between data fidelity and compression ratio. The sum of these convex functions is also convex.

References

  1. 7-zip. http://www.7-zip.org

  2. Ahmed, N., Natarajan, T., Rao, K.R.: Discrete cosine transform. IEEE Trans. Comput. C-23(1), 90–93 (1974)

    Google Scholar 

  3. Bajwa, W., Haupt, J., Sayeed, A., Nowak, R.: Compressive wireless sensing. In: Proceedings of the International Conference on Information Processing in Sensor Networks (IPSN ’06), pp. 134–142 (2006)

    Google Scholar 

  4. Baraniuk, R.G.: Compressive sensing [lecture notes]. IEEE Signal Process. Mag. 24(4), 118–121 (2007)

    Google Scholar 

  5. Baraniuk, R.G., Cevher, V., Duarte, M.F., Hegde, C.: Model-based compressive sensing. IEEE Trans. Inf. Theory 56(4), 1982–2001 (2010)

    Google Scholar 

  6. Baron, D., Duarte, M.F., Wakin, M.B., Sarvotham, S., Baraniuk, R.G.: Distributed compressive sensing (2009). arXiv preprint arXiv:0901.3403

    Google Scholar 

  7. Barr, K.C., Asanović, K.: Energy-aware lossless data compression. ACM Trans. Comput. Syst. 24(3), 250–291 (2006)

    Google Scholar 

  8. Berinde, R., Indyk, P.: Sparse recovery using sparse random matrices. Tech. Rep. MIT-CSAIL-TR-2008–001, Massachusetts Institute of Technology (2008)

    Google Scholar 

  9. Burke, J.A., Estrin, D., Hansen, M., Parker, A., Ramanathan, N., Reddy, S., Srivastava, M.B.: Participatory sensing. Center for Embedded Network Sensing (2006)

    Google Scholar 

  10. bzip2. http://www.bzip.org

  11. Candès, E.J., Wakin, M.B.: An introduction to compressive advanced sampling. IEEE Signal Process. Mag. 25(2), 21–30 (2008)

    Google Scholar 

  12. Cao, H., Wolfson, O., Trajcevski, G.: Spatio-temporal data reduction with deterministic error bounds. VLDB J. 15(3), 211–228 (2006)

    Google Scholar 

  13. Choi, J., Hu, K., Sim, A.: Relational dynamic Bayesian networks with locally exchangeable measures. Tech. Rep. LBNL-6341E, Lawrence Berkeley National Laboratory (2013)

    Google Scholar 

  14. Coding of audiovisual objects - Part 10: advanced video coding (2003)

    Google Scholar 

  15. Cohen, E., Kaplan, H.: Aging through cascaded caches: performance issues in the distribution of web content. In: Proceedings of the 2001 ACM Conference on Special Interest Group on Data Communication, pp. 41–53 (2001)

    Google Scholar 

  16. Cover, T.M., Thomas, J.A.: Elements of Information Theory, 2nd edn. Wiley, Hoboken (2006)

    Google Scholar 

  17. Do, T.T., Gan, L., Nguyen, N.H., Tran, T.D.: Fast and efficient compressive sensing using structurally random matrices. IEEE Trans. Signal Process. 60(1), 139–154 (2012)

    Google Scholar 

  18. Duarte, M.F., Wakin, M.B., Baraniuk, R.G.: Fast reconstruction of piecewise smooth signals from incoherent projections. In: Proceedings of the Workshop Signal Processing with Adaptive Sparse Structured Representations (SPARS ’05) (2005)

    Google Scholar 

  19. Duarte, M.F., Wakin, M.B., Baron, D., Baraniuk, R.G.: Universal distributed sensing via random projections. In: Proceedings of the International Conference on Information Processing in Sensor Networks (IPSN ’06), pp. 177–185 (2006)

    Google Scholar 

  20. Engmann, S., Cousineau, D.: Comparing distributions: the two-sample Anderson-Darling test as an alternative to the Kolmogorov-Smirnoff test. J. Appl. Quant. Methods 6(3), 1–17 (2011)

    Google Scholar 

  21. Esmaeilzadeh, H., Sampson, A., Ceze, L., Burger, D.: Architecture support for disciplined approximate programming. In: Proceedings of the International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS ’12), pp. 301–312 (2012)

    Google Scholar 

  22. Foucart, S., Rauhut, H.: A Mathematical Introduction to Compressive Sensing. Springer, New York (2013)

    Google Scholar 

  23. Ganesan, D., Estrin, D., Heidemann, J.: Dimensions: why do we need a new data handling architecture for sensor networks? SIGCOMM Comput. Commun. Rev. 33(1), 143–148 (2003)

    Google Scholar 

  24. Ganesan, D., Greenstein, B., Estrin, D., Heidemann, J., Govindan, R.: Multiresolution storage and search in sensor networks. Trans. Storage 1(3), 277–315 (2005)

    Google Scholar 

  25. Gantz, J.F., Chute, C., Manfrediz, A., Minton, S., Reinsel, D., Schlichting, W., Toncheva, A.: The diverse and exploding digital universe: an updated forecast of worldwide information growth through 2011. White Paper (2008)

    Google Scholar 

  26. Hang, H.M., Chen, J.J.: Source model for transform video coder and its application. I. Fundamental theory. IEEE Trans. Circuits Syst. Video Technol. 7(2), 287–298 (1997)

    Article  Google Scholar 

  27. Hilbert, M., López, P.: The world’s technological capacity to store, communicate, and compute information. Science 332(6025), 60–65 (2011)

    Article  Google Scholar 

  28. IDEALEM. http://datagrid.lbl.gov/idealem

  29. Lee, D., Choi, J.: Low complexity sensing for big spatio-temporal data. In: Proceedings of the International Conference on Big Data (BigData ’14), pp. 323–328 (2014)

    Google Scholar 

  30. Lee, D., Choi, J.: Learning compressive sensing models for big spatio-temporal data. In: Proceedings of the International Conference on Data Mining (SDM ’15), pp. 667–675 (2015)

    Google Scholar 

  31. Lee, D., Lee, Y., Lee, H., Lee, J., Shin, H.: Determining efficient bit stream extraction paths in H.264/AVC scalable video coding. In: Proceedings of the Asilomar Conference on Signals, on Systems, and Computers (ACSSC ’08), pp. 2233–2237 (2008)

    Google Scholar 

  32. Lee, D., Choi, J., Shin, H.: Low-complexity compressive sensing with downsampling. IEICE Electron. Express 11(3), 20130947 (2014)

    Article  Google Scholar 

  33. Lee, D., Choi, J., Shin, H.: A scalable and flexible repository for big sensor data. IEEE Sensors J. 15(12), 7284–7294 (2015)

    Article  Google Scholar 

  34. Lee, D., Ryu, J., Shin, H.: Scalable management of storage for massive quality-adjustable sensor data. Computing 97(8), 769–793 (2015)

    Article  MathSciNet  Google Scholar 

  35. Lee, D., Lima, R., Choi, J.: Improving imprecise compressive sensing models. In: Proceedings of the Thirty-Second Conference on Uncertainty in Artificial Intelligence (UAI ’16), pp. 397–406 (2016)

    Google Scholar 

  36. Lee, D., Sim, A., Choi, J., Wu, K.: Novel data reduction based on statistical similarity. In: Proceedings of the International Conference on Scientific and Statistical Database Management (SSDBM ’16), pp. 21:1–21:12 (2016)

    Google Scholar 

  37. Luo, C.,Wu, F., Sun, J., Chen, C.W.: Compressive data gathering for large-scale wireless sensor networks. In: Proceedings of the Mobile Computing and Networking (MobiCom ’09), pp. 145–156 (2009)

    Google Scholar 

  38. Marcelloni, F., Vecchio, M.: Enabling energy-efficient and lossy-aware data compression in wireless sensor networks by multi-objective evolutionary optimization. Inf. Sci. 180(10), 1924–1941 (2010)

    Article  Google Scholar 

  39. Massey, F.J. Jr.: The Kolmogorov-Smirnov test for goodness of fit. J. Am. Stat. Assoc. 46(253), 68–78 (1951)

    Article  MATH  Google Scholar 

  40. Moffat, A.: Implementing the PPM data compression scheme. IEEE Trans. Commun. 38(11), 1917–1921 (1990)

    Article  Google Scholar 

  41. Noh, D., Lee, D., Shin, H.: Mission-oriented selective routing for wireless sensor networks. In: Proceedings of the International Conference on Communications and Networking in China (CHINACOM ’07), pp. 809–813 (2007)

    Google Scholar 

  42. Noh, D., Lee, D., Shin, H.: QoS-aware geographic routing for solar-powered wireless sensor networks. IEICE Trans. Commun. 90(12), 3373–3382 (2007)

    Article  Google Scholar 

  43. Palmer, M.: Seven principles of effective RFID data management (2005)

    Google Scholar 

  44. Palpanas, T., Vlachos, M., Keogh, E., Gunopulos, D., Truppel, W.: Online amnesic approximation of streaming time series. In: Proceedings of the International Conference on Data Engineering (ICDE ’04), pp. 339–349 (2004)

    Google Scholar 

  45. Quer, G., Masiero, R., Munaretto, D., Rossi, M., Widmer, J., Zorzi, M.: On the interplay between routing and signal representation for compressive sensing in wireless sensor networks. In: Proceedings of the Information Theory and Applications Workshop (ITA ’09), pp. 206–215 (2009)

    Google Scholar 

  46. Quer, G., Masiero, R., Pillonetto, G., Rossi, M., Zorzi, M.: Sensing, compression, and recovery for WSNs: sparse signal modeling and monitoring framework. IEEE Trans. Wireless Commun. 11(10), 3447–3461 (2012)

    Article  Google Scholar 

  47. Quinsac, C., Basarab, A., Girault, J.M., Kouamé, D.: Compressed sensing of ultrasound images: sampling of spatial and frequency domains. In: Proceedings of the International Workshop on Signal Processing Systems (SiPS ’10), pp. 231–236 (2010)

    Google Scholar 

  48. Richardson, I.E.: The H.264 Advanced Video Compression Standard, 2nd edn. Wiley, Hoboken (2010)

    Google Scholar 

  49. Sadler, C.M., Martonosi, M.: Data compression algorithms for energy-constrained devices in delay tolerant networks. In: Proceedings of the International Conference on Embedded Network Sensor Systems (SenSys ’06), pp. 265–278 (2006)

    Google Scholar 

  50. Sampson, A., Nelson, J., Strauss, K., Ceze, L.: Approximate storage in solid-state memories. In: Proceedings of the International Symposium on Microarchitecture (MICRO ’46), pp. 25–36 (2013)

    Google Scholar 

  51. Sayood, K.: Introduction to Data Compression, 4th edn. Morgan Kaufmann, Burlington (2012)

    MATH  Google Scholar 

  52. Schwarz, H., Marpe, D., Wiegand, T.: Overview of the scalable video coding extension of the H.264/AVC standard. IEEE Trans. Circuits Syst. Video Technol. 17(9), 1103–1120 (2007)

    Google Scholar 

  53. Seabra, J., Sanches, J.: Modeling log-compressed ultrasound images for radio frequency signal recovery. In: Proceedings of the International Conference on Engineering in Medicine and Biology Society (EMBC ’08), pp. 426–429 (2008)

    Google Scholar 

  54. Sensorscope: Sensor networks for environmental monitoring. http://lcav.epfl.ch/op/edit/sensorscope-en

  55. Srisooksai, T., Keamarungsi, K., Lamsrichan, P., Araki, K.: Practical data compression in wireless sensor networks: a survey. J. Netw. Comput. Appl. 35(1), 37–59 (2012)

    Article  Google Scholar 

  56. The gzip home page. http://www.gzip.org

  57. Vuran, M.C., Akan, Ö.B., Akyildiz, I.F.: Spatio-temporal correlation: theory and applications for wireless sensor networks. Comput. Netw. 45(3), 245–259 (2004)

    Article  MATH  Google Scholar 

  58. Wang, Y.C., Hsieh, Y.Y., Tseng, Y.C.: Multiresolution spatial and temporal coding in a wireless sensor network for long-term monitoring applications. IEEE Trans. Comput. 58(6), 827–838 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  59. Wasserstein, R.L., Lazar, N.A.: The ASA’s statement on p-values: context, process, and purpose. Am. Stat. 70(2), 129–133 (2016). doi:10.1080/00031305.2016.1154108

    Article  MathSciNet  Google Scholar 

  60. Wien, M., Cazoulat, R., Graffunder, A., Hutter, A., Amon, P.: Real-time system for adaptive video streaming based on SVC. IEEE Trans. Circuits Syst. Video Technol. 17(9), 1227–1237 (2007)

    Article  Google Scholar 

  61. Xiong, Z., Ramchandran, K., Orchard, M.T., Zhang, Y.Q.: A comparative study of DCT-and wavelet-based image coding. IEEE Trans. Circuits Syst. Video Technol. 9(5), 692–695 (1999)

    Article  Google Scholar 

  62. Yu, J., Ongarello, S., Fiedler, R., Chen, X., Toffolo, G., Cobelli, C., Trajanoski, Z.: Ovarian cancer identification based on dimensionality reduction for high-throughput mass spectrometry data. Bioinformatics 21(10), 2200–2209 (2005)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Texas A&M University-Commerce, Commerce, TX, 75428, USA

    Dongeun Lee

Authors
  1. Dongeun Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dongeun Lee .

Editor information

Editors and Affiliations

  1. Department of Computer Science, Texas A&M University-Commerce, Commerce, Texas, USA

    Sang C. Suh

  2. Department of Electrical and Computer Engineering, The University of Alabama at Birmingham, Birmingham, Alabama, USA

    Thomas Anthony

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Lee, D. (2017). Big Sensor Data Acquisition and Archiving with Compression. In: Suh, S., Anthony, T. (eds) Big Data and Visual Analytics. Springer, Cham. https://doi.org/10.1007/978-3-319-63917-8_7

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-63917-8_7

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-63915-4

  • Online ISBN: 978-3-319-63917-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation