Fast Retrieval of Weather Analogues in a Multi-petabytes Archive Using Wavelet-Based Fingerprints

  • Baudouin Raoult
  • Giuseppe Di Fatta
  • Florian Pappenberger
  • Bryan Lawrence
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10861)

Abstract

Very large climate data repositories provide a consistent view of weather conditions over long time periods. In some applications and studies, given a current weather pattern (e.g. today’s weather), it is useful to identify similar ones (weather analogues) in the past. Looking for similar patterns in an archive using a brute force approach requires data to be retrieved from the archive and then compared to the query, using a chosen similarity measure. Such operation would be very long and costly. In this work, a wavelet-based fingerprinting scheme is proposed to index all weather patterns from the archive. The scheme allows to answer queries by computing the fingerprint of the query pattern, then comparing them to the index of all fingerprints more efficiently, in order to then retrieve only the corresponding selected data from the archive. The experimental analysis is carried out on the ECMWF’s ERA-Interim reanalyses data representing the global state of the atmosphere over several decades. Results shows that 32 bits fingerprints are sufficient to represent meteorological fields over a 1700 km \({\times }\) 1700 km region and allow the quasi instantaneous retrieval of weather analogues.

Keywords

Climate data repositories Weather analogues Information retrieval 

References

  1. 1.
    Delle Monache, L., Eckel, F.A., Rife, D.L., Nagarajan, B., Searight, K.: Probabilistic Weather Prediction with an Analog Ensemble. Mon. Wea. Rev. 141(10), 3498–3516 (2013)CrossRefGoogle Scholar
  2. 2.
    Van den Dool, H.: A new look at weather forecasting through analogues. Mon. Weather Rev. 117(10), 2230–2247 (1989)CrossRefGoogle Scholar
  3. 3.
    Van den Dool, H.: Searching for analogues, how long must we wait? Tellus A 46(3), 314–324 (1993)CrossRefGoogle Scholar
  4. 4.
    Zorita, E., von Storch, H.: The analog method as a simple statistical downscaling technique: comparison with more complicated methods, pp. 1–16, August 1999Google Scholar
  5. 5.
    Evans, M., Murphy, R.: A historical-analog-based severe weather checklist for central New York and northeast Pennsylvania, pp. 1–8, February 2013Google Scholar
  6. 6.
    Sanderson, M.G., Hanlon, H.M., Palin, E.J., Quinn, A.D., Clark, R.T.: Analogues for the railway network of Great Britain. Meteorol. Appl. 23(4), 731–741 (2016)CrossRefGoogle Scholar
  7. 7.
    Jacobs, C.E., Finkelstein, A., Salesin, D.H.: Fast multiresolution image querying. In: Proceedings of the 22nd Annual Conference on Computer Graphics and Interactive Techniques, pp. 277–286. ACM (1995)Google Scholar
  8. 8.
    Baluja, S., Covell, M.: Waveprint: efficient wavelet-based audio fingerprinting. Pattern Recogn. 41(11), 3467–3480 (2008)CrossRefGoogle Scholar
  9. 9.
    Orio, N.: Music Retrieval: A Tutorial and Review. Now Publishers Inc., Boston (2006)MATHGoogle Scholar
  10. 10.
    Veltkamp, R., Burkhardt, H., Kriegel, H.P.: State-of-the-Art in Content-Based Image and Video Retrieval. Springer Science & Business Media, Dordrecht (2013).  https://doi.org/10.1007/978-94-015-9664-0CrossRefMATHGoogle Scholar
  11. 11.
    Daubechies, I.: Orthonormal bases of compactly supported wavelets. Commun. Pure Appl. Math. 41(7), 909–996 (1988)MathSciNetCrossRefGoogle Scholar
  12. 12.
    Walker, J.S.: A primer on wavelets and their scientific applications, pp. 1–156, June 2005Google Scholar
  13. 13.
    Stollnitz, E.J., DeRose, T.D., Salesin, D.H.: Wavelets for computer graphics: a primer part 1, pp. 1–8 (1995)Google Scholar
  14. 14.
    Stollnitz, E.J., DeRose, T.D., Salesin, D.H.: Wavelets for computer graphics: a primer part 2, pp. 1–9 (1995)Google Scholar
  15. 15.
    Stollnitz, E.J., DeRose, T., Salesin, D.H.: Wavelets for Computer Graphics - Theory and Applications. Morgan Kaufmann, San Francisco (1996)Google Scholar
  16. 16.
    Balan, V., Condea, C.: Wavelets and Image Compression. Telecommunication Standardization Sector of lTU, Leden (2003)Google Scholar
  17. 17.
    Porwik, P., Lisowska, A.: The Haar-wavelet transform in digital image processing: its status and achievements. Mach. Graph. Vision 13(1/2), 79–98 (2004)MATHGoogle Scholar
  18. 18.
    Shapiro, J.M.: Embedded image coding using zerotrees of wavelet coefficients. IEEE Trans. Signal Process. 41(12), 3445–3462 (1993)CrossRefGoogle Scholar
  19. 19.
    Walker, J.S., Nguyen, T.Q.: Wavelet-based image compression. In: Rao, K.R. et al.: The Transform and Data Compression Handbook. CRC Press LLC, Boca Raton (2001)Google Scholar
  20. 20.
    Zeng, L., Jansen, C., Unser, M., Hunziker, P.: Extension of wavelet compression algorithms to 3D and 4D image data: exploitation of data coherence in higher dimensions allows very high compression ratios, pp. 1–7, October 2011Google Scholar
  21. 21.
    Patrikalakis, N.M.: Wavelet based similarity measurement algorithm for seafloor morphology. Massachusetts Institute of Technology (2006)Google Scholar
  22. 22.
    Regentova, E., Latifi, S., Deng, S.: A wavelet-based technique for image similarity estimation. In: ITCC-00, pp. 207–212. IEEE (2000)Google Scholar
  23. 23.
    Pauly, O., Padoy, N., Poppert, H., Esposito, L., Navab, N.: Wavelet energy map: a robust support for multi-modal registration of medical images. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2009, pp. 2184–2191. IEEE (2009)Google Scholar
  24. 24.
    Traina, A.J.M., Castañón, C.A.B., Traina, Jr., C.: MultiWaveMed: a system for medical image retrieval through wavelets transformations. In: IEEE Computer Society, June 2003Google Scholar
  25. 25.
    Marsolo, K., Parthasarathy, S., Ramamohanarao, K.: Structure-based querying of proteins using wavelets. In: Proceedings of the 15th ACM International Conference on Information and Knowledge Management, pp. 24–33. ACM (2006)Google Scholar
  26. 26.
    Cattani, C., Ciancio, A.: Wavelet clustering in time series analysis. Balkan J. Geom. Appl. 10(2), 33 (2005)MathSciNetMATHGoogle Scholar
  27. 27.
    Kocaman, Ç., Özdemir, M.: Comparison of statistical methods and wavelet energy coefficients for determining two common PQ disturbances: sag and swell. In: International Conference on Electrical and Electronics Engineering, ELECO 2009, pp. I-80–I-84. IEEE (2009)Google Scholar
  28. 28.
    Phuc, N.H., Khanh, T.Q., Bon, N.N.: Discrete wavelets transform technique application in identification of power quality disturbances (2005)Google Scholar
  29. 29.
    Gomez-Glez, J.F.: Wavelet methods for time series analysis, pp. 1–45, February 2009Google Scholar
  30. 30.
    Popivanov, I., Miller, R.J.: Similarity search over time-series data using wavelets. In: 18th International Conference on Data Engineering, Proceedings, pp. 212–221. IEEE (2002)Google Scholar
  31. 31.
    Raoult, B.: Architecture of the new MARS server. In: Sixth Workshop on Meteorological Operational Systems, ECMWF, 17–21 November 1997, Shinfield Park, Reading, pp. 90–100 (1997)Google Scholar
  32. 32.
    Woods, A.: Archives and graphics: towards MARS, MAGICS and Metview. In: The European Approach, Medium-Range Weather Prediction, pp. 183–193 (2006)Google Scholar
  33. 33.
    Frauenfeld, O.W., Zhang, T., Serreze, M.C.: Climate change and variability using European Centre for Medium-Range Weather Forecasts reanalysis (ERA-40) temperatures on the Tibetan Plateau. J. Geophys. Res. Atmos. (1984–2012) 110(D2) (2005)Google Scholar
  34. 34.
    Santer, B.D., Wigley, T.M., Simmons, A.J., Kållberg, P.W., Kelly, G.A., Uppala, S.M., Ammann, C., Boyle, J.S., Brüggemann, W., Doutriaux, C.: Identification of anthropogenic climate change using a second-generation reanalysis. J. Geophys. Res. Atmos. (1984–2012) 109(D21) (2004)CrossRefGoogle Scholar
  35. 35.
    Dee, D., Uppala, S., Simmons, A., Berrisford, P., Poli, P., Kobayashi, S., Andrae, U., Balmaseda, M., Balsamo, G., Bauer, P.: The ERA-Interim reanalysis: configuration and performance of the data assimilation system. Q. J. Royal Meteorol. Soc. 137(656), 553–597 (2011)CrossRefGoogle Scholar
  36. 36.
    Dee, D., Balmaseda, M., Balsamo, G., Engelen, R., Simmons, A., Thépaut, J.N.: Toward a consistent reanalysis of the climate system. Bull. Am. Meteorol. Soc. 95(8), 1235–1248 (2014)CrossRefGoogle Scholar
  37. 37.
    Sixta, S.: Hamming cube and other stuff, pp. 1–18, May 2014Google Scholar
  38. 38.
    Indyk, P., Naor, A.: Nearest-neighbor-preserving embeddings. ACM Trans. Algorithms (TALG) 3(3), 31 (2007)MathSciNetCrossRefGoogle Scholar
  39. 39.
    Mo, R., Ye, C., Whitfield, P.H.: Application potential of four nontraditional similarity metrics in hydrometeorology. J. Hydrometeorology 15(5), 1862–1880 (2015)CrossRefGoogle Scholar
  40. 40.
    Van Der Walt, S., Colbert, S.C., Varoquaux, G.: The NumPy array: a structure for efficient numerical computation. Comput. Sci. Eng. 13(2), 22–30 (2011)CrossRefGoogle Scholar
  41. 41.
    Jones, E., Oliphant, T., Peterson, P.: SciPy: open source scientific tools for Python (2014)Google Scholar
  42. 42.
    Hunter, J.D.: Matplotlib: a 2D graphics environment. Comput. Sci. Eng. 9(3), 90–95 (2007)CrossRefGoogle Scholar
  43. 43.
    Wasilewski, F.: PyWavelets: discrete wavelet transform in python (2010)Google Scholar
  44. 44.
    Fucile, E., Codorean, C.: GRIB API. A database driven decoding library. In: Twelfth Workshop on Meteorological Operational Systems, ECMWF, 2–6 November 2009, Shinfield Park, Reading, pp. 46–47 (2009)Google Scholar
  45. 45.
    O’Sullivan, P.: MAGICS - the ECMWF graphics package. ECMWF Newslett. (62) (1993)Google Scholar
  46. 46.
    Pérez, F., Granger, B.E.: IPython: a system for interactive scientific computing. Comput. Sci. Eng. 9(3), 21–29 (2007)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Baudouin Raoult
    • 1
  • Giuseppe Di Fatta
    • 2
  • Florian Pappenberger
    • 1
  • Bryan Lawrence
    • 2
    • 3
    • 4
  1. 1.European Centre for Medium-Range Weather ForecastsReadingUK
  2. 2.Department of Computer ScienceUniversity of ReadingReadingUK
  3. 3.Department of MeteorologyUniversity of ReadingReadingUK
  4. 4.National Centre for Atmospheric ScienceReadingUK

Personalised recommendations