Skip to main content
SpringerLink
Log in

Turbulence in the Era of Big Data: Recent Experiences with Sharing Large Datasets

  • Chapter
  • First Online:
Whither Turbulence and Big Data in the 21st Century?

Abstract

In the context of the contemporary push for “big data” in many fields, we review recent experiences building large databases for turbulence research. We consider data from direct numerical simulations (DNS) of various canonical flows and from experimental studies and related numerical simulations of wall-bounded turbulence, where the data storage needs are particularly challenging due to the very large range of length and time scales that exists in these flows at high Reynolds numbers. The focus is on a move from the traditional approach of data-handling and analysis where datasets are moved to individual computers, to one where much of the analysis is moved to the hosting system that stores these data. In this context we give a summary of a unique open numerical laboratory that archives over 200 Terabytes of DNS data, including full spatio-temporal flow fields of various canonical flows. Particular attention is given to the unique access requirements for large datasets to become open to the research community and the success the system has had in democratizing access to large datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 169.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. A. Hey, S. Tansley, K. Tolle, et al., The Fourth Paradigm: Data-intensive scientific discovery, Microsoft Research Redmond, WA, 2009

    Google Scholar 

  2. D. Donzis, P. Yeung, K. Sreenivasan, Dissipation and enstrophy in isotropic turbulence: resolution effects and scaling in direct numerical simulations. Phys. Fluids 20, 045108 (2008)

    Article  MATH  Google Scholar 

  3. T. Ishihara, T. Gotoh, Y. Kaneda, Study of high-Reynolds number isotropic turbulence by direct numerical simulation. Annu. Rev. Fluid Mech. 41, 165–180 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  4. X. Wu, P. Moin, Direct numerical simulation of turbulence in a nominally zero-pressure-gradient flat-plate boundary layer. J. Fluid Mech. 630, 5–41 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  5. Y. Khalighi, J. Nichols, S. Lele, F. Ham, P. Moin, Unstructured large eddy simulation for prediction of noise issued from turbulent jets in various configurations, AIAA paper 2886, 2011

    Google Scholar 

  6. M. Lee, R. Moser, Direct numerical simulation of turbulent channel flow up to Re τ = 5200. J. Fluid Mech. 774, 395–415 (2015)

    Article  Google Scholar 

  7. A. Smits, I. Marusic, Wall-bounded turbulence. Phys. Today 66, 25–30 (2013)

    Article  Google Scholar 

  8. I. Marusic, G.V. Candler, V. Interrante, P.K. Subbareddy, A. Moss, Real time feature extraction for the analysis of turbulent flows, in Data Mining for Scientific and Engineering Applications, ed. by R. Grossman, C. Kamath, P. Kegelmeyer, V. Kumar, R. Namburu (Springer, US, 2001), pp. 223–238

    Chapter  Google Scholar 

  9. R.J. Adrian, J. Westerweel, Particle Image Velocimetry (Cambridge University Press, Cambridge, 2011)

    MATH  Google Scholar 

  10. A. Schröder, R. Geisler, K. Staack, G. Elsinga, F. Scarano, B. Wieneke, A. Henning, C. Poelma, J. Westerweel, Eulerian and Lagrangian views of a turbulent boundary layer flow using time-resolved tomographic PIV. Exp. Fluids 50, 1071–1091 (2011)

    Article  Google Scholar 

  11. C.M. de Silva, E. Gnanamanickam, C. Atkinson, N.A. Buchmann, N. Hutchins, J. Soria, I. Marusic, High spatial range velocity measurements in a high Reynolds number turbulent boundary layer. Phys. Fluids 26, 025117 (2014)

    Article  Google Scholar 

  12. A.S. Szalay, P. Kunszt, A. Thakar, J. Gray, D. Slutz, R.J. Brunner, Designing and mining multi-terabyte astronomy archives: the Sloan digital sky survey, in Proceedings of the 2000 ACM SIGMOD International Conference on Management of Data (2000), pp. 451–462

    Google Scholar 

  13. W. O’Mullane, J. Gray, N. Li, T. Budavari, M. Nieto-Santisteban, A. Szalay, Batch query system with interactive local storage for SDSS and the VO. in Proceedings of the ADASS XIII, ASP Conference Series, ed. by F. Ochsenbein, M. Allen, D. Egret (2004), pp. 314–372

    Google Scholar 

  14. G. Lemson, the Virgo Consortium, Halo and galaxy formation histories from the millennium simulation: public release of a VO-oriented and SQL-queryable database for studying the evolution of galaxies in the lambdaCDM cosmogony. arXiv:astro-ph/0608019 (2006)

    Google Scholar 

  15. R. Overzier, G. Lemson, R. Angulo, et al., The millennium run observatory: first light. MNRAS 428, 778 (2013)

    Article  Google Scholar 

  16. R. Burns, J. Vogelstein, A. Szalay, From cosmos to connectomes: the evolution of data-intensive science. Neuron 83, 1249–1252 (2014)

    Article  Google Scholar 

  17. Y. Li, E. Perlman, M. Wan, Y. Yang, C. Meneveau, R. Burns, S. Chen, A. Szalay, G. Eyink, A public turbulence database cluster and applications to study Lagrangian evolution of velocity increments in turbulence. J. Turbul. 9, N31 (2008)

    Article  MATH  Google Scholar 

  18. K. Kanov, C. Burns, R. Lalescu, G. Eyink, The Johns Hopkins turbulence databases: an open simulation laboratory for turbulence research. Comput. Sci. Eng. 17 (5), 10–17 (2015)

    Article  Google Scholar 

  19. Y. Gu, R. Grossman, A. Szalay, A. Thakar, Distributing the Sloan Digital Sky Survey using UDT and sector, in e-Science (2006), p. 56

    Google Scholar 

  20. H. Samet, Foundations of Multidimensional and Metric Data Structures (Morgan Kaufmann Publishers, Los Altos, CA, 2006)

    MATH  Google Scholar 

  21. E. Perlman, R. Burns, Y. Li, C. Meneveau, Data exploration of turbulence simulations using a database cluster, in SC07 07 (2007)

    Google Scholar 

  22. J. Graham, K. Kanov, E. Givelberg, R. Burns, G. Eyink, A. Szalay, C. Meneveau, M. Lee, N. Malaya, R. Moser, A web-services accessible database for channel flow turbulence at Re-tau = 1000. Bull. Am. Phys. Soc. 58, 236 (2013)

    Google Scholar 

  23. M. Lee, N. Malaya, R. Moser, Petascale direct numerical simulation of turbulent channel flow on up to 786k cores, in Proceedings of SC13: International Conference for High Performance Computing, Networking, Storage and Analysis (Association for Computing Machinery, New York, 2013), p. 61

    Google Scholar 

  24. J. Graham, K. Kanov, X. Yang, M. Lee, N. Malaya, C. Lalescu, R. Burns, G. Eyink, A. Szalay, R. Moser, C. Meneveau, A web services-accessible database of turbulent channel flow and its use for testing a new integral wall model for LES. J. Turbul. 17 (2), 181–215 (2016)

    Article  Google Scholar 

  25. D. Livescu, J. Ristorcelli, Variable-density mixing in buoyancy-driven turbulence. J. Fluid Mech. 605, 145–180 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  26. D. Livescu, Numerical simulations of two-fluid turbulent mixing at large density ratios and applications to the Rayleigh–Taylor instability. Philos. Trans. R. Soc. A: Math. Phys. Eng. Sci. 371, 20120185 (2013)

    Article  MathSciNet  Google Scholar 

  27. K. Kanov, R. Burns, G. Eyink, C. Meneveau, A. Szalay, Data-intensive spatial filtering in large numerical simulation datasets, in International Conference for High Performance Computing, Networking, Storage and Analysis (SC) (IEEE Computer Society Press Los Alamitos, CA, 2012)

    Google Scholar 

  28. K. Kanov, R. Burns, C. Lalescu, Efficient evaluation of threshold queries of derived fields in a numerical simulation database, in Proceedings of the 18th International Conference on Extending Database Technology (EDBT), 2015

    Google Scholar 

  29. G. Eyink, E. Vishniac, C. Lalescu, H. Aluie, K. Kanov, K. Bürger, R. Burns, C. Meneveau, A. Szalay, Flux-freezing breakdown in high-conductivity magnetohydrodynamic turbulence. Nature 497, 466–469 (2013)

    Article  Google Scholar 

  30. F. Alexander, C. Meneveau, Foreword to the “open simulation laboratories” special issue. Comput. Sci. Eng. 17 (5), 7–9 (2015)

    Article  Google Scholar 

  31. K. Kanov, Efficient evaluation of data-intensive batch-queries in open simulation laboratories, Ph.D. thesis, Johns Hopkins University, 2015

    Google Scholar 

  32. P. Johnson, C. Meneveau, Large-deviation joint statistics of the finite-time Lyapunov spectrum in isotropic turbulence. Phys. Fluids 27 (8), 085110 (2015)

    Google Scholar 

  33. G. Eyink, Stochastic flux freezing and magnetic dynamo. Phys. Rev. E 83, 056405 (2011)

    Article  Google Scholar 

  34. H. Yu, C. Meneveau, Lagrangian refined Kolmogorov similarity hypothesis for gradient time evolution and correlation in turbulent flows. Phys. Rev. Lett. 104, 084502 (2010)

    Article  Google Scholar 

  35. H. Yu, C. Meneveau, Scaling of conditional Lagrangian time correlation functions of velocity and pressure gradient magnitudes in isotropic turbulence. Flow Turbul. Combust. 85, 457–472 (2010)

    Article  MATH  Google Scholar 

  36. Y. Li, L. Chevillard, C. Meneveau, G. Eyink, Matrix exponential-based closures for the turbulent subgrid-scale stress tensor. Phys. Rev. E 79, 016305 (2009)

    Article  MathSciNet  Google Scholar 

  37. K. Kanov, E. Perlman, R. Burns, Y. Ahmad, A. Szalay, I/O streaming evaluation of batch queries for data-intensive computational turbulence, in Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis (Association for Computing Machinery, New York, 2011), p. 29

    Google Scholar 

  38. X. Wang, E. Perlman, R. Burns, T. Malik, T. Budavári, C. Meneveau, A. Szalay, Jaws: job-aware workload scheduling for the exploration of turbulence simulations, in Proceedings of the 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis (IEEE Computer Society, Washington DC, 2010), pp. 1–11

    Google Scholar 

  39. H. Yu, K. Kanov, E. Perlman, J. Graham, E. Frederix, R. Burns, A. Szalay, G. Eyink, C. Meneveau, Studying Lagrangian dynamics of turbulence using on-demand fluid particle tracking in a public turbulence database. J. Turbul. 13, N12 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  40. M. Treib, K. Burger, F. Reichl, C. Meneveau, A. Szalay, R. Westermann, Turbulence visualization at the Terascale on desktop PCs. IEEE Trans. Vis. Comput. Graph. 18, 2169–2177 (2012)

    Article  Google Scholar 

  41. G. Eyink, D. Benveniste, Diffusion approximation in turbulent two-particle dispersion. Phys. Rev. E 88, 041001 (2013)

    Article  Google Scholar 

  42. J. Graham, E. Givelberg, K. Kanov, Run-time creation of the turbulent channel flow database by an HPC simulation using MPI-DB, in Proceedings of the 20th European MPI Users’ Group Meeting (Association for Computing Machinery, New York, 2013), pp. 151–156

    Google Scholar 

  43. D. Benveniste, T. Drivas, Asymptotic results for backwards two-particle dispersion in a turbulent flow. Phys. Rev. E 89, 041003 (2014)

    Article  Google Scholar 

  44. C. Meneveau, Lagrangian dynamics and models of the velocity gradient tensor in turbulent flows. Annu. Rev. Fluid Mech. 43, 219–245 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  45. C. Meneveau, Germano identity-based subgrid-scale modeling: a brief survey of variations on a fertile theme. Phys. Fluids 24, 121301 (2012)

    Article  Google Scholar 

  46. J. Boschung, P. Schaefer, N. Peters, C. Meneveau, The local topology of stream-and vortex lines in turbulent flows. Phys. Fluids 26, 045107 (2014)

    Article  Google Scholar 

  47. Y. Ahmad, R. Burns, M. Kazhdan, C. Meneveau, A. Szalay, A. Terzis, Scientific data management at the Johns Hopkins Institute for Data Intensive Engineering and Science. ACM SIGMOD Rec. 39, 18–23 (2011)

    Article  Google Scholar 

  48. H. Xu, A. Pumir, G. Falkovich, E. Bodenschatz, M. Shats, H. Xia, N. Francois, G. Boffetta, Flight–crash events in turbulence. Proc. Natl. Acad. Sci. 111, 7558–7563 (2014)

    Article  Google Scholar 

  49. J. Jucha, H. Xu, A. Pumir, E. Bodenschatz, Time-reversal-symmetry breaking in turbulence. Phys. Rev. Lett. 113, 054501 (2014)

    Article  Google Scholar 

  50. K. Gustavsson, J. Einarsson, B. Mehlig, Tumbling of small axisymmetric particles in random and turbulent flows. Phys. Rev. Lett. 112, 014501 (2014)

    Article  Google Scholar 

  51. N. Buchmann, C. Willert, J. Soria, Pulsed, high-power LED illumination for tomographic particle image velocimetry. Exp. Fluids 53, 1545–1560 (2012)

    Article  Google Scholar 

  52. X. Liu, J. Katz, Vortex-corner interactions in a cavity shear layer elucidated by time-resolved measurements of the pressure field. J. Fluid Mech. 728, 417–457 (2013)

    Article  MATH  Google Scholar 

  53. D. Xu, J. Chen, Accurate estimate of turbulent dissipation rate using PIV data. Exp. Therm. Fluid Sci. 44, 662–672 (2013)

    Article  Google Scholar 

  54. D. Fiscaletti, J. Westerweel, G. Elsinga, Long-range μPIV to resolve the small scales in a jet at high Reynolds number. Exp. Fluids 55, 1–15 (2014)

    Article  Google Scholar 

  55. J.M. Lawson, J.R. Dawson, On velocity gradient dynamics and turbulent structure. J. Fluid Mech. 780, 60–98 (2015)

    Article  MATH  Google Scholar 

  56. B. Luethi, M. Holzer, A. Tsinober, Expanding the Q–R space to three dimensions. J. Fluid Mech. 641, 497–507 (2010)

    Article  Google Scholar 

  57. J. Cardesa, D. Mistry, L. Gan, J. Dawson, Invariants of the reduced velocity gradient tensor in turbulent flows. J. Fluid Mech. 716, 597–615 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  58. H. Lu, Assessment of the modulated gradient model in decaying isotropic turbulence. Theor. Appl. Mech. Lett. 1, 041004 (2011)

    Article  Google Scholar 

  59. A.G. Gungor, S. Menon, A new two-scale model for large eddy simulation of wall-bounded flows. Prog. Aerospace Sci. 46, 28–45 (2010)

    Article  Google Scholar 

  60. F. Grinstein, A. Gowardhan, A. Wachtor, Simulations of Richtmyer–Meshkov instabilities in planar shock-tube experiments. Phys. Fluids 23, 034106 (2011)

    Article  Google Scholar 

  61. W. Liu, E. Ribeiro, Scale and rotation invariant detection of singular patterns in vector flow fields, in Structural, Syntactic, and Statistical Pattern Recognition (Springer, Berlin, Heidelberg, 2010), pp. 522–531

    Book  Google Scholar 

  62. P. Bhat, K. Subramanian, Fluctuation dynamos and their Faraday rotation signatures. Month. Not. R. Astron. Soc. 429, 2469–2481 (2013)

    Article  Google Scholar 

  63. C. Keylock, T. Tokyay, G. Constantinescu, A method for characterising the sensitivity of turbulent flow fields to the structure of inlet turbulence. J. Turbul. 12, N45 (2011)

    Article  Google Scholar 

  64. M. Holzner, M. Guala, B. Lüthi, A. Liberzon, N. Nikitin, W. Kinzelbach, A. Tsinober, Viscous tilting and production of vorticity in homogeneous turbulence. Phys. Fluids 22, 061701 (2010)

    Article  MATH  Google Scholar 

  65. C. Wu, T. Chang, Rank-ordered multifractal analysis (ROMA) of probability distributions in fluid turbulence. Nonlinear Processes Geophys. 18, 261–268 (2011)

    Article  MathSciNet  Google Scholar 

  66. C. Keylock, K. Nishimura, J. Peinke, A classification scheme for turbulence based on the velocity-intermittency structure with an application to near-wall flow and with implications for bed load transport. J. Geophys. Res.: Earth Surf. (2003–2012) 117, F1 (2012)

    Google Scholar 

  67. W. Liu, E. Ribeiro, Detecting singular patterns in 2D vector fields using weighted Laurent polynomial. Pattern Recognit. 45, 3912–3925 (2012)

    Article  Google Scholar 

  68. M. Mishra, X. Liu, M. Skote, C.-W. Fu, Kolmogorov spectrum consistent optimization for multi-scale flow decomposition. Phys. Fluids 26, 055106 (2014)

    Article  Google Scholar 

  69. L. Moriconi, R. Pereira, Vorticity statistics and the time scales of turbulent strain. Phys. Rev. E 88, 013005 (2013)

    Article  Google Scholar 

  70. F. Grinstein, A. Gowardhan, J. Ristorcelli, A. Wachtor, On coarse-grained simulations of turbulent material mixing. Phys. Scripta 86, 058203 (2012)

    Article  MATH  Google Scholar 

  71. T. Chang, C. Wu, M. Echim, H. Lamy, M. Vogelsberger, L. Hernquist, D. Sijacki, Complexity phenomena and ROMA of the earth’s magnetospheric cusp, hydrodynamic turbulence, and the cosmic web. Pure Appl. Geophys. 172 (7), 2025–2043 (2015)

    Article  Google Scholar 

  72. A. Pumir, H. Xu, G. Boffetta, G. Falkovich, E. Bodenschatz, Redistribution of kinetic energy in turbulent flows. Phys. Rev. X 4, 041006 (2014)

    Google Scholar 

Download references

Acknowledgements

The authors congratulate Prof. William George, for whom this Festschrift is intended, on occasion of his 70th birthday and salute him for his many accomplishments in turbulence research. CM gratefully acknowledges the large collaborative and interdisciplinary effort that has made the JHTDB open numerical turbulence laboratory possible, specifically the following colleagues, scholars, students, and staff: A. Szalay, R. Burns, G. Eyink, S. Chen, E. Vishniac, T. Zaki, E. Perlman, Y. Li, T. Budavari, M. Wan, H. Yu, E. Frederix, K. Buerger, H. Aluie, J. Graham, C. Lalescu, K. Kanov, S. Hamilton, P. Johnson, D. Livescu, R.D. Moser, M. Lee and N. Malaya, K. Yang, J. Lee, J. Vandenberg, S. Werner, V. Paul, and G. Lemson. The funding from the National Science Foundation (grants CMMI-0941530 and CBET-1507469) is gratefully acknowledged. IM thanks the support of the Australian Research Council.

Author information

Authors and Affiliations

  1. Department of Mechanical Engineering, and Institute for Data Intensive Engineering and Science (IDIES), The Johns Hopkins University, 3400 N. Charles Street, Baltimore, MD, 21218, USA

    Charles Meneveau

  2. Department of Mechanical Engineering, University of Melbourne, Melbourne, VIC, 3010, Australia

    Ivan Marusic

Authors
  1. Charles Meneveau
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ivan Marusic
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Charles Meneveau .

Editor information

Editors and Affiliations

  1. Department of Mechanical and Materials Engineering, Queen’s University, Kingston, Ontario, Canada

    Andrew Pollard

  2. Department of Mechanical Engineering, Texas Tech University, Lubbock, Texas, USA

    Luciano Castillo

  3. Department of Mechanical Engineering, University of Rouen, Mont-Saint-Aignan, France

    Luminita Danaila

  4. College of Engineering and Computer Science, Syracuse University, Syracuse, New York, USA

    Mark Glauser

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing Switzerland

About this chapter

Cite this chapter

Meneveau, C., Marusic, I. (2017). Turbulence in the Era of Big Data: Recent Experiences with Sharing Large Datasets. In: Pollard, A., Castillo, L., Danaila, L., Glauser, M. (eds) Whither Turbulence and Big Data in the 21st Century?. Springer, Cham. https://doi.org/10.1007/978-3-319-41217-7_27

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-41217-7_27

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-41215-3

  • Online ISBN: 978-3-319-41217-7

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation