Abstract
The explosive growth in supercomputers capacity has changed simulation paradigms. Simulations have shifted from a few lengthy ones to an ensemble of multiple simulations with varying initial conditions or input parameters. Thus, an ensemble consists of large volumes of multi-dimensional data that could go beyond the exascale boundaries. However, the disparity in growth rates between storage capabilities and computing resources results in I/O bottlenecks. This makes it impractical to utilize conventional post-processing and visualization tools for analyzing such massive simulation ensembles. In situ visualization approaches alleviate I/O constraints by saving predetermined visualizations in image databases during simulation. Nevertheless, the unavailability of output raw data restricts the flexibility of post hoc exploration of in situ approaches. Much research has been conducted to mitigate this limitation, but it falls short when it comes to simultaneously exploring and analyzing parameter and ensemble spaces. In this paper, we propose an expert-in-the-loop visual exploration analytic approach. The proposed approach leverages: feature extraction, deep learning, and human expert–AI collaboration techniques to explore and analyze image-based ensembles. Our approach utilizes local features and deep learning techniques to learn the image features of ensemble members. The extracted features are then combined with simulation input parameters and fed to the visualization pipeline for in-depth exploration and analysis using human expert + AI interaction techniques. We show the effectiveness of our approach using several scientific simulation ensembles.
Graphical abstract
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Ahrens J, Jourdain S, O’Leary P, et al (2014) An image-based approach to extreme scale in situ visualization and analysis. In: Proceedings of the international conference for high performance computing, networking, storage and analysis. IEEE Press, Piscataway, NJ, USA, SC ’14, pp 424–434, https://doi.org/10.1109/SC.2014.40
Bauer AC, Abbasi H, Ahrens J, et al (2016) In situ methods, infrastructures, and applications on high performance computing platforms. In: Computer Graphics Forum, Wiley Online Library, pp 577–597. https://doi.org/10.1111/cgf.12930
Bensema K, Gosink L, Obermaier H et al (2015) Modality-driven classification and visualization of ensemble variance. IEEE Trans Visual Comput Graphics 22(10):2289–2299. https://doi.org/10.1109/TVCG.2015.2507569
Chen X, Li J, Zhang Y et al (2020) Automatic feature extraction in x-ray image based on deep learning approach for determination of bone age. Futur Gener Comput Syst 110:795–801. https://doi.org/10.1016/j.future.2019.10.032
Dahshan M, Polys N, Jayne R, et al (2020) Making sense of scientific simulation ensembles with semantic interaction. In: Computer Graphics Forum, Wiley Online Library, pp 325–343. https://doi.org/10.1111/cgf.14029
de Souza CVF, Barcellos PCL, Crissaff L et al (2022) Visualizing simulation ensembles of extreme weather events. Comput Graph 104:162–172. https://doi.org/10.1016/j.cag.2022.01.007
Di S, Cappello F (2016) Fast error-bounded lossy hpc data compression with sz. In: 2016 IEEE international parallel and distributed processing symposium (ipdps), IEEE, pp 730–739. https://doi.org/10.1109/IPDPS.2016.11
Fernandes O, Frey S, Sadlo F, et al (2014) Space-time volumetric depth images for in-situ visualization. In: 2014 IEEE 4th symposium on large data analysis and visualization (LDAV), IEEE, pp 59–65. https://doi.org/10.1109/LDAV.2014.7013205
Gao S, Duan L, Tsang IW (2015) Defeatnet: a deep conventional image representation for image classification. IEEE Trans Circuits Syst Video Technol 26(3):494–505. https://doi.org/10.1109/TCSVT.2015.2389413
He N, Paoletti ME, Haut JM et al (2018) Feature extraction with multiscale covariance maps for hyperspectral image classification. IEEE Trans Geosci Remote Sens 57(2):755–769. https://doi.org/10.1109/TGRS.2018.2860464
He W, Wang J, Guo H et al (2019) Insitunet: deep image synthesis for parameter space exploration of ensemble simulations. IEEE Trans Visual Comput Graphics 26(1):23–33. https://doi.org/10.1109/TVCG.2019.2934312
Höllt T, Magdy A, Zhan P et al (2014) Ovis: a framework for visual analysis of ocean forecast ensembles. IEEE Trans Visual Comput Graphics 20(8):1114–1126. https://doi.org/10.1109/TVCG.2014.2307892
Hummel M, Obermaier H, Garth C et al (2013) Comparative visual analysis of Lagrangian transport in cfd ensembles. IEEE Trans Visual Comput Graphics 19(12):2743–2752. https://doi.org/10.1109/TVCG.2013.141
Kovalchuk SV, Boukhanovsky A (2015) Towards ensemble simulation of complex systems. In: ICCS, pp 532–541. https://doi.org/10.1016/j.procs.2015.05.280
Kumpf A, Rautenhaus M, Riemer M et al (2018) Visual analysis of the temporal evolution of ensemble forecast sensitivities. IEEE Trans Visual Comput Graphics 25(1):98–108. https://doi.org/10.1109/TVCG.2018.2864901
Kumpf A, Stumpfegger J, Härtl PF et al (2021) Visual analysis of multi-parameter distributions across ensembles of 3d fields. IEEE Trans Visual Comput Graphics 28(10):3530–3545. https://doi.org/10.1109/TVCG.2021.3061925
Kunang YN, Nurmaini S, Stiawan D, et al (2018) Automatic features extraction using autoencoder in intrusion detection system. In: 2018 International Conference on Electrical Engineering and Computer Science (ICECOS), IEEE, pp 219–224. https://doi.org/10.1109/ICECOS.2018.8605181
Leistikow S, Nahardani A, Hoerr V, et al (2020) Interactive visual similarity analysis of measured and simulated multi-field tubular flow ensembles. In: Eurographics Workshop on Visual Computing for Biology and Medicine. The Eurographics Association, pp 139–150. https://doi.org/10.2312/vcbm.20201180
Luciani T, Burks A, Sugiyama C et al (2018) Details-first, show context, overview last: supporting exploration of viscous fingers in large-scale ensemble simulations. IEEE Trans Visual Comput Graphics 25(1):1–11. https://doi.org/10.1109/TVCG.2018.2864849
Lukasczyk J, Garth C, Larsen M, et al (2020) Cinema darkroom: a deferred rendering framework for large-scale datasets. In: 2020 IEEE 10th Symposium on Large Data Analysis and Visualization (LDAV), IEEE, pp 37–41. https://doi.org/10.1109/LDAV51489.2020.00011
Luo X, Li X, Wang Z et al (2019) Discriminant autoencoder for feature extraction in fault diagnosis. Chemom Intell Lab Syst 192:103814. https://doi.org/10.1016/j.chemolab.2019.103814
Ma B, Entezari A (2018) An interactive framework for visualization of weather forecast ensembles. IEEE Trans Visual Comput Graphics 25(1):1091–1101. https://doi.org/10.1109/TVCG.2018.2864815
Maack RG, Rogers DH, Hagen H, et al (2020) Exploring cinema databases using multi-dimensional image measures
Mahajan S, Gaddis AL, Evans KJ et al (2017) Exploring an ensemble-based approach to atmospheric climate modeling and testing at scale. Procedia Computer Sci 108:735–744. https://doi.org/10.1016/j.procs.2017.05.259
Meyer M, Pfister H, Hansen C, et al (2005) Image-based volume rendering with opacity light fields. No UUSCI-2005-002 Tech Report
Mirzargar M, Whitaker RT, Kirby RM (2014) Curve boxplot: generalization of boxplot for ensembles of curves. IEEE Trans Visual Comput Graphics 20(12):2654–2663. https://doi.org/10.1109/TVCG.2014.2346455
Orban D, Keefe DF, Biswas A et al (2018) Drag and track: a direct manipulation interface for contextualizing data instances within a continuous parameter space. IEEE Trans Visual Comput Graphics 25(1):256–266. https://doi.org/10.1109/TVCG.2018.2865051
Orban D, Banesh D, Tauxe C et al (2020) Cinema: Bandit: a visualization application for beamline science demonstrated on xfel shock physics experiments. J Synchrotron Radiat 27(1):1–10. https://doi.org/10.1107/S1600577519014322
Petz C, Pöthkow K, Hege HC (2012) Probabilistic local features in uncertain vector fields with spatial correlation. Computer Graph Forum 31(3pt2):1045–1054. https://doi.org/10.1111/j.1467-8659.2012.03097.x
Ping Tian D et al (2013) A review on image feature extraction and representation techniques. Int J Multim Ubiquit Eng 8(4):385–396
Pollyea RM, Fairley JP, Podgorney RK et al (2014) Physical constraints on geologic CO2 sequestration in low-volume basalt formations. GSA Bull 126(3–4):344–351. https://doi.org/10.1130/B30874.1
Pollyea RM, Mohammadi N, Taylor JE et al (2018) Geospatial analysis of Oklahoma (USA) earthquakes (2011–2016): quantifying the limits of regional-scale earthquake mitigation measures. Geology 46(3):215–218. https://doi.org/10.1130/G39945.1
Pollyea RM, Chapman MC, Jayne RS et al (2019) High density oilfield wastewater disposal causes deeper, stronger, and more persistent earthquakes. Nat Commun. https://doi.org/10.1038/s41467-019-11029-8
Ribés A, Pouderoux J, Iooss B (2019) A visual sensitivity analysis for parameter-augmented ensembles of curves. J Verif Valid Uncertain Quantif. https://doi.org/10.1115/1.4046020
Risojević V, Babić Z (2012) Fusion of global and local descriptors for remote sensing image classification. IEEE Geosci Remote Sens Lett 10(4):836–840. https://doi.org/10.1109/LGRS.2012.2225596
Sanyal J, Zhang S, Dyer J et al (2010) Noodles: a tool for visualization of numerical weather model ensemble uncertainty. IEEE Trans Visual Comput Graphics 16(6):1421–1430. https://doi.org/10.1109/TVCG.2010.181
Sedlmair M, Heinzl C, Bruckner S et al (2014) Visual parameter space analysis: a conceptual framework. IEEE Trans Visual Comput Graphics 20(12):2161–2170. https://doi.org/10.1109/TVCG.2014.2346321
Shareef N, Lee TY, Shen HW, et al (2006) An image-based modelling approach to gpu-based unstructured grid volume rendering. In: Volume Graphics, pp 31–38. https://doi.org/10.2312/VG/VG06/031-038
Sun M, Konstantelos I, Strbac G (2018) A deep learning-based feature extraction framework for system security assessment. IEEE Trans Smart Grid 10(5):5007–5020. https://doi.org/10.1109/TSG.2018.2873001
Tikhonova A, Correa CD, Ma KL (2010) Explorable images for visualizing volume data. PacificVis 10:177–184. https://doi.org/10.1109/PACIFICVIS.2010.5429595
Torsney-Weir T, Saad A, Moller T et al (2011) Tuner: principled parameter finding for image segmentation algorithms using visual response surface exploration. IEEE Trans Visual Comput Graphics 17(12):1892–1901. https://doi.org/10.1109/TVCG.2011.248
Tuytelaars T, Mikolajczyk K, et al (2008) Local invariant feature detectors: a survey. Foundations and trends®. Comput Graph Vision, 3(3):177–280. https://doi.org/10.1561/0600000017
Wang J, Hazarika S, Li C et al (2018) Visualization and visual analysis of ensemble data: a survey. IEEE Trans Visual Comput Graphics 25(9):2853–2872. https://doi.org/10.1109/TVCG.2018.2853721
Xu K, Xia M, Mu X et al (2018) Ensemblelens: ensemble-based visual exploration of anomaly detection algorithms with multidimensional data. IEEE Trans Visual Comput Graphics 25(1):109–119. https://doi.org/10.1109/TVCG.2018.2864825
Xu W, Keshmiri S, Wang G (2019) Adversarially approximated autoencoder for image generation and manipulation. IEEE Trans Multimedia 21(9):2387–2396. https://doi.org/10.1109/TMM.2019.2898777
Ye YC, Wang Y, Miller R, et al (2015) In situ depth maps based feature extraction and tracking. In: 2015 IEEE 5th symposium on large data analysis and visualization (LDAV), IEEE, pp 1–8. https://doi.org/10.1109/LDAV.2015.7348065
Zhang Y, Li G, Yue R et al (2022) Peviz: an in situ progressive visual analytics system for ocean ensemble data. J Visualiz. https://doi.org/10.1007/s12650-022-00883-2
Zhen X, Zheng F, Shao L et al (2017) Supervised local descriptor learning for human action recognition. IEEE Trans Multimedia 19(9):2056–2065. https://doi.org/10.1109/TMM.2017.2700204
Acknowledgements
This work was supported by the US Department of Energy through the Los Alamos National Laboratory. Los Alamos National Laboratory is operated by Triad National Security, LLC, for the National Nuclear Security Administration of the US Department of Energy (Contract No. 89233218CNA000001). This research was supported by the Exascale Computing Project (17-SC-20-SC), a collaborative effort of the US Department of Energy Office of Science, and the National Nuclear Security Administration. This work was released under LA-UR-21-23424.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Dahshan, M., Polys, N., House, L. et al. Human–machine partnerships at the exascale: exploring simulation ensembles through image databases. J Vis (2024). https://doi.org/10.1007/s12650-024-00999-7
Received:
Revised:
Accepted:
Published:
DOI: https://doi.org/10.1007/s12650-024-00999-7