1. Ultra-Scale Visualization Workshop November 13, 2006 clyne@ncar.ucar.edu VAPOR Visualization and Analysis Platform for Ocean, atmosphere, and solar Research SC06 Ultra-Scale Visualization Workshop John Clyne & Alan Norton Scientific Computing Division National Center for Atmospheric Research Boulder, CO USA This work is funded in part through a U.S. National Science Foundation, Information Technology Research program grant

2. Ultra-Scale Visualization Workshop November 13, 2006 clyne@ncar.ucar.edu [Numerical] models that can currently be run on typical supercomputing platforms produce data in amounts that make storage expensive, movement cumbersome, visualization difficult, and detailed analysis impossible. The result is a significantly reduced scientific return from the nation's largest computational efforts. Mark Rast University of Colorado, LASP Problem Numerical simulations in the earth sciences have reached such extraordinary sizes that researchers can no longer effectively extract insight from their simulation outputs. Result: loss of scientific productivity!!!

3. Ultra-Scale Visualization Workshop November 13, 2006 clyne@ncar.ucar.edu Dichotomy of simulation and analysis needs and resources in today’s HPC environments SimulationAnalysis Large systems (O(1000) processors)Small systems (O(10) processors) Batch computing modelInteractive computing model Modest on-line storage requirements Large on-line storage requirements CPU/interconnect boundIO bound Highly tuned, custom, parallel codes COTS serial software applications Historical focus of centersEmerging focus of centers

4. Ultra-Scale Visualization Workshop November 13, 2006 clyne@ncar.ucar.edu A sampling of various technology performance curves Not all technologies advance at same rate Impact of parallelization not shown

5. Ultra-Scale Visualization Workshop November 13, 2006 clyne@ncar.ucar.edu Communication limits for volume rendering assuming theoretical peak performance 256 3 512 3 1024 3 2048 3 GFX Memory 76 GB/sec (GeForce 7900 GX2) 4551568719 CPU Memory 10 GB/sec (AMD Opteron 1000 series) 5997591 PCI Express 16x 4 GB/sec 2403040.5 SATA 3.0 0.3 GB/sec 1820.30.04 Table shows limits expressed as frames per second imposed by communication alone Assumes only 8-bit data quantities

6. Ultra-Scale Visualization Workshop November 13, 2006 clyne@ncar.ucar.edu Visualization and Analysis Platform for Ocean, atmosphere, and solar Research (VAPOR) Key components 1.Domain specific application focus: simulated earth sciences fluid flow 2.Coupled Visualization and quantitative data interrogation and manipulation capabilities 3.Multiresolution enabled terascale data exploration on the desktop Combination of visualization with multiresolution data representation that provide sufficient data reduction to enable interactive work on terascale data from a desktop Visual data browsing Data manipulation Quantitative analysis Refine Coarsen

7. Ultra-Scale Visualization Workshop November 13, 2006 clyne@ncar.ucar.edu Fluid flow in the geosciences E.g. Numerically simulated turbulence –Cartesian grids (usually) 512 3 to 1024 3 Up to 4096 3 “hero” calculations –5 to 8 variables Temperature & Pressure Velocity field components Magnetic field components (MHD calculations) –Hundreds of time steps saved Terabytes of data per experiment –Numerical “experiments” Substantial analysis requirements Yannick Ponty, CNRS 2006

8. Ultra-Scale Visualization Workshop November 13, 2006 clyne@ncar.ucar.edu Key Component (1) : Domain specific support Only limited support for: –Grid & data types Cartesian grids, stretched and uniform sampling AMR grids Scalar and vector quantities –Visualization algorithms Volume rendering, flow visualization, cutting planes/probe –Misc. Publication quality graphics Filters File formats (one!) Extensive support for: –Time varying data Uniform as well as non-uniform sampling Missing time steps –Quantitative investigation Mathematical operators and data manipulators –Science driven specialized features Keep it simple! Keep it focused! Make it scientist friendly!

9. Ultra-Scale Visualization Workshop November 13, 2006 clyne@ncar.ucar.edu Interactive exploration of time varying data Reduce bandwidth requirements –Regions of interest –Multiresolution –Caching 256 3 512 3 1024 3 2048 3 GFX Memory 76 GB/sec (GeForce 7900 GX2) 4551568719 CPU Memory 10 GB/sec (AMD Opteron 1000 series) 5997591 PCI Express 16x 4 GB/sec 2403040.5 SATA 3.0 0.3 GB/sec 1820.30.04

10. Ultra-Scale Visualization Workshop November 13, 2006 clyne@ncar.ucar.edu Future??? VAPOR Interactive visual browsing IDL Data manipulation & analysis VAPOR Data Collection Disk Array Multi-resolution access and rapid sub-region extraction Key Component (2) : Coupled visualization, quantitative analysis and manipulation capabilities IDL - array based 4GL for scientific data processing –Thousands of mathematical functions –Basic 2D plotting –Array manipulation

11. Ultra-Scale Visualization Workshop November 13, 2006 clyne@ncar.ucar.edu Key component (3) : Multiresolution data access Wavelet transformed data –Two parameter linear function decomposition –Hierarchical data representation –Invertible and lossless –Numerically efficient (O(n)) forward and inverse transform –No additional storage cost Enable speed/quality tradeoffs 504x504x2048 Full 252x252x1024 1/8 126x126x512 1/64 63x63x256 1/512

12. Ultra-Scale Visualization Workshop November 13, 2006 clyne@ncar.ucar.edu Visual comparison of a 512 3 compressible convection simulation 128 3 coarsened M. Rast, 2002 512 3 native

13. Ultra-Scale Visualization Workshop November 13, 2006 clyne@ncar.ucar.edu Performance of forward and inverse Haar wavelet transform System Linux RHEL 3.0 2 x Intel 3.4 GHz Xeon EMT64 8 GBs RAM 1Gb/sec Fibre Channel storage Data Scalar Single precision Gains in microprocessor technology enable transforms at very low cost

14. Ultra-Scale Visualization Workshop November 13, 2006 clyne@ncar.ucar.edu VAPOR Demo

15. Ultra-Scale Visualization Workshop November 13, 2006 clyne@ncar.ucar.edu Summary VAPOR is a domain-specific platform for analysis, not a general purpose visualization tool Target users: fluid flow researchers in earth sciences –Limited value for medical, oil & gas, aerospace, etc. Desktop data exploration of terabyte data possible –Visualization enables rapid ROI identification –Multiresolution enables speed/quality tradeoffs

16. Ultra-Scale Visualization Workshop November 13, 2006 clyne@ncar.ucar.edu Acknowledgements Steering Committee –Nic Brummell - CU –Yuhong Fan - NCAR, HAO –Aimé Fournier – NCAR, IMAGe –Pablo Mininni, NCAR, IMAGe –Aake Nordlund, University of Copenhagen –Helene Politano - Observatoire de la Cote d'Azur –Yannick Ponty - Observatoire de la Cote d'Azur –Annick Pouquet - NCAR, ESSL –Mark Rast - CU –Duane Rosenberg - NCAR, IMAGe –Matthias Rempel - NCAR, HAO –Geoff Vasil, CU Developers –Alan Norton – NCAR, SCD –John Clyne – NCAR, SCD –Kenny Gruchalla - CU Research Collaborators –Kwan-Liu Ma, U.C. Davis –Hiroshi Akiba, U.C. Davis –Han-Wei Shen, Ohio State –Liya Li, Ohio State Systems Support –Joey Mendoza, NCAR, SCD

17. Ultra-Scale Visualization Workshop November 13, 2006 clyne@ncar.ucar.edu Questions??? www.vapor.ucar.edu

18. Ultra-Scale Visualization Workshop November 13, 2006 clyne@ncar.ucar.edu Inverse Haar transform with 1/8 th volume subregion extraction System Linux RHEL 3.0 2 x Intel 3.4 GHz Xeon EM64 8 GBs RAM 1Gb/sec Fibre Channel storage Data Scalar Single precision Data blocking permits rapid subregion extraction

19. Ultra-Scale Visualization Workshop November 13, 2006 clyne@ncar.ucar.edu The Lifting Method of wavelet construction in the spatial domain [Sweldens, 95] Split Predict Update Transform 1 Transform 2Transform j 1) Split: 2) Predict: 3) Update: Split signal into even (λ) and odd (γ) coefficients. λ will contain low frequency information, γ will contain high frequency information. Local correlation permits prediction of odd samples by even using a prediction operator, P. Capture difference between prediction and actual coefficient value. Update λ coefficients to preserve a property (e.g. mean) of original signal. A signal λ j consisting of 2 j samples

20. Ultra-Scale Visualization Workshop November 13, 2006 clyne@ncar.ucar.edu Example: Lifting Method with the Haar Wavelet 17316095 4237 35 λ 3,k λ 2,k λ 1,k λ 0,k 6-2-6-4 γ 2,k γ 0,k γ 1,k Haar operators -24 4 2

21. Ultra-Scale Visualization Workshop November 13, 2006 clyne@ncar.ucar.edu NCAR Historical Estimated Sustained GFLOPS (Batch Production Systems)

22. Ultra-Scale Visualization Workshop November 13, 2006 clyne@ncar.ucar.edu NCAR Historical Estimated Sustained GFLOPS (Interactive Production Systems) Current NCAR visualization and analysis resources –~32 processors 8 nodes (6 with gfx) –~100 TB on-line storage –~800 MBs/sec aggregate storage bandwidth –~100 users (99 of which will not leave office)