Ultra-Scale Visualization Workshop November 13, 2006 VAPOR Visualization and Analysis Platform for Ocean, atmosphere, and solar Research SC06 Ultra-Scale Visualization Workshop John Clyne & Alan Norton Scientific Computing Division National Center for Atmospheric Research Boulder, CO USA This work is funded in part through a U.S. National Science Foundation, Information Technology Research program grant

Ultra-Scale Visualization Workshop November 13, 2006 [Numerical] models that can currently be run on typical supercomputing platforms produce data in amounts that make storage expensive, movement cumbersome, visualization difficult, and detailed analysis impossible. The result is a significantly reduced scientific return from the nation's largest computational efforts. Mark Rast University of Colorado, LASP Problem Numerical simulations in the earth sciences have reached such extraordinary sizes that researchers can no longer effectively extract insight from their simulation outputs. Result: loss of scientific productivity!!!

Ultra-Scale Visualization Workshop November 13, 2006 Dichotomy of simulation and analysis needs and resources in today’s HPC environments SimulationAnalysis Large systems (O(1000) processors)Small systems (O(10) processors) Batch computing modelInteractive computing model Modest on-line storage requirements Large on-line storage requirements CPU/interconnect boundIO bound Highly tuned, custom, parallel codes COTS serial software applications Historical focus of centersEmerging focus of centers

Ultra-Scale Visualization Workshop November 13, 2006 A sampling of various technology performance curves Not all technologies advance at same rate Impact of parallelization not shown

Ultra-Scale Visualization Workshop November 13, 2006 Communication limits for volume rendering assuming theoretical peak performance GFX Memory 76 GB/sec (GeForce 7900 GX2) CPU Memory 10 GB/sec (AMD Opteron 1000 series) PCI Express 16x 4 GB/sec SATA GB/sec Table shows limits expressed as frames per second imposed by communication alone Assumes only 8-bit data quantities

Ultra-Scale Visualization Workshop November 13, 2006 Visualization and Analysis Platform for Ocean, atmosphere, and solar Research (VAPOR) Key components 1.Domain specific application focus: simulated earth sciences fluid flow 2.Coupled Visualization and quantitative data interrogation and manipulation capabilities 3.Multiresolution enabled terascale data exploration on the desktop Combination of visualization with multiresolution data representation that provide sufficient data reduction to enable interactive work on terascale data from a desktop Visual data browsing Data manipulation Quantitative analysis Refine Coarsen

Ultra-Scale Visualization Workshop November 13, 2006 Fluid flow in the geosciences E.g. Numerically simulated turbulence –Cartesian grids (usually) to Up to “hero” calculations –5 to 8 variables Temperature & Pressure Velocity field components Magnetic field components (MHD calculations) –Hundreds of time steps saved Terabytes of data per experiment –Numerical “experiments” Substantial analysis requirements Yannick Ponty, CNRS 2006

Ultra-Scale Visualization Workshop November 13, 2006 Key Component (1) : Domain specific support Only limited support for: –Grid & data types Cartesian grids, stretched and uniform sampling AMR grids Scalar and vector quantities –Visualization algorithms Volume rendering, flow visualization, cutting planes/probe –Misc. Publication quality graphics Filters File formats (one!) Extensive support for: –Time varying data Uniform as well as non-uniform sampling Missing time steps –Quantitative investigation Mathematical operators and data manipulators –Science driven specialized features Keep it simple! Keep it focused! Make it scientist friendly!

Ultra-Scale Visualization Workshop November 13, 2006 Interactive exploration of time varying data Reduce bandwidth requirements –Regions of interest –Multiresolution –Caching GFX Memory 76 GB/sec (GeForce 7900 GX2) CPU Memory 10 GB/sec (AMD Opteron 1000 series) PCI Express 16x 4 GB/sec SATA GB/sec

Ultra-Scale Visualization Workshop November 13, 2006 Future??? VAPOR Interactive visual browsing IDL Data manipulation & analysis VAPOR Data Collection Disk Array Multi-resolution access and rapid sub-region extraction Key Component (2) : Coupled visualization, quantitative analysis and manipulation capabilities IDL - array based 4GL for scientific data processing –Thousands of mathematical functions –Basic 2D plotting –Array manipulation

Ultra-Scale Visualization Workshop November 13, 2006 Key component (3) : Multiresolution data access Wavelet transformed data –Two parameter linear function decomposition –Hierarchical data representation –Invertible and lossless –Numerically efficient (O(n)) forward and inverse transform –No additional storage cost Enable speed/quality tradeoffs 504x504x2048 Full 252x252x1024 1/8 126x126x512 1/64 63x63x256 1/512

Ultra-Scale Visualization Workshop November 13, 2006 Visual comparison of a compressible convection simulation coarsened M. Rast, native

Ultra-Scale Visualization Workshop November 13, 2006 Performance of forward and inverse Haar wavelet transform System Linux RHEL x Intel 3.4 GHz Xeon EMT64 8 GBs RAM 1Gb/sec Fibre Channel storage Data Scalar Single precision Gains in microprocessor technology enable transforms at very low cost

Ultra-Scale Visualization Workshop November 13, 2006 VAPOR Demo

Ultra-Scale Visualization Workshop November 13, 2006 Summary VAPOR is a domain-specific platform for analysis, not a general purpose visualization tool Target users: fluid flow researchers in earth sciences –Limited value for medical, oil & gas, aerospace, etc. Desktop data exploration of terabyte data possible –Visualization enables rapid ROI identification –Multiresolution enables speed/quality tradeoffs

Ultra-Scale Visualization Workshop November 13, 2006 Acknowledgements Steering Committee –Nic Brummell - CU –Yuhong Fan - NCAR, HAO –Aimé Fournier – NCAR, IMAGe –Pablo Mininni, NCAR, IMAGe –Aake Nordlund, University of Copenhagen –Helene Politano - Observatoire de la Cote d'Azur –Yannick Ponty - Observatoire de la Cote d'Azur –Annick Pouquet - NCAR, ESSL –Mark Rast - CU –Duane Rosenberg - NCAR, IMAGe –Matthias Rempel - NCAR, HAO –Geoff Vasil, CU Developers –Alan Norton – NCAR, SCD –John Clyne – NCAR, SCD –Kenny Gruchalla - CU Research Collaborators –Kwan-Liu Ma, U.C. Davis –Hiroshi Akiba, U.C. Davis –Han-Wei Shen, Ohio State –Liya Li, Ohio State Systems Support –Joey Mendoza, NCAR, SCD

Ultra-Scale Visualization Workshop November 13, 2006 Questions???

Ultra-Scale Visualization Workshop November 13, 2006 Inverse Haar transform with 1/8 th volume subregion extraction System Linux RHEL x Intel 3.4 GHz Xeon EM64 8 GBs RAM 1Gb/sec Fibre Channel storage Data Scalar Single precision Data blocking permits rapid subregion extraction

Ultra-Scale Visualization Workshop November 13, 2006 The Lifting Method of wavelet construction in the spatial domain [Sweldens, 95] Split Predict Update Transform 1 Transform 2Transform j 1) Split: 2) Predict: 3) Update: Split signal into even (λ) and odd (γ) coefficients. λ will contain low frequency information, γ will contain high frequency information. Local correlation permits prediction of odd samples by even using a prediction operator, P. Capture difference between prediction and actual coefficient value. Update λ coefficients to preserve a property (e.g. mean) of original signal. A signal λ j consisting of 2 j samples

Ultra-Scale Visualization Workshop November 13, 2006 Example: Lifting Method with the Haar Wavelet λ 3,k λ 2,k λ 1,k λ 0,k γ 2,k γ 0,k γ 1,k Haar operators

Ultra-Scale Visualization Workshop November 13, 2006 NCAR Historical Estimated Sustained GFLOPS (Batch Production Systems)

Ultra-Scale Visualization Workshop November 13, 2006 NCAR Historical Estimated Sustained GFLOPS (Interactive Production Systems) Current NCAR visualization and analysis resources –~32 processors 8 nodes (6 with gfx) –~100 TB on-line storage –~800 MBs/sec aggregate storage bandwidth –~100 users (99 of which will not leave office)