N ATIONAL P ARTNERSHIP FOR A DVANCED C OMPUTATIONAL I NFRASTRUCTURE Fast Adaptive Storage and Retrieval Scott B. Baden Department of Computer Science and Engineering University of California, San Diego

N ATIONAL P ARTNERSHIP FOR A DVANCED C OMPUTATIONAL I NFRASTRUCTURE Motivation Some applications are able to distinguish interesting features from background data using on-line analysis

N ATIONAL P ARTNERSHIP FOR A DVANCED C OMPUTATIONAL I NFRASTRUCTURE Features

N ATIONAL P ARTNERSHIP FOR A DVANCED C OMPUTATIONAL I NFRASTRUCTURE Animation

N ATIONAL P ARTNERSHIP FOR A DVANCED C OMPUTATIONAL I NFRASTRUCTURE Fast Adaptive Storage and Retrieval If the volume fraction of interesting data is small, then we can reduce storage, memory, and network bandwidth requirements significantly by storing only what is “needed” We call a scheme that realizes this capability Adaptive Storage and Retrieval (FASTR) This is a new paradigm for scientific users, since they are reluctant to part with their data We use resources only to the extent that we require them: remote knowledge discovery and data browsing

N ATIONAL P ARTNERSHIP FOR A DVANCED C OMPUTATIONAL I NFRASTRUCTURE The KeLP Project C++ run time libraries for parallel application & library development Hide low level details without sacrificing performance Irregular block structured data Express communication at a high level using intuitive geometric set operations Also applies to data intensive applications KeLP I/O: out of core (Bradley Broom, Rice)

N ATIONAL P ARTNERSHIP FOR A DVANCED C OMPUTATIONAL I NFRASTRUCTURE Data intensive application of KeLP KDistuf Turbulent flow with Direct Numerical Simulation Collaboration involving K. Nomura (UCSD MAE), W. Kerney and D. Shalit (UCSD CSE), G. Balls (UCSDSC), P. Diamessis (USC) Content-based data compression Borrow structured adaptive mesh refinement grid techniques to… Capture features at full resolution Discard remaining background data

N ATIONAL P ARTNERSHIP FOR A DVANCED C OMPUTATIONAL I NFRASTRUCTURE More about the application Turbulent mixing in stably stratified flow under the influence of background shear Solve the incompressible Navier Stokes equations Follow the time evolution of regions of overturned dense fluid, which are the main agents of stirring and mixing “The efficiency of mixing in turbulent patches: inferences from direct simulations and microstructure observations,” in press, J. Phys. Ocean. Smyth, Moum, and Caldwell, 2001.

N ATIONAL P ARTNERSHIP FOR A DVANCED C OMPUTATIONAL I NFRASTRUCTURE Information discovery Oceanographic observations are incomplete: restricted to 1 dimensional observations Discovery: time evolution, energy dissipation and lifetime of overturn regions, which have irregular shapes Bill Smyth, Dept. Oceanic & Atmospheric Sciences, Oregon State University

N ATIONAL P ARTNERSHIP FOR A DVANCED C OMPUTATIONAL I NFRASTRUCTURE Fast Adaptive Storage and Retrieval Compression depends on the data, currently on Best case ~ 20:1 compression (10 GB  500 MB), worst case ~ 2.8:1 Lempel-Ziv (gzip) give us only 10%

N ATIONAL P ARTNERSHIP FOR A DVANCED C OMPUTATIONAL I NFRASTRUCTURE Further savings: another application Use volume tracking [Silver, Rutgers] to follow individual features FASTR permits us to extract only the data we need out of the many features present Computational volume: 2M pts Average feature size: 1K points Maximum feature size: 20K pts Saves additional two orders of magnitude in communication bandwidth requirements Perform local analysis on a workstation

N ATIONAL P ARTNERSHIP FOR A DVANCED C OMPUTATIONAL I NFRASTRUCTURE Future plans Develop remote analysis capability Integrate with DTF data handling infrastructure Larger scale simulations on Blue Horizon and on clusters: Study vortex pairs in a stratified turbulent environment Improved understanding of aircraft wake vortices Practical importance for air traffic control

N ATIONAL P ARTNERSHIP FOR A DVANCED C OMPUTATIONAL I NFRASTRUCTURE Remote analysis capability Perform analysis on data sets stored remotely, e.g. Data Cutter We can perform some data analysis on a local workstation For highly intensive data analysis, we can use higher end resources, but again we access only the data we need

N ATIONAL P ARTNERSHIP FOR A DVANCED C OMPUTATIONAL I NFRASTRUCTURE Publications and people FASTR is based on a research prototype called MOLD, which is the M.S. thesis research of UCSD CSE student William Kerney “MOLD: A System for Breaking Down Large Visualization and Post-Processing Problems.” Expected March Peter Diamessis, then a PhD student with Keiko Nomura (UCSD MAE Dept), used MOLD to carry out an exploration of overturns An Investigation of Vortical Structures and Density Overturns in Stably Stratified Homogeneous Turbulence by Means of Direct Numerical Simulation, P. Diamessis, PhD thesis, 2001 “Automated Tracking of Turbulent Structures in Direct Numerical Simulation,” P. Diamessis et al, PARA 2002, Helsinki, Finland. To appear.

N ATIONAL P ARTNERSHIP FOR A DVANCED C OMPUTATIONAL I NFRASTRUCTURE Software availability FASTR- contact us KeLP Hardened version of KeLP, AKA KeLP1.4 NPACI Blue Horizon, Sun HPC, Cray T3E, Linux clusters Workstations: Solaris, Linux, etc. Dual tier variant, KeLP2.1: hierarchical KeLP for SMP clusters and SMP based machines (e.g. BH)