Presentation on theme: "Environmental Quality Modeling: A DOD/HPC Perspective R. S. Maier, D. Hampton, and D. R. Richards* U.S. Army Engineer Research and Development Center HPC."— Presentation transcript:
Environmental Quality Modeling: A DOD/HPC Perspective R. S. Maier, D. Hampton, and D. R. Richards* U.S. Army Engineer Research and Development Center HPC User Forum September 8-10, 2008 Tucson, AZ * HPCMP Environmental Quality Modeling and Simulation Computational Technology Area Leader
Working Definition of EQM Coupled models of [surface water] [groundwater] and related land resources.
EQM Subject Matter Physical phenomena ●hydrodynamics ●geophysics ●multi-constituent fate/transport Physical domains ●coupled atmospheric/land ●coupled surface/subsurface ●interconnection with biological species ●interconnection with anthropogenic activities.
EQM: Model Space ●Variants of Navier-Stokes equations ▪Shallow water ▪Darcy regime ▪Unsaturated flow (Richards equation) ●Advection-Diffusion (transport) equations ▪Thermal ▪Sediment ▪Salinity ▪Chemistry (What’s this?) ▪Turbulence
What’s EQM? ●A “technology area” for HPC resource allocation. ●Supports military missions and national priorities. ●Strongly identified with ERDC. ●Technology shared with Civil Works missions.
What’s EQM? Broad EQM-Supported Missions ●Stewardship and restoration of natural and cultural resources on military installations ●Predict effect of environmental conditions on mobility operations in the battlespace
EQM Computational Technology Area CFD CWO CCM EQM
EQM CTA Statistics ●90 active users ●Mainly Cray XT3, XT4 ●12M CPU hours FY08 …
EQM: Major Subprojects ●Coastal & Hydraulic (ENQ) ●Other ▪Fate and Effects (FAE) ▪Env. Mat. Sci. (EMS) ▪Countermine (C2W)
EQM Broad Missions Wave and storm surge analysis is the largest consumer of EQM HPC resources in FY08.
Storm Surge and Wave Analysis
Wave and Surge Codes Wave Modeling ●time-dependent Gulf-of-Mexico-scale wave models on basin and regional scales provide boundary conditions to ●shallow water wave models of coastal domains ●WAM (300m), WaveWatch III, STWAVE (50-100m) Storm Surge Modeling ●time-dependent free-surface calculation in coastal domains using wind fields and radiation stress gradients generated by shallow-water wave model. ●ADCIRC (30 m)
Storm Surge Analysis Requirements ●Parameter space ▪Radius of maximum wind ▪Central pressure ▪Track angle ▪Track location ▪Forward speed ●Storm set size ▪NOLA Gulf Coast simulations ▪Texas Coastline 600+ simulations ●Typical simulation ▪256 PE X 8 hrs ▪2M mesh points (2-D)
Simulation Mesh for Coastal Domain
Storm Surge Elevation
Multi-physics and Code Coupling What are the computational issues with code coupling and multi-physics?
Multi-physics and Code Coupling Wetting/Drying elements Disparate time scales Adaptive time steps Mesh adaption Inverse problems Mixed 1D/2D/3D Meshing Load balancing Well placement Material property identification Model coupling (boundary conditions) Local/global mass conservation
EQM: Broad Missions Stewardship and mobility missions have a common need for surface/subsurface models.
Countermine Systems ●ATR algorithms must discriminate between geo- environmental features and real threats. ●Field-testing airborne sensors is: ▪relatively expensive, ▪not controllable, ▪subject to registration errors, ▪affected by dynamic ground conditions. ●Simulated imagery is used to: ▪predict sensor performance under different conditions ▪train ATR algorithms.
Infrared Signature Detection
Signature Detection Modeling A key component of the energy balance is heat emitted or absorbed by the soil Heat transport is modeled in 3D to capture the surface expression of subsurface heat transport Thermal properties depend on soil moisture content. The effect of hydrologic processes requires that water movement be simulated. Moisture and thermal energy movement in the soil is computed with ADH, a surface/groundwater code.
Infrared Signature Detection
EQM: Broad Missions Stewardship mission includes predicting the fate and effects of contaminants in the subsurface.
Fate and Effects Prediction of UXO fate and toxicity in water ●degradation pathways ●toxicity ●reduction potentials ●solubility and octanol-water partition coefficients ●current interest: ▪fate of 2,4-dinitrotoluene ▪role of nitroreductase in nitrocompound breakdown ▪toxicity of nitrocompounds
Fate and Effects Nitroreductase workflow: Structure refinement with classical MD Analysis of active site QM/MM modeling Homology modeling/sequence alignment
Nitroreductase basic model Protein: ~25kDa ~ waters ~ atoms Modeled with CPMD
Structure of active site
Ab Initio QM Calculations ●Small processor counts, long run times ●Memory-intensive Codes ●good performance on SMP systems ▪Gaussian 03 ▪GAMESS --electronic structure packages that utilize ab initio, density functional theory and semi-empirical methods to calculate properties of atomic systems ▪VASP --ab initio quantum mechanical code ●Highly-scalable Codes: Parallelized with MPI and MPI/SMP mix (good performance on Distributed Memory (DM) systems) ▪CPMD --ab initio molecular dynamics code ▪NAMD --molecular dynamics code for large biomolecule
Emerging Computational Technology Stewardship mission includes effects on biological species.
Numerical Fish Surrogate McNary Lock and Dam, Columbia River
Numerical Fish Surrogate Passage of salmon around hydropower dams Design and location of bypass structures Forecast response to hydrodynamic patterns Fish swim-path selection model coupled to CFD Eulerian-Lagrangian-Agent simulation
Numerical Fish Surrogate
Frontiers of fluid-surface simulation.
Fluid-surface interactions at micro-scales ●Models of separations, filtration, percolation, dispersion, boundary layers. ●Novel methods (DEM, LBM, MD) with coupling. cement paste velocity distribution sphere packing
Closing Remarks Capabilities we have lost (developer’s view) ●Roll-in/roll-out (checkpoint restart) ●Development using full machine capacity ●Visualization in place What we have gained ●Annual capacity and performance improvements ●Exceptional management Problem: Development requirements translate to underused capacity.
Talking Points ●EQM-Supported Missions ●EQM Allocations and Technology Areas ●Largest Consumer ●Models Support Multiple Missions ●Emerging Computational Technology