1 1.Used AWP-ODC-GPU to run 10Hz Wave propagation simulation with rough fault rupture in half-space with and without small scale heterogeneities. 2.Used Hercules to run La Habra 1Hz simulations with alternative source descriptions and alternative velocity models and validate against observations. 3.Used AWP-ODC-GPU-SGT to run 1Hz CyberShake Hazard Map Calculation started April SCEC Titan Results

10Hz SORD Dynamic Rupture and Wave Propagation with and without small scale Heterogeneities 2 Image Credit: Kim Olsen, Yifeng Cui, Amit Chourasia

3 Scientific Research Objectives: High frequency deterministic wave propagation simulation in 1D layered model with and without small scale heterogeneities in velocity models. Computational Information: Systems Used: Titan Software Used: AWP-GPU Number of Nodes/GPU/Cores: 16,640 GPUs Input Data Description: 20,800 x 10,400 x 2,048 = 443 billion grid points (dx=20 m), 1D velocity model. Fractal small-scale heterogeneities with H=0.2, σ=5%. 6.8 Tbytes of input consisting of rough-fault source description and velocity model. Simulation Runtime: 5.5 hours completing 170 s of wave propagation Output Data Description: 170 Gbytes of wave propagation Other Notable Information: 2.3 Pflop/s Science Lead: Kim Bak Olsen, Yifeng Cui Ground motion simulation animation available for these simulations 10Hz Wave propagation simulation with rough fault rupture in half-space

4 1.Used AWP-ODC-GPU to run 10Hz Wave propagation simulation with rough fault rupture in half-space with and without small scale heterogeneities. 2.Used Hercules to run La Habra 1Hz simulations with alternative source descriptions and alternative velocity models and validate against observations. 3.Used AWP-ODC-GPU-SGT to run 1Hz CyberShake Hazard Map Calculation started April SCEC Titan Results

5 Scientific Research: Validation of Hercules-GPU against well observed earthquake. Compare point sources, extended (BBP) sources, CVM-S4, and improved CVM-S4.26 Computational Information: Systems Used: Titan Software Used: Hercules-GPU Number of Nodes/GPUs/Cores: : 800 Nodes / 800 GPUs for (SGTs), 40 Nodes / 640 CPUs (velocity mesh) Input Data Description: 120GB etree for each velocity model Earthquake sources: Two point source descriptions, one extended source description for LH earthquake Velocity Models: Two etrees, one each derived from cvms4 and cvms4.26 Observations: 341 stations in merged Simulation Runtime: ~1700 sec/simulation (varied for each run, and there were six runs) Output Data Description: Surface plane (250m resolution), 341 station seismograms with dt=0.05 La Habra 1Hz (Hercules), pt src, BBP extd src, cvms4, cvms4.26

6 Science Objective: Compare impact of 1D ground motion simulations to 3D ground motion simulations when used in Broadband validation tests. Computational Information (1)Systems Used: Titan (2)Software Used: Hercules GPU (3) Number of Nodes/GPU/Cores: 800 GPUs (SGTs), 640 CPUs (velocity mesh) (4) Input Data Description: 120 GB velocity mesh per Site (5) Simulation Runtime: 50 minutes for each component (6) Output Data Description: 440 GB x 2 components = 880 GB per Site Chino Hills 1Hz (Hercules) BBP with Extended Sources

CVM-S4 CVM-S4.26 Velocity Model (100 m) Point Source Validation Extended Source Validation 100 m depth Vs (m/s)Goodness-of-fit score Hercules on Titan – Validation of La Habra Validation Results Goodness-of-fit score

8 1.Used AWP-ODC-GPU to run 10Hz Wave propagation simulation with rough fault rupture in half-space with and without small scale heterogeneities. 2.Used Hercules to run La Habra 1Hz simulations with alternative source descriptions and alternative velocity models and validate against observations. 3.Used AWP-ODC-GPU-SGT to run 1Hz CyberShake Hazard Map Calculation started April SCEC Titan Results

9 CyberShake 15.4 Study using Titan and Blue Waters Los Angeles area Hazard Model based on 336 sites at 1Hz Estimated Total 40M CPU-Hours Titan (SGTs): 355K Node-Hours (10.7M CPU-Hours) Blue Waters: 928K Node-Hours (29.6M CPU-Hours) Estimated 1PB+ temporary data split between Blue Waters and Titan Estimated 11TB persistent data at SCEC Estimated Duration of Calculation: 12 Weeks CyberShake 15.4 Status 14 May 2015: 188 of 336 (56%) Sites Completed Reviewed CyberShake 15.4 preliminary results with civil engineers at May 2015 Meeting Started CyberShake 1Hz Los Angeles Seismic Hazard Model Calculation

CyberShake Rupture Generator Updated in 2014 CyberShake Rupture Generator that created rupture variations was changed from simple hypocenter spacing model to (top) to regular (center) and random (bottom). Center being used for first Los Angeles area 1Hz CyberShake model. CyberShake 15.4 preliminary results (left) provide information at frequencies not available 0.5Hz results (right) calculated in 2014.

11 Thomas H. Jordan [1], Yifeng Cui [2], Kim Olsen [3], Ricardo Taborda[4], Jacobo Bielak [5], Patrick Small [6], Efecan Poyraz [2], Jun Zhou [2], Po Chen [14], En-Jui Lee [1], Scott Callaghan [1], Robert Graves [7], Philip J Maechling [1], David Gill [1], Kevin Milner [1], Fabio Silva [1], Steven Day [3], Kyle Withers [3], William Savran [3], Zheqiang Shi [3], Mathew Norman [8], Hal Finkel [9], Gideon Juve [10], Karan Vahi [10], Ewa Deelman [10], Haydar Karaoglu [5], Yigit Isbiliroglu [11], Dorian Restrepo [12], Leonardo Ramirez-Guzman [13] [1] Southern California Earthquake Center [2] San Diego Supercomputer Center [3] San Diego State University [4] University of Memphis [5] Carnegie Mellon University [6] University of Southern California [7] U.S. Geological Survey, Pasadena, CA [8] Oak Ridge Leadership Computing Facility [9] Argonne Leadership Computing Facility [10] Information Science Institute [11] Paul C. Rizzo Associates, Inc. [12] Departamento de Ingenieria Civil Universidad Eafit [13] National University of Mexico [14] University of Wyoming Contributors