1. EPOS use case: High-resolution seismic tomography of Italy
Giuseppe Fiameni
with the collaboration of Alberto Michelini, Federica Magnoni, Emanuele Casarotti

2. Outline Objective Motivation EPOS’s use case Requirements
Actual status
Proposal

3. Objective High resolution tomographic velocity model of Italy
(i.e., invert the recorded seismic waveform data of earthquakes in Italy and adjacent regions for the 3D, laterally heterogeneous velocity structure of Italy.)

4. Motivation
Big scientific challenge: waveform tomography of te 3D seismic velocity structure of the Italian region up to frequencies of 0.5 Hz (i.e., maximum spatial resolution ~2 Km) has never been feasible before
Very relevant implications for modeling at high resolution strong ground shaking (large earthquakes) scenarios with evident seismic engineering applications
The use case workflow conjugates data storage retrieval, staging, HPC modeling and stage-out of large volumes of simulated data, detailed 3D velocity model and adjoint kernels

5. EPOS Workflow Observed Data Forward Modeling Inverse Visualization
Input
Parameters

6. Forward Modeling velocity model Synthetic 3D wavefield source model
2009 L’Aquila earthquake simulation for 3D velocity model and finite fault
La struttura del progetto è articolata in due fasi
velocity model
source model
topography/geology
RUN
Synthetic 3D wavefield
simulations

7. Iteratively improve the velocity model
Inverse Modeling
velocity model
source model
topography/geology
TOMOGRAPHY
Misfit between
synthetics and data
Iteratively improve the velocity model
observed data

8. Remarks on Computational Resources
All the computing times and storage presented refers to:
Central Italy adjoint tomography (Magnoni 2012) using
SPECFEM3D
INGV HPC Resources:
ELIOS
64 compute nodes, each with 2 quad-core AMD Opteron 2374 processors
at 2.4 GHz and with 16 GB RAM (512 total cores, 2 GB RAM/core)
SELENE
32 compute nodes, each with 2 oct-core AMD Opteron 6136 processors
at 2.4 GHz and with 32 GB RAM (512 total cores, 2 GB RAM/core)
then extrapolated for Italy

9. DATA & Input Parameters
Forward
Modeling
Inverse
Visualization
Observed
Data
Input
Parameters
Observed
waveforms
Source
solutions
Geology
i dati li prendiamo in FULL SEED format
tutti e tre gli elementi sono ottenuti ‘manualmente’
geology: include known information depending on frequency x x
Quality check:
no gaps
no saturation
signal/noise
station metadata
Selection:
spatial distribution
magnitude
MESH

10. MESH Geological Discontinuities (Moho, basins...) Topography
Forward
Modeling
Inverse
Visualization
Observed
Data
Input
Parameters
Geological Discontinuities (Moho, basins...)
Topography
Velocity model
MESHER (CUBIT/GEOCUBIT)

11. MESH Geological Discontinuities (moho, basin...) Topography
Velocity model
MESHER (CUBIT/GEOCUBIT)

12. Db of saved state variables for the last timestep
Forward Modeling
Observed
Data
Forward
Modeling
Inverse
Visualization
Input
Parameters
Station
location
Source
solutions
Mesh
Partitioning
SCOTCH
Synthetic
seismogram db
(ASCII - SAC)
Generate Database
Db of saved state variables for the last timestep
(binary)
SPECFEM3D
Solver
SPECFEM3D

13. Processing (user routines)
Inverse Modeling
Synthetic
seismogram db
Observed
Processing (user routines)
Source
timing
Instrument
metadata
Synchronization
Instrument response
Filtering
Data
Forward
Modeling
Inverse
Visualization
Input
Parameters
station metadata --> strumento

14. Inverse Modeling Syncronization Instrument response Filtering
Synthetic
seismogram db
Observed
Processing
Source
timing
Instrument
metadata
Syncronization
Instrument response
Filtering
Data
Forward
Modeling
Inverse
Visualization
Input
Parameters
Text file with selected windows
55DATA/MN.AQU..HHR.D.SAC.6.20.realSYN/AQU.MN.FXR.semd.sac.6.20.synt DATA/MN.AQU..HHT.D.SAC.6.20.realSYN/AQU.MN.FXT.semd.sac.6.20.synt
...
Time window selection
Serial fortran code FLEXWIN (Maggi et al 2009)
Adjoint
source db
(ASCII)
Misfit measurement and adjoint source calculation
Serial fortran code measure_adj (Tape et al. 2009)
Text file with measurement estimates +
Adjoint
source db
(ASCII)

15. (Forward simulation only if attenuation is included)
Inverse Modeling
Observed
Data
Forward
Modeling
Inverse
Visualization
Input
Parameters
Adjoint
source db
(ASCII)
Station
location
Source
solutions
Mesh
Saved state variables
for the last timestep
(Forward simulation only if attenuation is included)
Adjoint event
kernel db
(binary)
Adjoint Simulations
SPECFEM3D

16. Inverse Modeling Adjoint event kernel db (binary) Adjoint misfit
Observed
Data
Forward
Modeling
Inverse
Visualization
Input
Parameters
Adjoint event
kernel db
(binary)
Adjoint misfit
kernel db
(binary)
Regularized misfit kernel
db (binary)
SUM
SUM
Sum
SPECFEM3D fortran utilities
Preconditioning - Smoothing
SPECFEM3D fortran utilities

17. Inverse Modeling Regularized misfit kernel Starting velocity model
Observed
Data
Forward
Modeling
Inverse
Visualization
Input
Parameters
Regularized misfit kernel
db (binary)
Starting velocity model
(binary)
SPECFEM3D fortran utilities
computationally inexpensive
1.7 external_mesh.bin
3 vp vs rho in vtk
UPDATED velocity models
(binary & VTK ASCII)
Forward Simulations for steplength test
SPECFEM3D

18. Seismic source inversion
Input
Parameters
Observed
Data
Inverse Modeling
Forward Simulations
Forward
Modeling
Forward
Modeling
Inverse
Modeling
Visualization 1 2 3
Updated velocity
model db 4 5
Iterative improvement of the tomographic model 6
Seismic source inversion
FUTURE TASK

19. KERNEL SUMMATION and REGULARIZATION
Inverse Modeling
Wall Time on 256 CPUs Cluster
C. Italy
256 cores
63 events
Italy
2300 cores
150 events
Forward Simulations
Forward
Modeling
Forward
Modeling
Inverse
Modeling
Visualization
FORWARD
SIMULATIONS
42 h +
0.5/31 h
84 h
3 h
169/231 h
4 days +
0.5/75 h
8 days
3 h
16/22 days
WINDOW SELECTION
MEASUREMENTS
ADJOINT SOURCES
Updated velocity model db
(binary & vtk ASCII)
ADJOINT
SIMULATIONS
-il numero prima dello slash è se faccio selez win e meas (compreso quello nello steplength test) in parallelo cioè per tutti gli ev contemp
-steplength test ha sia le sim forw che il calcolo delle meas
Per l’Italia è fino a 0.5 Hz
KERNEL SUMMATION and REGULARIZATION
Iterative improvement of the tomographic model
STEP LENGTH TEST
~ 13/16
days
~ 28/37
days
TOTAL TIME 1 ITERATION

20. KERNEL SUMMATION and REGULARIZATION
Inverse Modeling
Storage
C. Italy
256 cores
63 events
Italy
2300 cores
150 events
Forward
Modeling
Forward Simulations
Forward
Modeling
Inverse
Modeling
Visualization
FORWARD
SIMULATIONS
60 GB +
15 GB
70 GB
7 GB
240 GB +
600 GB
WINDOW SELECTION
MEASUREMENTS
ADJOINT SOURCES
144 GB
Updated velocity model db
(binary & vtk ASCII)
ADJOINT
SIMULATIONS
-lo storage per la win select è pratic nullo, è tutto nel calcolo meas (cioè stima adj src)
-lo storage per adj sim comprende sia ev kernel che sismogrammi
-nella voce kernel c’è anche lo storage x model update di 1 iteraz (slide 14)
-steplength test = forw*4
825 GB
KERNEL SUMMATION and REGULARIZATION
30 GB
Iterative improvement of the tomographic model
STEP LENGTH TEST
2400 GB
392 GB
4 TB
TOT STORAGE 1 ITERATION

21. Visualization Seismograms (SAC, GMT, ObsPy, Google Earth)
Observed
Data
Forward
Modeling
Inverse
Visualization
Input
Parameters
Seismograms
(SAC, GMT, ObsPy, Google Earth)
Mesh - Velocity Models
(Paraview)
parallelo spt per ultimi 3 punti
Movies (Paraview)
Kernels (Paraview)
Presently using ‘manual’ scripts Parallel environment