17-Nov-18 Parallel 2D and 3D Acoustic Modeling Application for hybrid computing platform of PARAM Yuva II Abhishek Srivastava, Ashutosh Londhe*, Richa Rastogi PARCOMPTECH 2017 Presented By: Ashutosh Londhe Project Engineer Seismic Data Processing (SDP) team, High Performance Computing – Scientific and Engineering Application (HPC-S&EA) group, Centre for Development of Advanced Computing (CDAC), Pune

Content Background Objective Methodology Computational complexity 17-Nov-18 Content Background Objective Methodology Computational complexity Serial algorithm Parallel implementation Results Conclusion 17-Nov-18

Background Synthetic seismogram Wavefield BP 2004 velocity model 17-Nov-18 Background Synthetic seismogram BP 2004 velocity model Source wavelet Wavefield 17-Nov-18

17-Nov-18 Objective Development of parallel 2D/3D seismic acoustic modeling application using staggered grid Finite Difference Method (FDM). Objective of acoustic modelling is to generate synthetic seismogram for given earth model by propagating acoustic wave through earth subsurface model. 17-Nov-18

Methodology – Finite Difference operator 17-Nov-18 Methodology – Finite Difference operator 𝜕 2 𝑃 𝜕 𝑡 2 = 𝑐 2 𝑃 𝜕 𝑥 2 𝜕 𝑥 2 + 𝜕 𝑧 2 𝜕 𝑧 2 +𝐹(𝑥,𝑧) (1)   𝜕 2 𝑃 𝜕 𝑡 2 = 𝑐 2 𝑃 𝜕 𝑥 2 𝜕 𝑥 2 + 𝜕 𝑦 2 𝜕 𝑦 2 + 𝜕 𝑧 2 𝜕 𝑧 2 +𝐹(𝑥,𝑦,𝑧) (2) Where, P is acoustic pressure, c is acoustic wave speed, F is the source and (x, y, z) is dimensional positions approximation on bounded model space for corresponding unbounded model space 17-Nov-18

Methodology – Staggered grid and Leap frog scheme 17-Nov-18 Methodology – Staggered grid and Leap frog scheme Staggered grid: Leap frog scheme: Importance and uses What is staggered grid- acoustic wave equation has two variables pressure and velocity. In case of staggered pressure variable are located at grid center and velocity variables are located on grid face. Use: enable evaluation of partial derivatives over smaller grid interval which increases resolution of results and decrease truncation error. truncation error in fdm depends upon number of terms used in Taylor’s series. Leap frog schemes is used for discretisation of continuous function. In this scheme position and velocity is updated at interleaved time steps. Which u can see in fig that position is updated at dt timestep and velocity at dt+1/2 time step 17-Nov-18

Computational complexity 17-Nov-18 Computational complexity Description Values Total grid points 3832920 FD Time Step 2500 Grid point evaluated 9.58 * 109 Floating-point Operations to update one grid point 18 Memory required by one grid point (in Bytes) 4 Bytes Total floating-point operations for one simulation 1.73 * 1011 Total memory required (in Bytes) 38.32 * 109 + X* X*: Runtime memory Floating point operations/grid point = [ (7*k)/2] + 4, where k is order of FDM model size: 540m X 3500m X 4180m with grid spacing of 20m Time sampling rate: 0.002s and seismogram recorded for 5s 17-Nov-18

2D/3D Acoustic modeling Algorithm 17-Nov-18 2D/3D Acoustic modeling Algorithm Algorithm to generate synthetic seismogram using acoustic modelling is depicted in fig. Velocity and density model are two major inputs. Other inputs to the application include, grid spacing, time sampling rate, peak frequency for source wavelet and source and receiver geometry for which seismogram has to be generated and other relevant inputs. The velocity model is first checked for dispersion and stability criteria. Dispersion check is performed for checking whether the selected grid spacing will result in dispersion in result or not.  finite difference approximation is stable if the errors (truncation, round-off etc) decay as the computation proceeds from one marching step to the next. Then model is extended for absorbing wave at boundaries so there will be no reflection. Now for each shot we have to propagate the wave using FDM and record the reflection arises from interfaces in earth subsurface model on the receivers of the shot. Source wavelet is used to generate wave which get propagated in earth subsurface as acoustic wave. Here we have used a Ricker wavelet as a source wavelet. The above steps are repeated for each shot. 17-Nov-18

Parallel implementation 17-Nov-18 Parallel implementation The fig. shows the algorithm implementation. The data decomposition in our case shot workload distribution is achieved using MPI. Further parallelisation is achieved thro’ OpenMP implementation in FD module as shown by red box. 17-Nov-18

17-Nov-18 Optimizations Apart from MPI and OpenMP parallelization the application performance is enhanced using the different optimization methodologies shown in fig. 17-Nov-18

List of relevant parameter used for 2D & 3D acoustic modeling 17-Nov-18 List of relevant parameter used for 2D & 3D acoustic modeling Description 2D Synthetic Data 3D Extracted Data Model Y dimension (y) - 540m Model X dimension (x) 60000m 13500m Model Z dimension (z) 40000m 4180m Grid size in Y dimension (dy) 20m Grid size in X dimension (dx) 40m Grid size in Z dimension (dz) Grid point in Y direction (ny) 27 Grid point in X direction (nx) 1501 676 Grid point in Z direction (nz) 1001 210 Sampling interval for FDTD (dt) 0.003s 0.002s Resampling interval (Seisdt) 0.008s Record Length (RecLen) 15.0s 5.0s Grid points for PML (NSP) 60 17-Nov-18

CPU compute time for 100 shots of 2D/3D wave propagation 17-Nov-18 2D/3D parallel acoustic modeling compute time CPU compute time for 100 shots of 2D/3D wave propagation 17-Nov-18

CPU Node scaling & efficiency for 100 shots of 2D wave propagation 17-Nov-18 2D parallel acoustic modeling node scaling and efficiency CPU Node scaling & efficiency for 100 shots of 2D wave propagation 17-Nov-18

CPU Node scaling & efficiency for 100 shots of 3D wave propagation 17-Nov-18 3D parallel acoustic modeling node scaling and efficiency CPU Node scaling & efficiency for 100 shots of 3D wave propagation 17-Nov-18

2D wave propagation snapshots at different time 17-Nov-18 2D acoustic wavefield 2D Velocity model 2D wave propagation snapshots at different time 17-Nov-18

Generated 2D synthetic seismogram. 17-Nov-18 2D acoustic seismogram Generated 2D synthetic seismogram. 17-Nov-18

3D wave propagation snapshot at 1 s, 2 s, 3 s and 5 s 3D acoustic wavefield 3D Velocity model 17-Nov-18 3D wave propagation snapshot at 1 s, 2 s, 3 s and 5 s

Generated 3D synthetic seismogram. 17-Nov-18 3D acoustic seismogram Generated 3D synthetic seismogram. 17-Nov-18

Conclusion The developed modeling technique is efficient, scalable and robust for modeling complex subsurface. Load imbalance occurs in case of uneven distribution of shot points. Further work in this is to implement domain decomposition for 3D modeling. It will be major module for seismic imaging techniques such as Full Waveform Inversion (FWI) and Reverse Time Migration (RTM) which give detailed structure of earths subsurface. 17-Nov-18

17-Nov-18 Thank You Email: ashutoshl@cdac.in 17-Nov-18