EUFORIA FP7-INFRASTRUCTURES , Grant JRA4 Overview and plans M. Haefele, E. Sonnendrücker Euforia kick-off meeting 22 January 2008 Gothenburg

EUFORIA FP7-INFRASTRUCTURES , Grant Outline Presentation of JRA4 tasks Task 1: Existing visualization tools Task 2: Visualization & Kepler platform Task 3: Lossy data compression format Task 4: 4D and 5D visualization

EUFORIA FP7-INFRASTRUCTURES , Grant Outline Presentation of JRA4 tasks Task 1: Existing visualization tools Task 2: Visualization & Kepler platform Task 3: Lossy data compression format Task 4: 4D and 5D visualization

EUFORIA FP7-INFRASTRUCTURES , Grant Purpose Provide visualization support for project Unified visualization framework for all codes to be integrated  Post-processing visualization Integration of visualization tools in Kepler workflow  Monitoring  Steering

EUFORIA FP7-INFRASTRUCTURES , Grant Outline Presentation of JRA4 tasks Task 1: Existing visualization tools Task 2: Visualization & Kepler platform Task 3: Lossy data compression format Task 4: 4D and 5D visualization

EUFORIA FP7-INFRASTRUCTURES , Grant Python Matplotlib Pure Python library based on numpy Can be used in  python scripts  web application server that can call python Syntax close to Matlab visualization tools => Replace Matlab visualization for free ! Only 2D plots are available 3D plots will use the VTK library

EUFORIA FP7-INFRASTRUCTURES , Grant Python matplotlib

EUFORIA FP7-INFRASTRUCTURES , Grant VTK : Visualization Toolkit Open source C++ library Community of more than 1000 developers Architecture managed by a private company : Kitware Exists and evolves since 1998 => powerful and stable tool

EUFORIA FP7-INFRASTRUCTURES , Grant ParaView Software based on parallel VTK Can be seen as a GUI for VTK The user designs the VTK pipeline by himself  The user must know the different pieces of the pipeline  Great flexibility Parallelization of both processing and rendering Plugin system based on the use of new VTK classes Open source software designed by Los Alamos Lab and Kitware

EUFORIA FP7-INFRASTRUCTURES , Grant Paraview

EUFORIA FP7-INFRASTRUCTURES , Grant VisIt Software based on parallel VTK Tool to visualize different kind of data The VTK pipeline is hidden from the user  More user friendly  Less flexibility Parallelization of processing (remote processing) The plugin system proposes an intermediate layer on top of VTK Open source software designed by Lawrence Livermore Lab

EUFORIA FP7-INFRASTRUCTURES , Grant VisIt

EUFORIA FP7-INFRASTRUCTURES , Grant Task1: Work plan List visualization requirements of codes in platform Develop data (MDS+, UAL) import plugin for  Matplotlib  VisIt Develop sample visualizations for typical data in Matplotlib and VisIt Write tutorials

EUFORIA FP7-INFRASTRUCTURES , Grant Outline Presentation of JRA4 tasks Task 1: Existing visualization tools Task 2: Visualization & Kepler platform Task 3: Lossy data compression format Task 4: 4D and 5D visualization

EUFORIA FP7-INFRASTRUCTURES , Grant Monitoring tools Visualization during the simulation Check if the simulation is still OK 1D, 2D (light) dataset are enough Visualization can be performed on the gateway Visualization must be integrated to Kepler Which quantities are reliable for code monitoring ? Which tools have to be integrated into kepler ?

EUFORIA FP7-INFRASTRUCTURES , Grant Interpretation tools Visualization after the simulation Huge dataset generated by distributed simulations 1. Copy every dataset to a central visualization resource  Requires big storage and network bandwidth  Requires a single visualization resource 2. Leave dataset on the remote simulation resource  Requires no extra storage or network bandwidth  Requires remote visualization resource

EUFORIA FP7-INFRASTRUCTURES , Grant Visualization strategy In both case, we suggest to use an existing tool based on VTK: VisIt (or ParaView) Example of CSCS (J. Favre) Shared remote visualization resource strategy Data reading, processing and rendering are performed at the computer center Only resulting pictures go through the network Interactive visualization performed with ParaView

EUFORIA FP7-INFRASTRUCTURES , Grant Remaining questions Build a link between visualization tool and Kepler in order to keep track of generated pictures ? What are physicists looking for in the data ?

EUFORIA FP7-INFRASTRUCTURES , Grant Outline Presentation of JRA4 tasks Task 1: Existing visualization tools Task 2: Visualization & Kepler platform Task 3: Lossy data compression format Task 4: 4D and 5D visualization

EUFORIA FP7-INFRASTRUCTURES , Grant Large dataset management Size S of a uniformly discretized function in Bytes d=2, N=128 => 128 KB d=3, N=128 => 16 MB d=4, N=128 => 2 GB d=4, N=256 => 32 GB d=6, N=128 => 32 TB

EUFORIA FP7-INFRASTRUCTURES , Grant Large dataset management Strategy : Use data compression to reduce the size of the numerical problem Hierarchical bases of finite elements Local compression algorithm Compression integrated into the parallel simulation Efficient algorithm for slice reconstruction

EUFORIA FP7-INFRASTRUCTURES , Grant Results for a 32 GB function Particles beam result

EUFORIA FP7-INFRASTRUCTURES , Grant Results for a 32 GB function Laser-plasma interaction result

EUFORIA FP7-INFRASTRUCTURES , Grant Present status Interactive exploration of 4D data with 2D slices Integration of the compression process into the parallel simulation => Allow the export of the distribution function => Without damaging the speedup of the simulation Data reduction with a factor between 8 and 150

EUFORIA FP7-INFRASTRUCTURES , Grant Task 3: Work plan Enhance compression algorithm Parallelization of slice reconstruction algorithm Integration of these tools into VisIt, Kepler... Handle 5D dataset

EUFORIA FP7-INFRASTRUCTURES , Grant Outline Presentation of JRA4 tasks Task 1: Existing visualization tools Task 2: Visualization & Kepler platform Task 3: Lossy data compression format Task 4: 4D and 5D visualization

EUFORIA FP7-INFRASTRUCTURES , Grant Specific difficulties Visualization of data coming from kinetic simulations Particle distribution function lives in phase space The full model requires 6D + time RR Objective : visualize functions f: R 4  R How to visualize a 4D function ? How to represent the large dataset (2-32GB) for an interactive visualization ?

EUFORIA FP7-INFRASTRUCTURES , Grant Multidimensional visualization 4D distribution function f(r,  v // ) Dimension reduction Representation of a set of slices/projections Data coming from Gysela5D (CEA Cadarache)

EUFORIA FP7-INFRASTRUCTURES , Grant Demonstration Demo time !