1. EDG Application The European DataGrid Project Team http://www.eu-datagrid.org

2. EDG Applications Tutorial – n° 2 EDG Application Areas High Energy Physics Biomedical Applications Earth Observation Science Applications

3. EDG Applications Tutorial – n° 3 High Energy Physics 4 Experiments on LHC CMS ATLAS LHCb ~6-8 PetaBytes / year ~10 8 events/year ~10 3 batch and interactive users

4. EDG Applications Tutorial – n° 4 Europe: 267 institutes, 4603 users Elsewhere: 208 institutes, 1632 users CERN’s Network in the World

5. EDG Applications Tutorial – n° 5 Data Flow in LHC

6. EDG Applications Tutorial – n° 6 Example: CMS Monte Carlo Production

7. EDG Applications Tutorial – n° 7 CMS jobs description CMKIN : MC Generation of the proton- proton interaction for a physics channel (dataset) CMSIM: Detailed simulation of the CMS detector, processing the data produced during the CMKIN step CMKIN Job CMSIM Job Output data Output data Grid Storage Write to Grid Storage Element Write to Grid Storage Element Read from Grid Storage Element * PIII 1GHz 512MB  46.8 SI95 size/eventtime * /event CMKIN ~ 0.05MB~ 0.4-0.5 sec CMSIM ~ 1.8 MB ~ 6 min

8. EDG Applications Tutorial – n° 8 CMSEDG SE CE CMS software CMS production components interfaced to EDG middleware BOSS DB Workload Management System JDL RefDB parameters Push data or info Pull info UI IMPALA/BOSS CE CMS software CE CMS software CE SE  Production is managed from the EDG User Interface with IMPALA/BOSS  CMS Virtual Organization server at NIKHEF (Amsterdam)

9. EDG Applications Tutorial – n° 9 CMSEDG SE CE CMS software BOSS DB Workload Management System JDL RefDB parameters data registration input data location Push data or info Pull info UI IMPALA/BOSS Replica Manager CE CMS software CE CMS software CE WN SE CE CMS software SE  CMKIN jobs running on all EDG Testbed sites with CMS software installed  CMSIM jobs running on CE close to the input data  produced data: scripts for batch replication to a dedicated SE X CMS production components interfaced to EDG middleware

10. EDG Applications Tutorial – n° 10 CMSEDG SE CE CMS software CMS production components interfaced to EDG middleware BOSS DB Workload Management System JDL RefDB parameters data registration Job output filtering Runtime monitoring input data location Push data or info Pull info UI IMPALA/BOSS Replica Manager CE CMS software CE CMS software CE WN SE CE CMS software SE  Job monitoring and bookkeeping: BOSS DBs, EDG Logging & Bookkeeping service

11. EDG Applications Tutorial – n° 11 CMS use of the system (Statistics) CEsSEs Nb. of evts time Events Production within EDG is part of the Official CMS production http://cmsdoc.cern.ch/cms/production/www/html/general/index.html

12. EDG Applications Tutorial – n° 12 Summary of CMS work and the planning for use of EDG middleware  RESULTS n We can distribute and run CMS s/w in the EDG environment n We have generated ~250K events for physics with ~10000 jobs in 3 week period  OBSERVATIONS and PLANNING for the future n We were able to quickly add new sites to provide extra resources n There was a fast turnaround in bug fixing and installing new software n The stress test was labor intensive (since software was developing and th n Release EDG 2.0 should fix the major problems and allow for enhanced scalability,and we look forward to evaluating it and using it in our Data Challenge work

13. EDG Applications Tutorial – n° 13 ESA(IT) – KNMI(NL) Processing of raw GOME data to ozone profiles. 2 alternative algorithms ~28000 profiles/day IPSL(FR) Validate some of the GOME ozone profiles (~10 6 /y) Coincident in space and time with Ground-Based measurements Visualization & Analyze EDG EO challenge: Processing / validation of 1y of GOME data LIDAR data (7 stations, 2.5MB per month) DataGrid environment Level 2 (example of 1 day total O 3 ) Level 1 Raw satellite data from the GOME instrument (~75 GB - ~5000 orbits/y)

14. EDG Applications Tutorial – n° 14 EO WebMap Portal

15. EDG Applications Tutorial – n° 15 Web Portal EO Product Catalogue EDG Storage Element EDG User Interface EDG Resource Broker EDG Computing Element EO Replica Catalogue Processing Sequence EO Grid Engine EO Product Archive 1. Search Level-1 catalogue 2. Retrieve Level-2 products 3. Level-2 Products already registered in RC? 8. Submit jobs to process Level-1 data 7. Register Level-1 data 11. Register level-2 data 9. Process Level-1 data 10. Transfer Level-2 data to SE 12. Return new Level-2 products Yes? 4. Return available Level-2 products No? 5. Perform GRID processing on-the-fly 6. Transfer Level-1 data from Archive to the Grid

16. EDG Applications Tutorial – n° 16 GOME Ozone Profile Validation Goals of the DataGrid application validate satellite data with all ground based data available in an easy way:  Comparison of ozone profiles provided by satellite with lidar data in different locations and times (see the web portal)  Statistical comparison and analysis in order to improve algorithms. OZONE LAYER 50 km 10 km ERS/GOME satellite Lidar at the Haute Provence Observatory

17. EDG Applications Tutorial – n° 17 Validation Processing Sequence Level 2 Catalogue Lidar data catalogue Queries and data information retrieval from the Lidar metadata catalogue GRID Computing Element Storage Elements with Lidar data Queries and data information retrieval from the Gome Level 2 orbit or pixel metadata catalogues When completed comparison between lidar and satellite ozone profiles Satellite data validation Lidar site Level 2 Catalogue GRID Portal Storage Elements with Gome L2 data Submission of the Job in the GRID 1 2 3 4

18. EDG Applications Tutorial – n° 18 Validation Output Figure 1: Estimation of the bias between Gome and Lidar using one month of data. Figure 2 : example of 2 profiles : Comparison between Gome profile and lidar profile for the 2nd October 2000.

19. EDG Applications Tutorial – n° 19 Perspectives for Biomedical Applications Grids open new perspectives in large scale genomics analysis Complete genome annotation Cross-genomes analysis Data mining on distributed databases Pipelining of huge automatic bio-informatics analysis Medical image processing Large databases processing Anatomy and physiology modeling Epidemiological studies

20. EDG Applications Tutorial – n° 20 Biomedical Applications  Bio-informatics n Phylogenetics : BBE Lyon (T. Sylvestre) n Search for primers : Centrale Paris (K. Kurata) n Statistical genetics : CNG Evry (N. Margetic) n Bio-informatics web portal : IBCP (C. Blanchet) n Parasitology : LBP Clermont, Univ B. Pascal (N. Jacq) n Data-mining on DNA chips : Karolinska (R. Médina, R. Martinez) n Geometrical protein comparison : Univ. Padova (C. Ferrari)  Medical imaging n MR image simulation : CREATIS (H. Benoit-Cattin) n Medical data and metadata management : CREATIS (J. Montagnat) n Mammographies analysis ERIC/Lyon 2 (S. Miguet, T. Tweed) n Simulation platform for PET/SPECT based on Geant4 : GATE collaboration (L. Maigne) Applications deployed Applications tested on EDG Applications under preparation

21. EDG Applications Tutorial – n° 21 Medical Imaging Medical images Metadata H 1. query 2. visualisation 3. similarity search 4. scores 5. best results visualisation LFN image patient hospital...

22. EDG Applications Tutorial – n° 22 Graphic layer Job Monitoring Grid File Browsing File registration and retrieval

23. EDG Applications Tutorial – n° 23 Graphical Interfaces Image registration Image retrieval Local filesGrid files Metadata Query over metadata Query result

24. EDG Applications Tutorial – n° 24 Image Registration LFN image patient hospital... Imager SE

25. EDG Applications Tutorial – n° 25 Similarity search Similarity computation Results visualization Job monitoring Ranked list of images Source image Most similar images Low score images

26. EDG Applications Tutorial – n° 26 Future: Interfacing medical data with the Grid Client 1 interface Client 2 interface RS interface core grid - server interface header blanking encryption Storage Element Replica Catalog Replication Service RC interface Metadata interface Medical (trusted) site Grid middleware File metadata ACL size checksum... Application metadata ACL encryption key sensitive metadata... Medical server Storage Element MSS Master File Replica Imager

27. EDG Applications Tutorial – n° 27 Parallel Processing Magnetic Resonance Images simulation using the grid 3 levels of parallelism: Parallel isochromat computations Multi-slice MRI computation Parallel magnetization kernel Magnetisation computation kernel Reconstruction algorithm MRI Image Virtual object MRI sequence

28. EDG Applications Tutorial – n° 28 Summary  Use Cases n High Energy Physics n Earth Observation n Biomedical Applications

29. EDG Applications Tutorial – n° 29 Further Information  High Energy Physics http://datagrid-wp8.web.cern.ch/DataGrid-WP8/  Bio-Informatics http://marianne.in2p3.fr/datagrid/wp10/index.html  Earth Observation http://styx.esrin.esa.it/grid/