1. GRID computing Frascati 14/10/2004 migliori@enea.it www.telegrid.enea.it www.enea.it

2. GRID computing Frascati 14/10/2004 migliori@enea.it www.telegrid.enea.it 26 April 2005 Johon Cabot University GRID Computing: esperienze realizzative ed elementi architetturali. Outline ENEA-GRID Infrastructure and Resources Case studies: numerical simulations and support to experimental facilities Connection with other GRID projects GRID paradigm & ENEA-GRID GRID COMPUTING Real Case GRID At ENEA Ing. S. Migliori ENEA INFO [Servizio Centralizzato Informatica e Reti], Roma

3. GRID computing Frascati 14/10/2004 migliori@enea.it www.telegrid.enea.it The Grid: The Web on Steroids http:// Web: Uniform access to HTML documents Grid: Flexible, high-perf access to all significant resources Ian Foster On-demand creation of powerful virtual computing systems Sensor nets Data archives Computers Software catalogs Colleagues

4. GRID computing Frascati 14/10/2004 migliori@enea.it www.telegrid.enea.it TELEMICROSCOPY & GRID - BASED COMPUTING REMOTE ACCESS FOR DATA ACQUISITION AND ANALYSIS NETWORK DATA ACQUISITION DATA ANALYSIS ADVANCED COMPUTER GRAPHICS Cell Centered Data Base CCDB By Mark Ellisman IMAGINGINSTRUMENTS COMPUTATIONAL RESOURCES MULTI-SCALEDATA-BASES

5. GRID computing Frascati 14/10/2004 migliori@enea.it www.telegrid.enea.it Architecture --- > Function Application Fabric Connectivity Resource Collective User Sen sor net s Data archives Computers Software catalogs Colleagues ? Le Funzioni / Esigenze comandano sulla tecnologia

6. GRID computing Frascati 14/10/2004 migliori@enea.it www.telegrid.enea.it Network Exponentials Network vs. computer performance –Computer speed doubles every 18 months –Network speed doubles every 9 months –Difference = order of magnitude per 5 years 1986 to 2000 –Computers: x 500 –Networks: x 340,000 2001 to 2010 –Computers: x 60 –Networks: x 4000 Moores Law vs. storage improvements vs. optical improvements. Graph from Scientific American (Jan- 2001) by Cleo Vilett, source Vined Khoslan, Kleiner, Caufield and Perkins. Ian Foster

7. GRID computing Frascati 14/10/2004 migliori@enea.it www.telegrid.enea.it Performance Extrapolation PFlop/s computer TFlop/s To enter the list Jac. Dongarra SC2002

8. GRID computing Frascati 14/10/2004 migliori@enea.it www.telegrid.enea.it Selected Major Grid Projects Name URL & Sponsors Focus Access Gridwww.mcs.anl.gov/FL/ accessgrid; DOE, NSF Create & deploy group collaboration systems using commodity technologies BlueGridIBMGrid testbed linking IBM laboratories DISCOMwww.cs.sandia.gov/ discom DOE Defense Programs Create operational Grid providing access to resources at three U.S. DOE weapons laboratories DOE Science Grid sciencegrid.org DOE Office of Science Create operational Grid providing access to resources & applications at U.S. DOE science laboratories & partner universities Earth System Grid (ESG) earthsystemgrid.org DOE Office of Science Delivery and analysis of large climate model datasets for the climate research community European Union (EU) DataGrid eu-datagrid.org European Union Create & apply an operational grid for applications in high energy physics, environmental science, bioinformatics g g g g g g New Ian Foster

9. GRID computing Frascati 14/10/2004 migliori@enea.it www.telegrid.enea.it Selected Major Grid Projects NameURL/SponsorFocus TeraGridteragrid.org NSF U.S. science infrastructure linking four major resource sites at 40 Gb/s UK Grid Support Center grid-support.ac.uk U.K. eScience Support center for Grid projects within the U.K. UnicoreBMBFTTechnologies for remote access to supercomputers g g New Also many technology R&D projects: e.g., Condor, NetSolve, Ninf, NWS See also www.gridforum.org Ian Foster

10. GRID computing Frascati 14/10/2004 migliori@enea.it www.telegrid.enea.it ENEA Italian Agency for New Tecnologies, Energy and Environment. 12 research centers in Italy A Central Computer and Network Service (INFO) 6 Computer Centres Casaccia, Frascati, Bologna,Trisaia, Portici, Brindisi Multiplatform resources for serial & parallel computation and graphical post-processing. Others computer resources in ENEA: departments & individuals Centro di Brindisi

11. GRID computing Frascati 14/10/2004 migliori@enea.it www.telegrid.enea.it ENEA computational resources are distributed over WAN, connected by GARR, the Italian Academic & Research Network ENEA Network connection ENEA-GARR 9 PoP, 4-32 Mbps Brindisi Bologna Casaccia Frascati Portici Trisaia Palermo Pisa Roma Sede

12. GRID computing Frascati 14/10/2004 migliori@enea.it www.telegrid.enea.it ENEA-GRID has been developed since 1998 and now it is an integrated infrastructure: offering a production quality, service oriented system covering most of the computational resources belonging to ENEA INFO Scientific Computation Service, connected in a Wide Area Network. providing a unified user environment and an homogenous access method for all ENEA researchers irrespective of their location implementing tools to facilitate the integration of department and individual resources and support to experimental facilities ENEA GRID

13. GRID computing Frascati 14/10/2004 migliori@enea.it www.telegrid.enea.it Services for research activity in ENEA Multiplatform Parallel systems: AIX, Linux (Alpha,x86,AMD64), IRIX, [Unicos (Cray)] Graphical simulation & code result post-processing: SGI + 3D immersive facilities Software resources: Commercial Codes: Fluent, Gambit, Abacus, Catia, Ansys Research codes: mcpn/x, eranos, fluka,… Elaboration environments: IDL, Matlab, Mathematica, SAS Windows applications ENEA GRID Services

14. GRID computing Frascati 14/10/2004 migliori@enea.it www.telegrid.enea.it ENEA GRID Computational resources Portici, Brindisi & Trisaia resources increased in the framework of TELEGRID Project Frascati(6), Portici(4),Trisaia(4), Brindisi(4) 5018Windows Frascati(1), Trisaia(13) 8021Mac OS X Trisaia(4), Casaccia(2), Bologna(2) 108Solaris Frascati(8), Casaccia(4), Portici(1),Trisaia(8), Brindisi(1), Bologna(5) 4026IRIX Casaccia 10080Linux Alpha Frascati(28), Casaccia(54), Portici(4),Trisaia(4), Brindisi(4) 200 96 Linux x86 32/64 Frascati(128), Bologna(8), Portici(18) 1000154AIX Gflops#cpu OS

15. GRID computing Frascati 14/10/2004 migliori@enea.it www.telegrid.enea.it Citrix Metaframe ICA WEB(ICA) power3.frascati.enea.it boquad.bologna.enea.it dafne.casaccia.enea.it Kleos.portici.enea.it GEANT infocal.trisia.enea.it ……...brindisi.enea.it Portici Brindis i Citrix Nfuse new

16. GRID computing Frascati 14/10/2004 migliori@enea.it www.telegrid.enea.it AFS Geographical cross platform & File System AIX SGI SUN HP WinLinux Linux Alpha MAC LSF User programs and commercial codes LSF multi-cluster as integrator XTERM Graphical User Interface Application Portal (Metaframe/Nfuse) ICAWEB Quality of service Monitoring, Automatic recovery Accaunting Load Leveler RMS ENEA GRID

17. GRID computing Frascati 14/10/2004 migliori@enea.it www.telegrid.enea.it Software Conceptual Structure AFS Load LevelerLSF Graphic User Interface LSF multi-cluster as integrator Telnet User programs & commercial code ICA WEB Cluster 1° ( Bologna) AFS Load LevelerLSF Graphic User Interface LSF multi-cluster as integrator Telnet User programs & commercial code ICA WEB Cluster 3° (Frascati) AFS Load LevelerLSF Graphic User Interface LSF multi-cluster as integrator Telnet User programs & commercial code ICA WEB Cluster 2° (Nord Rome) AFS Load LevelerLSF Graphic User Interface LSF multi-cluster as integrator Telnet User programs & commercial code ICA WEB Cluster 4° (Trisaia ) LSF Graphic User Interface Telnet User programs & commercial code ICA WEB Cluster 5° ( Portici) File System LSF Graphic User Interface ICA WEB Cluster 6° ( Brindisi) NEW

18. GRID computing Frascati 14/10/2004 migliori@enea.it www.telegrid.enea.it Gestione generale di una specifica richiesta al sistema Interfaccia Grafica (RICHIESTA) LSF Cluster Configuration (STATICO) Stato delle Risorse (TEMPO REALE) Politica delle code (STATICO) Accoda la richiesta Assegna la risorsa Lancia il comando sul calcolatore selezionato DATI (SW/Prog.) File Server Client AFS Cache locale Risolve la piattaforma Verifica i diritti di accesso Mantiene la coerenza dei dati AFS Data Base Server

19. GRID computing Frascati 14/10/2004 migliori@enea.it www.telegrid.enea.it ENEA GRID Graphics and 3D resources (1) C.R. Frascati C.R. Bologna C.R. Casaccia SGI systems are coupled with 3D immersive facilities together with 3D modelling and rendering software as CAVELIB, DV-Mockup, AVS

20. GRID computing Frascati 14/10/2004 migliori@enea.it www.telegrid.enea.it ENEA GRID Graphics and 3D resources (2) C.R. Brindisi C.R. Trisaia C.R. Portici SGI systems are coupled with 3D immersive facilities together with 3D modelling and rendering software as CAVELIB, DV-Mockup, AVS

21. GRID computing Frascati 14/10/2004 migliori@enea.it www.telegrid.enea.it ENEA GRID ARCHITECTURE A choice of mature components for reliability and easiness of support and maintenance: Distributed File System: AFS Job and resources manager: LSF Multicluster Unified GUI access: Java and Citrix Technologies Quality monitoring system: Patrol Licence Servers Integration with department and individual resources Distributed File System: AFS for software and data Licence pool sharing Possible integration with other institutions

22. GRID computing Frascati 14/10/2004 migliori@enea.it www.telegrid.enea.it GRID usage, examples and some case studies 650 registered users 2 TB stored software/data out of 4 TB available in enea.it AFS cell Examples and cases studies Running FLUENT on ENEA GRID Support to department experimental research: Biotechnology Electron microscopy Nuclear fusion research

23. GRID computing Frascati 14/10/2004 migliori@enea.it www.telegrid.enea.it Citrix Metaframe ICA WEB(ICA) power3.frascati.enea.it boquad.bologna.enea.it dafne.casaccia.enea.it Kleos.portici.enea.it GEANT infocal.trisia.enea.it ……...brindisi.enea.it Portici Brindis i Citrix Nfuse new

24. GRID computing Frascati 14/10/2004 migliori@enea.it www.telegrid.enea.it CITRIX / NFUSE ENEA GRID makes use of Citrix Metaframe to publish an application providing a unified GUI interface all the available resources Application components: Java (GUI) shell scripts EnginFrame

25. GRID computing Frascati 14/10/2004 migliori@enea.it www.telegrid.enea.it FLUENT (CFD) code batch submission LSF Options -o filename.%J- output file -w done(idjob) - start after idjob -u e-mail userid - output by E-mail -b begintime - job start time …. A FLUENT case can be run from a dedicated GUI for submission

26. GRID computing Frascati 14/10/2004 migliori@enea.it www.telegrid.enea.it Monitoring job status by xlsbatch (LSF) xlsbatch information Jobid Users Job status Queues Submission host Run host Submission time Submission command ….. Job status:

27. GRID computing Frascati 14/10/2004 migliori@enea.it www.telegrid.enea.it Monitoring host status by xlsmon (LSF) Status of sp3-1 host using xlsmon utility

28. GRID computing Frascati 14/10/2004 migliori@enea.it www.telegrid.enea.it 3D CFD Simulation on Aircraft Axial Turbin Stage with Wake & Vortex Shedding Analysis on Turbine Blades [FLUENT] Temperature & wakes interactions Large Eddy Simulation (LES) provides high resonable results on wake analysis and turbine stage performance. Secondary flow prevision and 3D Path Lines analysis Temperature contours on Stator blades & Cp experimental data comparison Mesh: 1 million cells CPU Time: 6000 hours Elapsed Time: 525 hours on 12 IBM SP3 platforms

29. GRID computing Frascati 14/10/2004 migliori@enea.it www.telegrid.enea.it Caratteristica parabolica dello smorzatore con valvola aperta fino a 3 mm Punto di progetto (apertura valvola) CFD Analysis of a fluid dumper [FLUENT ] Vortex stagnation Secondary flux Primary Flux Mesh: 600,000 cells CPU Time: 960 hours Elapsed Time: 130 hours on 8 Linux platforms

30. GRID computing Frascati 14/10/2004 migliori@enea.it www.telegrid.enea.it Blast Wave Analysis for VEGA Programm DMA-Uniroma1 – AviogroupRome 2003

31. GRID computing Frascati 14/10/2004 migliori@enea.it www.telegrid.enea.it La simulazione con Cover: il campo di Temperatura Grid: 1,5 million cells CPU Time: 5000hours Elapsed Time: 672 hours on 6 SGI platforms ( onyx2ced )

32. GRID computing Frascati 14/10/2004 migliori@enea.it www.telegrid.enea.it Uno sguardo verso il futuro: la parametrizzazione via Web 4.0 100.0 V Profilo NACA xxx STEPS Definire la geometria e le condizioni al contorno. Generare la griglia di calcolo necessaria Eseguire il calcolo Risultati del Post-processing (su Web o home/directory)

33. GRID computing Frascati 14/10/2004 migliori@enea.it www.telegrid.enea.it FLUENT performance on different platforms in ENEA GRID Cpu time normalized to the case: Sp4-1, 2 CPUs GHz#CPU IRIX 0.48Onyx. 1.88bw 34C..03 Linux 1.38Pace 1.316Ostro 1.132Sp4-1.37516Sp3-1 AIX Scaling with #CPU & platform type

34. GRID computing Frascati 14/10/2004 migliori@enea.it www.telegrid.enea.it DB_1 CPUS ENEA GRID WEB ICA SSH DNA Sequence system ( ABI Prism 3700) Trisaia DB_3 DB_2 Electronic Microscope (Brindisi) 300 Kev (sept. 2004) ENEA GRID and experimental facilities Controlled Nuclear Fusion: FTU Frascati Tokamak Upgrade Video Acquisition

35. GRID computing Frascati 14/10/2004 migliori@enea.it www.telegrid.enea.it ENEA GRID: FTU video acquisition data

36. GRID computing Frascati 14/10/2004 migliori@enea.it www.telegrid.enea.it Conceptual structure for TEM Progetto UTS Materiali Electronic Microscope (Brindisi) 300 KeV (sept. 2004) Remote Local Operation TEM Room Local Operation Remote Controlled Web-cam Remote User Local Data AFS Data Integration > 4 TByte Mtaframe AFS Special Driver for TEM Local GRID Cluster

37. GRID computing Frascati 14/10/2004 migliori@enea.it www.telegrid.enea.it ENEA-Geographic SUBMISSION PROCEDURE Input Files LIV1 List of path Input File LIV1 IDL GOME Program Output File LIV2.jpg Image Input File LIV1 List of path Input File LIV1 IDL Program GOME Output File LIV2.jpg Image Input File LIV1 List of path Input File LIV1 Case-1 Case_2 Case n WEB User afs/home ENEA GRID/DATAGRID Test case: GOME data analysis

38. GRID computing Frascati 14/10/2004 migliori@enea.it www.telegrid.enea.it ENEA GRID/DATAGRID : GOME run cases in ENEA

39. GRID computing Frascati 14/10/2004 migliori@enea.it www.telegrid.enea.it ENEA GRID/DATAGRID: GOME results on Web

40. GRID computing Frascati 14/10/2004 migliori@enea.it www.telegrid.enea.it ENEA GRID ANSALDO (GE) ENEA (ISPRA) ORNL (USA) CRIOTEC (TO) DATA sharing Applications in ENEA GRID Sharing Working Sessions (Citrix) ENEA IGNITOR/Pellet Collaboration Activity

41. GRID computing Frascati 14/10/2004 migliori@enea.it www.telegrid.enea.it Collaborazioni DMA ( Prof. ENRICO Scubba) Fluidodinamica e sviluppo di metodologie dellutilizzo di grid … Istituto Matemetica Castelnuovo (In fase di perfezionamento)

42. GRID computing Frascati 14/10/2004 migliori@enea.it www.telegrid.enea.it ENEA Main activities & for EGEE Main activities & resources for EGEE Integration of EGEE-GRID to ENEA-GRID;Integration of EGEE-GRID to ENEA-GRID; 100 CPUs at 20% of time100 CPUs at 20% of time DataGrid at ESA\ESRIN Globus Based gateway GigaBit-Link EGEE GRID Globus Based gateway ENEA-GRID S-PACI GRID Globus Based gateway GRID.IT WP13 Chem-Grid (MAT)

43. GRID computing Frascati 14/10/2004 migliori@enea.it www.telegrid.enea.it Windows servicesUnix/Linux Services CITRIX: ENEA GRID Web Access http://www.afs.enea.it

44. GRID computing Frascati 14/10/2004 migliori@enea.it www.telegrid.enea.it ENEA GRID & GRID PARADIGM GRID concept: coordinated resource sharing and problem solving in dynamic, multi-institutional virtual organizations Foster & Kesselman, The Anatomy of the Grid, 2001 ENEA GRID infrastructure has been designed to provide a multisite resource sharing inside ENEA, both to optimize the utilization of Central Computer Service (ENEA INFO) resources and to integrate the other resources available inside the organization. VO concept can not be fully developed inside an unique organization but similarities can be found with Project Groups sharing specific resources and ENEA GRID components can provide support to this (AFS groups, LSF resources). There are cases where ENEA GRID infrastructure has been used in support to the collaboration with other institutions (+ other GRID projects)

45. GRID computing Frascati 14/10/2004 migliori@enea.it www.telegrid.enea.it ENEA GRID architecture answers to many of the challenges posed by the GRID Problem : unique authentication, authorization, resource access and resource discovery, Foster & Kesselman 2001 ENEA GRID & GRID PARADIGM The choice of mature components (LSF & Citrix - proprietary, AFS - opensource project since 2000) reduce the generality of ENEA GRID as a GRID model but improves greatly the reliability and the easiness of update & management: production quality infrastructure. Sensor nets Colleagues Data archives Computers Software catalogs by Ian Foster

46. GRID computing Frascati 14/10/2004 migliori@enea.it www.telegrid.enea.it ENEA GRID EXPERIENCE Some consideration can be drawn from the analysis of the operation of ENEA GRID infrastructure in the last years. User acceptance of GRID working mode requires a new cultural approach to the access to computational resources and depends strongly on the service quality: Value of the accessed resources Efficient networking and system faults monitoring Support of user customization for GRID environment The administration of a distributed & integrated resource system is a technical, cultural and organization challenge in a diversified structure as ENEA.

47. GRID computing Frascati 14/10/2004 migliori@enea.it www.telegrid.enea.it ITALIAN NATIONAL AGENCY FOR NEW TECNOLOGY, ENERGY AND THE ENVIRONMENT

48. GRID computing Frascati 14/10/2004 migliori@enea.it www.telegrid.enea.it ENEA GRID improvements in progress Increasing resources: AIX +32 SP nodes/Frascati, Cluster 16 AMD 64bit/Casaccia, Cluster 8 G5/Trisaia, 2 CPU AIX/Brindisi (TELEGRID Project). Extending Windows services. AFS optimization. Studies in collaboration with CASPUR to manage the replication of large amount of data over WAN with better performances than AFS. Participation in CASPUR storage technology tracking activity (SAN systems and file systems testing and evaluation, AFS implementation for parallel systems [IPP Garching]=> DEISA Grid Project)

49. GRID computing Frascati 14/10/2004 migliori@enea.it www.telegrid.enea.it Connection with other GRID Projects Each GRID project is characterized by its own architectural design and middleware. The focus of the partecipation of ENEA INFO in other GRID projects is GRID interoperability Access to ENEA GRID through other GRID models should be compatible with ENEA GRID architecture: gateway implementation. Current GRID projects: EGEE [Datagrid] FIRB/Grid.it : WP13, Computational Chemistry (CHEMGRID), Perugia Univ., ENEA Casaccia, et al. ENEA activity starting July 2004 at Frascati

50. GRID computing Frascati 14/10/2004 migliori@enea.it www.telegrid.enea.it ENEA GRID and DATAGRID PROJECT ENEA has partecipated (unfunded) in DATAGRID project (ended 3/2004) through the collaboration with ESA, ESRIN Laboratory in Frascati. A gateway bewteen DATAGRID infrastructure and ENEA- GRID has been set-up at ESRIN enabling the submission of jobs from DATAGRID to ENEA-GRID The gateway has been tested running the analysis of the data from the GOME satellite (Ozone level) using IDL codes which can be easily run in a multiplatform environment. The gateway provided a limited integration between the different GRID environment: EGEE should a attain a full integration

51. GRID computing Frascati 14/10/2004 migliori@enea.it www.telegrid.enea.it ENEA-GRID and EGEE Project EGEE (Enabling GRID for e-science in Europe) is a project in the 6 th EU framework program, with the mission to deliver production level GRID services [www.eu-egee.org]. ENEA is one of the funded partners in the project, inside the Italian partecipation which is coordinated by INFN. EGEE project is starting (April 2004); a kick-off Meeting has been hold in Cork, April 2004. The installation of a small testing environment (7 Linux boxes) with standard EGEE middleware is currently in progress at ENEA Frascati. The development of a gateway to ENEA-GRID will follow.