1. Grid Computing Jorge Gomes Laboratório de Instrumentação e Física Experimental de Partículas R-ECFA Workshop 2008, Lisbon, 28 March 2008

2. Summary Motivation and Context Projects and Grid Infrastructures At Work The Portuguese National Grid Initiative The Portuguese LCG Tier-2 deployment Future

3. F Motivation and Context

4. LIP and grid computing LIP participates in the ATLAS and CMS experiments The LHC data management and processing requires a novel computing approach: –Highly distributed community and resources both geographically and administrative wise –Integration of all computing and storage resources from the collaborating institutes –Common interface to access the resources –Transparent access to the resources –Single sign-on –e-Infrastructure … Grid computing was adopted to implement the LHC computing infrastructure Consequently LIP become involved in grid computing Now grid computing is also being used at LIP by non-LHC experiments

5. LIP in international grid computing projects DataGridCrossGridLCGEGEE-IEELAEGEE-IIInt.Eu.Grid 2001 2002 2003 2004 2005 2006 2007 2008

6. LIP in international grid computing projects DataGrid + EGEE I and II: EU funded projects coordinated by CERN middleware and production infrastructures for multidisciplinary data intensive grid computing CrossGrid and Int.Eu.Grid: EU funded projects, middleware and production grids for data intensive parallel and interactive applications EELA: EU funded project to build a pilot grid in Latin America LCG: CERN long-term project to support the LHC computing All projects based on the same middleware EDG/LCG/gLite DataGridEGEE-IEGEE-II CrossGrid LHC Computing Grid (LCG) Int.Eu.Grid EDGLCGgLite EELA

7. Strategy 1.Learn about grid middleware (DataGrid, CrossGrid, …) –Gain know-how and experience –Follow the evolution 2.Contribute to the technology and build a team –Engineers at CERN working for LCG (AdI coordinated by LIP) –Participate in international grid projects (EGEE etc…) Infrastructure operations –Core services –User support / Helpdesk –Deployment coordination Middleware test / validation Middleware integration Resource provider Support services –Certification Authority for Portugal –Grid training 3.Deploy and operate an LCG Tier-2 for Portugal Focus in the same areas in all projects: take advantage of knowledge and synergies maximize resources work on areas related with IT operations

8. People TeamWorking Jorge GomesCoordination Mário DavidGrid Computing Gonçalo BorgesGrid Computing Manuel MonteceloGrid Computing João MartinsSystems administration Nuno DiasSystems administration, LIP CA manager Miguel OliveiraGrid Computing (Coimbra site) Hugo GomesWeb José AparícioTechnical support 9 FTEs in the computing team including grid

9. Fundings ProjectOriginDateFunding CrossGridEU2002-2005186.921 € EGEE-IEU2004-2006247.500 € EGEE-additionalFCT2005-2006100.000 € EGEE-complementFCT2005-2006119.250 € EELAEU2006-200778.000 € EGEE-IIEU2006-2008274.888 € Int.Eu.GridEU2006-2008154.000 € Main-nodeINGRID2007-2008695.336 € Middleware-validationINGRID2007-2010180.700 € EGEE-IIIEU2008-2010307.000 €

10. F Projects and Grid Infrastructures

11. The LCG depends on two major grid computing infrastructures... The biggest grid computing infrastructure worldwide LCG Infrastructure

12. 240 sites 45 countries 41,000 CPUs 5 PetaBytes >10,000 users >150 VOs >100,000 jobs/day Archeology Astronomy Astrophysics Civil Protection Comp. Chemistry Earth Sciences Finance Fusion Geophysics High Energy Physics Life Sciences Multimedia Material Sciences … 91 partners in 32 countries 25 collaborating projects

13. EGEE in Portugal Portugal and Spain constitute the EGEE Southwest federation LIP coordinates EGEE in Portugal EGEE Resource Centres: –LIP Lisboa (core services, production and pre-prod) Coimbra –Univ Lusiada Famalicão –Univ Porto Porto (3 clusters) –Univ Minho Braga –CFP-IST Lisbon –IEETA Aveiro (pre-prod)

14. The Southwest federation Total CPU COREs in the federation > 1400 Total CPU COREs in Portugal ~ 240

15. Int.EU.Grid infrastructure Grid infrastructure focused on: –Parallel processing –Interactivity –Extending the glite MW 12 centers 7 countries Virtual Organizations: –ifusion –ienvmod –iusct –ibrain –ihep –iplanck –iwien2k –icompchem –... Grid Operations Management LIP Coordinates the infrastructure operation

16. EELA E-Infrastructure shared between Europe and Latin America EU funded project coordinated by CIEMAT January 2006 – December 2007 25 partners in Europe and Latin America Mexico, Brazil, Cuba, Chile, Venezuela, Argentina, Portugal, Italy e Spain, CERN, CLARA EGEE extension to Latin America Pilot grid infrastructure Dissemination and training LIP responsible for the authentication and VO management task Coordinate the deployment of grid certification authorities Brazil, Argentina, Chile, Mexico, catch-all Virtual organizations management and authorization File catalogue core services

17. F At Work

18. Processing and storage capacity at LIP Lisbon –Farm (EGEE/LCG, Int.Eu.Grid, EELA) 88 CPU COREs AMD Opteron and Intel Xeon 2 to 4 CPU COREs per machine 1 to 2GB RAM per CPU CORE Gigabit Ethernet LRMS: Sun Grid Engine (SGE) dCache storage –Pré-production (EGEE) Small farm (Pentium III and Pentium IV) Some storage Coimbra –Farm (EGEE/LCG) 84 Workstations Pentium IV CPUs 2.2 to 3.0GHz 1 CPU per machine 1 to 2GB RAM per CPU Fast Ethernet LRMS: Torque/MAUI DPM storage

19. LIP grid core services LIP operates grid core services for several infrastructures –EGEE –Int.Eu.Grid –EELA These services include: –Resource Brokers / CrossBrokers –Information Indexes –RAS servers –LFC servers –VOMS servers –Myproxy server Hosted at: –FCCN datacenter –LIP-Lisbon datacenter

20. LIP Certification Authority The LIP CA is the grid certification authority for Portugal –Accredited by the International Grid Trust Federation (IGTF) –Registered at the TERENA TACAR trust anchor The LIP CA is member of the EUgridPMA Registration authorities at: –Centro de Física de Plasmas (IST) –Centro de Sistemas Inteligentes (UALG) –FCT/UNL –Instituto de Engenharia Electrónica e Telemática de Aveiro (UA) –LIP Lisboa –LIP Coimbra (UC) –Universidade Lusíada (Famalicão) –Universidade Autónoma de Lisboa –Universidade do Minho –Universidade do Porto http://ca.lip.pt

21. The SWE federation usage SWE federation usage: –1.863.000 jobs between Nov-2006 and Out-2007 –301.400 days normalized CPU time –257.000 days absolute CPU time –Equivalent to > 700 machines working 24h/day

22. Portugal in EGEE EGEE production infrastructure usage including LCG Grid Computing resources: LIP-Lisbon 28 - 75 CPU cores LIP-Coimbra 84 CPU cores

23. CPU usage in the EGEE SWE federation

24. Atlas + CMS + Compass + Auger

25. LIP federated Tier-2 LCG Tier-2 reliability report

26. Internet usage at LIP-Lisbon Statistics collected by FCCN The usage is mainly grid file transfer Last year Last month FCCN atinge os 2Gbps de tráfego na sua ligação internacional, dos quais mais de 800Mbps são de tráfego de investigação - Outubro, 2007 News section in www.fccn.ptwww.fccn.pt

27. FCCN international link: usage Statistics from FCCN Current capacity: 2.5Gbps Impact of LIP ~ 10 a 20% Last year Last month

28. F The Portuguese National Grid Initiative

29. National grid initiative Officially launched in April of 2006 by the Portuguese Ministry of Science –Support the development of grid infrastructures for complex problem solving –Development of competences –Integrate Portugal in major international grid infrastructures LIP participates in the coordination of the initiative Activities of the initiative –Funding: 15 pilot projects (1.500.000 €) –Networks for grid computing –Infrastructures: INGRID+ (national grid infrastructure based on EGEE) Main node for grid computing

30. International network connectivity International connectivity being deeply changed with new links being deployed: –North: Minho - Galicia –South: through Spanish Estremadura Objectives –Better geant connectivity (higher bandwidth) –Redundancy between both countries –Grid computing support –Have dedicated fibres Grid computing connectivity –Dedicated bandwidth for grid computing –Separate commodity and grid network traffic

31. Main node for grid computing Deployment of the Portuguese main node for grid computing –First step towards INGRID+ –The project began in the summer of 2007 –Expected to become operational in September of 2008 –Consortium: LIP, FCCN and LNEC Datacenter dedicated to grid computing –Host the core grid services for Portugal –Host major computing and storage resources –Host resources from other organizations Users –National grid initiative projects –National Tier-2 for the LCG –Projects in the context of IBERGRID –Portuguese researchers with demanding computing requirements

32. F The Portuguese Tier-2 deployment

33. LHC Computing Grid The Portuguese government has signed the LCG MoU A Portuguese Tier-2 for LCG is being deployed: –Resource centre providing processing and storage capacity integrated into the LHC Computing Grid –Supporting the ATLAS and CMS VOs The Portuguese Tier-2 will have 3 centres: –LIP-Lisbon Hosted at the LIP datacenter in Lisbon Operational –LIP-Coimbra Hosted in partnership with CFC Operational since mid 2007 –Main node for grid computing National grid initiative project Being built at the LNEC campus in Lisbon

34. LCG MoU The LIP federated Tier-2 is delivering 13% of the CPU capacity foreseen for 2008 The LIP Tier-2 storage capacity initially foreseen in the MoU is not sufficient LHC is about to start we need to ramp-up The available network bandwidth is already above the MoU The LCG Tier-1 for Portugal is PIC in Barcelona

35. Ongoing: processing and storage upgrade Lisbon and Coimbra – ≥360 - 384 CPU COREs two x quad core CPUs 24GB RAM 750GB SATA disks Gigabit Ethernet IPMI mgmt – >300TB storage ~ 11 - 15 servers Storage based on SATA disks RAID 5 and 6 Quad gigabit Ethernet IPMI mgmt Attachment –Fibre Channel –SAS or SATA Currently benchmarking and testing hardware from several vendors

36. Ongoing: infrastructures Lisbon –Floor reinforcement –Cooling optimization Optimize cold air delivery Close the hot corridor –Replace some racks –Optimize the room Coimbra –Reinforce AVAC + 40kW –Several civil engineering works/CFC –Fire detection and extinguishing –New racks

37. Internet connectivity Lisbon: –New fibre link LIP FCCN Fibre rented from PT Max bandwidth 10 Gbps –Agreed capacity with FCCN 1 Gigabit/s full duplex GBICs Gigabit Ethernet LX 995Mbps academic (Geant tagging) + 5Mbps commercial Coimbra: –The LIP farm is hosted at CFC and shares the network connectivity with CFC CFC shares the University of Coimbra Internet link now being upgraded The LIP cluster will have a dedicated link

38. Ongoing: network FCCN will provide network connectivity for grid computing: –Possibly enabling direct connectivity between Lisbon and Coimbra clusters –Possibly enabling more bandwidth at both sites –The FCCN infrastructure already has the hardware capabilities to provide the service –FCCN workshop on 15 April Upgrade the datacentre LANs in Lisbon and Coimbra –New core switches –Modular switches wire speed non-blocking –High density gigabit Ethernet –Some 10gigabit Ethernet ports –Redundant configuration –Ready for L2/L3 WAN connectivity

39. Infrastructure upgrades in 2007 Lisbon –New AVAC unit with 22kW capacity (total installed now 66kW close control) –Reorganization of the farm racks –Tape robot for backups LTO-3 (28TB) Coimbra –Cluster entered into operation (hosted at the CFC datacenter) –Installation of storage capacity –New 40kVA UPS

40. Main node for grid computing GRID datacenter –Being built at LNEC Civil construction ongoing Ready on September 2008 –Located near the FCCN NOC Direct connectivity to the FCCN backbone and Geant point of presence –Adequate power and cooling infrastructure for grid computing –Budget > 3.200.000€ Host: –Core GRID services INGRID+ and other projects –GRID computing cluster > 600 CPU COREs and ~ 600TB in the end of 2008 Up to ~ 2000 CPU COREs and ~ 1000TB of storage Resources managed by LIP –FCCN nearline storage Tape robot for data repositories –Host grid resources from other organizations LNEC GRID cluster Integrated in the Tier-2 LCG will be the main user

41. F Future

42. IBERGRID –A common Iberian grid infrastructure is being prepared: In the context of agreements between the Portuguese and Spanish governments Sharing of resources between Portugal and Spain –Main areas: Networking, grid, supercomputing and applications –Based on and profiting from the current collaboration in the framework of European projects: EGEE, int.eu.grid, EELA –LIP is deeply involved: Coordination of the initiative Infrastructure coordination for Portugal –Conferences: 1st conference took place in Santiago de Compostela in May of 2007 2nd conference will take place at the University of Porto

43. EGEE-III EGEE-III project approved and being signed by the partners –Expand the EGEE infrastructure More resources More users and communities –Prepare the migration towards a future European Infrastructure (EGI) based on the national grid initiatives (NGIs) Will start after EGEE-II (two years duration) Total budget 68.900.000 € –EU contribution 36.250.000 € –EU contribution for LIP 300.000 €

44. EGI Design Study proposal approved to the European Commission (started 1 st September’07) Supported by 31 National Grid Initiatives (NGIs) 2 year project to prepare the setup and operation of a new organizational model for a sustainable pan- European grid infrastructure Federated model bringing together NGIs to build a European organisation Well defined, complimentary responsibilities between NGIs and EGI http://www.eu-egi.org EGI workshop held during EGEE’07 conference last week in Budapest Prof. Gaspar Barreira appointed spokesman of the advisory committee

45. Concerns Deployment and operation of the main node for grid computing –big challenge due to the dimension and characteristics –serving new communities –core services for national grid initiative and IBERGRID The LHC is starting –the LCG usage will increase and so the problems Need more human resources –user support and tier-2 operation –but attract people has been very difficult … Costs of infrastructure operation –electricity, maintenance etc… –these costs will increase a lot with the new deployments and upgrades

46. Thank you...