1. Giuseppe Andronico INFN Catania
16-26 June 2008, Catania (Italy)
The INFNGrid Project
Giuseppe Andronico
INFN Catania

2. First South Africa Grid Training in Catania
Outline
A little of history
INFNGRID Overview
INFNGRID Release
INFNGRID Services
Monitoring and Accountig
Users and Sites Support
First South Africa Grid Training in Catania
16-26 June 2008, Catania (Italy) 2

3. First South Africa Grid Training in Catania
The INFN GRID project
The 1° National Project (Feb. 2000) aiming to develop the grid technology and the new e-infrastructure to solve LHC (and e-Science) computing requirements
e-Infrastructure = Internet + new WEB and Grid Services on top of a physical layer composed by Network, Computing, Supercomputing and Storage Resources, made properly available in a shared fashion by the new Grid services
Since then many Italian and EU projects made this a reality
Many scientific sectors in italy, EU and the entire World base now their research activities on the Grid
INFN Grid continues to be the national container used by INFN to reach its goals coordinating all the activities:
In the national, european and international Grid projects
In the standardization processes of the Open Grid Forum (OGF)
In the definition of EU policies in the ICT sector of Research Infrastructures
Through its managerial structure: Executive Board, Technical Board…
First South Africa Grid Training in Catania
16-26 June 2008, Catania (Italy) 3

4. First South Africa Grid Training in Catania
The INFN GRID portal
First South Africa Grid Training in Catania
16-26 June 2008, Catania (Italy)

5. First South Africa Grid Training in Catania
The strategy
Clear and stable objectives: development of the technology and of the infrastructure needed for the LHC computing but of general value
Variable instruments: use of projects and external funds ( from EU, MIUR...) to reach the goal
Coordination among all the projects (Executive Board)
Grid middleware & infrastructure Grid needed by INFN and LHC within a number of core European and International projects, often coordinated by CERN
DataGrid, DataTAG, EGEE, EGEE II, WLCG
Often fostered by INFN itself
International collaboration with US Globus and Condor for the middleware and Grid projects like Open Science Grid e Open Grid Forum in order to reach global interoperability among developed services and the adoption of international standards
National pioneer developments of the MW and the national infrastructure in the areas not covered by EU projects via national projects like Grid.it , LIBI, EGG …
Strong contribution to political committees: e-Infrastructure Reflection Group (eIRG ->ESFRI), EU Concertation meetings and with involved Units of Commission (F2 e F3) to establish activities programs (Calls)
First South Africa Grid Training in Catania
16-26 June 2008, Catania (Italy)

6. Some history … LHC  EGEE Grid
1999 – Monarc Project
Early discussions on how to organise distributed computing for LHC
2000 – growing interest in grid technology
HEP community was the driver in launching the DataGrid project
EU DataGrid project / EU DataTAG project
middleware & testbed for an operational grid
– LHC Computing Grid – LCG
deploying the results of DataGrid to provide a
production facility for LHC experiments
– EU EGEE project phase 1
starts from the LCG grid
shared production infrastructure
expanding to other communities and sciences
– EU EGEE-II
Building on phase 1
Expanding applications and communities …
… and in the future – Worldwide grid infrastructure??
Interoperating and co-operating infrastructures?
CERN
First South Africa Grid Training in Catania
16-26 June 2008, Catania (Italy)

7. Other FP6 activities of INFN GRID in Europe/1
To guarantee Open Source Grid Middleware evolutions towards international standards
OMII Europe
…and its availability through an effective repository
ETICS
To contribute to R&D informatics activities
Core Grid
To Coordinate EGEE extension in the world
EUMedGrid
Eu-IndiaGrid
EUChinaGrid
EELA
EU-ASIAGRID
First South Africa Grid Training in Catania
16-26 June 2008, Catania (Italy)

8. Other FP6 activities of INFN GRID in Europe/2
To promote EGEE for new scientific communities
GRIDCC (real time applications and instruments control)
BioInfoGrid (Bioinformatics: Coordinated by CNR)
LIBI (MIUR, Bionfomatics in Italy)
Cyclops (Civil Protection)
To contribute to e-IRG, the e-Infrastructure Reflection Group born in Rome the December 2003
Initiative of Italian Presidency on “eInfrastructures (Internet and Grids) – The new foundation for knowledge-based Societies” Event organised by MIUR, INFN and EU Commission
Representatives in EIRG appointed by EU Science Ministres
Policies and Roadmap for e-Infrastrutture development in EU
To coordinate participation to Open Grid Forum (ex GGF)
First South Africa Grid Training in Catania
16-26 June 2008, Catania (Italy)

9. INFN GRID / FP6 active projects
First South Africa Grid Training in Catania
16-26 June 2008, Catania (Italy)

10. First South Africa Grid Training in Catania
FP7:guarantee sustainability
The future of Grids in FP7 after 2008
EGEE proposed to European Parlament to set up an European Grid Initiative (EGI) in order to:
Guarantee long-term support & development to European e-Infrastructure based on EGEE, DEISA and the Grid national projects being fundend by the National Grid Initiatives (NGI)
Provide a coordination framework at EU level as done for the research networks by Geant, DANTE and the National Networks like GARR
The Commission asked that a plan for long-term sustainability Grid infrastructure (EGI + EGEE-III, …) to be included among the goals of EGEE-II (other than DANTE+ Geant 1-2)
The building of EGI at EU level and of a National Grid Initiave at national level is among the main goals of FP7
First South Africa Grid Training in Catania
16-26 June 2008, Catania (Italy) 10

11. First South Africa Grid Training in Catania
The future of INFNGRID :IGI
In 2006 ended Grid.IT, the 3+1 years National Project funded by MIUR with 12 M€ ( )
The future: the Italian Grid Infrastructure (IGI) Association
EU (eIRG, ESFRI) requires the fusion of different pieces of National Grids into a single National Organisation (NGI) to be unique interface to EU --> IGI for Italy
Substantial consensus for the creation of IGI for a common governance of the italian e-Infrastructure from all involved public bodies:INFN Grid, S-PACI, ENEA Grid, CNR, INAF, Centri Nazionali di supercalcolo : CINECA, CILEA, CASPUR, and new consortia “nuovi PON”
Under evaluation with MIUR the evolution of GARR towards a more general body to manage all the components of the infrastructure: Network, Grid, Digital Libraries…
Crucial for INFN in will be to manage the transition from INFN Grid to IGI, in such a way to preserve and if possible enhance the organisation levels which allowed Italy to reach world leadership and become a leading partner of EGI
First South Africa Grid Training in Catania
16-26 June 2008, Catania (Italy) 11

12. First South Africa Grid Training in Catania
Overview
INFNGRID Overview
First South Africa Grid Training in Catania
16-26 June 2008, Catania (Italy) 12

13. First South Africa Grid Training in Catania
Supported Sites
40 Sites supported:
31 INFN Sites
9 NON INFN Sites
Total Resources:
About 4600 CPUs
About 1000 TB Disk Storage
(+ About 700 TB Tape)
First South Africa Grid Training in Catania
16-26 June 2008, Catania (Italy) 13

14. First South Africa Grid Training in Catania
Supported VOs
40 VOs supported:
4 LHC (ALICE, ATLAS, CMS, LHCB)
3 cert (DTEAM, OPS, INFNGRID)
8 Regional (BIO, COMPCHEM, ENEA, INAF, INGV, THEOPHYS, VIRGO)
1 catch all VO: GRIDIT
23 Other VOs
Recentrly a new regional VO enabled: COMPASSIT
First South Africa Grid Training in Catania
16-26 June 2008, Catania (Italy) 14

15. Components of the production Grid
Grid is not only CPUs and Storage
Other elements are as much fundamental for running, managing and monitoring the grid:
Middleware
Grid Services
Monitoring tools
Accounting tools
Management and control infrastructure
Users
First South Africa Grid Training in Catania
16-26 June 2008, Catania (Italy) 15

16. First South Africa Grid Training in Catania
GRID Management
Grid management is performed by the Italian Regional Operation
Center (ROC). Its main activities are:
Production of the INFNGRID release and test it
Deployment of the release to the sites, support to local administrators and sites certification
Deployment of the release into central grid services
Maintenance of grid services
Periodical check of the resources and services status
Account the resources usage
Support at an Italian level to site managers and users
Support at an European level to site managers and users
Introduction of new Italian sites
Introduction of new regional VOs
The IT-ROC is involved in many other activities, not directly related to the production infrastructure, i.e. PreProduction, PreView and Certification Testbeds
First South Africa Grid Training in Catania
16-26 June 2008, Catania (Italy) 16

17. The Italian Regional Operation Center (ROC)
Operations Coordination Centre (OCC)
Management, oversight of all operational and support activities
Regional Operations Centres (ROC)
providing the core of the support infrastructure, each supporting a number of resource centres within its region
Grid Operator on Duty
Grid User Support (GGUS)
At FZK, coordination and management of user support, single point of contact for users
One of 10 existing ROCs in EGEE
First South Africa Grid Training in Catania
16-26 June 2008, Catania (Italy) 17

18. First South Africa Grid Training in Catania
Middleware
INFNGRID RELEASE
First South Africa Grid Training in Catania
16-26 June 2008, Catania (Italy) 18

19. First South Africa Grid Training in Catania
INFNGRID Release
The m/w installed on INFNGRID nodes is a customization of the gLite m/w used in the LCG/EGEE community. The customized INFNGRID release is packaged by the INFN release team (grid-release<at>infn.it). The ROC is responsible for the deployment of the release. At the moment the INFNGRID-3.0-Update28 (based on gLite3.0-Update 28) is deployed.
LCG
LCG 1.0
INFN-GRID
1.0
EGEE
EGEE II
2004
2007
2003
2008
LCG 2.0
2.0
gLite 3.0
3.0
2005
2006
First South Africa Grid Training in Catania
16-26 June 2008, Catania (Italy) 19

20. INFNGRID customizations: why?
VOs not supported by EGEE: define once configuration parameters (e.g. VO servers, poolaccounts, add VOMS certificates, ...) to reduce misconfiguration risks
MPI (requested by non-HEP sciences), additional GridICE config (monitor Wns), AFS read-only (CDF requirement), ...
Deploy additional middleware in a non intrusive way:
Since Nov VOMS, now in EGEE; DGAS (DataGrid Accounting System); NetworkMonitor (monitor network connection metrics)
First South Africa Grid Training in Catania
16-26 June 2008, Catania (Italy) 20

21. INFNGRID customizations
Additional VOs (~20)
GridICE on almost all profiles (including WN)
Preconfigured support for MPI:
WN without home shared, but home synchronization using scp with host based authentication
DGAS accounting:
New profile (HLR server) + additional packages on CE
NME (Network Monitor Element)
Collaboration with CNAF-T1 for Quattor
UI “PnP”
UI installable without administrator privilegies
NTP
AFS (read-only) on WN (needed by CDF VO)
First South Africa Grid Training in Catania
16-26 June 2008, Catania (Italy) 21

22. Packages and metapackages
The packages are distributed in repositories available via HTTP
For each release EGEE, there are 2 repositories collecting different types of packages:
Middleware
Security
INFNGRID customizations => 3-rd repository
First South Africa Grid Training in Catania
16-26 June 2008, Catania (Italy) 22

23. Metapackages management process
1: starting from EGEE lists, update INFNGRID lists (maintained in SVN repository)
2: once the lists are ok, to test them generate a first version of INFNGRID metapackages
3: install and/or upgrade the metapackages on the release testbed
4: if there are errors, correct and goto 2:
5: publish the new metapackages on the official repositories so they are available for everybody
First South Africa Grid Training in Catania
16-26 June 2008, Catania (Italy) 23

24. Metapackages management
our metapackages are supersets of the EGEE ones:
INFNGRID metapackage = EGEE metapackage + INFNGRID additional rpms
EGEE distributed metapackages
Flat rpm lists are available:
We maintain a customized copy of the lists and resync them easily
Using another tool (bmpl) we can generate all artifacts starting from the lists
“Our” (INFNGRID) customized metapackages
HTML files with the lists of the packages (one list per profile)
Quattor templates lists:
First South Africa Grid Training in Catania
16-26 June 2008, Catania (Italy) 24

25. First South Africa Grid Training in Catania
ig-yaim
The package ig-yaim is an extension of glite-yaim. It provides:
Additional functions or functions that override existing ones. Both are stored in functions/local instead of functions/
e.g to configure NTP, AFS, LCMAPS gridmapfile/groupmapfile, ..
More poolaccounts => ig-users.def instead of users.def
More configuration parameters => ig-site-info.def instead of site-info.def
Both packages (glite-yaim, ig-yaim) are needed!!
First South Africa Grid Training in Catania
16-26 June 2008, Catania (Italy) 25

26. First South Africa Grid Training in Catania
Documentation
Documentation is published at each release
Release notes, upgrade and installation guides:
written in LaTeX and published in html, pdf and txt
Additional informations about Updates, various Notes are published also in wiki pages:
Everything is available for site managers on a central repository
First South Africa Grid Training in Catania
16-26 June 2008, Catania (Italy) 26

27. First South Africa Grid Training in Catania
Updates
Updates deployment – Since the introduction of gLite3.0, from EGEE there where no more big release changes, but a series of smaller frequent updates (about weekly) – INFNGRID release was updated consequently
Steps:
gLite Update announcement
INFNGRID release alignment to announced update (ig-metapackages, ig-yaim)
Local testing
IT-ROC deployment
gLite Updates:
17/10/ gLite Update 06
20/10/ gLite Update 07
24/10/ gLite Update 08
14/11/ gLite Update 09
11/12/ gLite Update 10
19/12/ gLite Update 11
22/01/ gLite Update 12
05/02/ gLite Update 13
19/02/ gLite Update 14
26/02/ gLite Update 15
…….
INFNGRID Updates:
27/10/ INFNGRID Update 06/07/08
(+ new dgas, gridice packages)
15/11/ INFNGRID Update 09
19/12/ INFNGRID Update 10/11
29/01/ INFNGRID Update 12
14/02/ INFNGRID Update 13
20/02/ INFNGRID Update 14
27/02/ INFNGRID Update 15 ……
First South Africa Grid Training in Catania
16-26 June 2008, Catania (Italy) 27

28. INFNGRID services Overview
First South Africa Grid Training in Catania
16-26 June 2008, Catania (Italy) 28

29. First South Africa Grid Training in Catania
The general web portal
First South Africa Grid Training in Catania
16-26 June 2008, Catania (Italy) 29

30. The technical web portal
First South Africa Grid Training in Catania
16-26 June 2008, Catania (Italy) 30

31. First South Africa Grid Training in Catania
General Purpose Services
First South Africa Grid Training in Catania
16-26 June 2008, Catania (Italy) 31

32. First South Africa Grid Training in Catania
General purpose services – VOMS servers
First South Africa Grid Training in Catania
16-26 June 2008, Catania (Italy) 32

33. First South Africa Grid Training in Catania
VOMSes Stats
VOMS NUMBER OF USERS PER VO
VO User
argo
bio
compchem
enea
eumed
euchina
gridit
inaf
infngrid
ingv
libi
pamela
planck 16
theophys
virgo
Cdf
Egrid
TOP USERS (about 85% of total proxies):
CDF (~50k proxies/month) EUMED (~500 proxies/month) PAMELA (~500 proxies/month)
EUCHINA (~400 proxies/month) INFNGRID (Test purposes ~ 200 proxies/month)
First South Africa Grid Training in Catania
16-26 June 2008, Catania (Italy) 33

34. First South Africa Grid Training in Catania
General purpose Services - HLRs
Accounting:
Home Location Register
DGAS (Distributed Grid Accounting System) is used to account jobs running on the farm (grid and not-grid jobs)
12 HLR (1st level) distributed
1 experimental 2nd level HLR to aggregate data from 1st level
DGAS2Apel used to send job to the GOC for all sites.
First South Africa Grid Training in Catania
16-26 June 2008, Catania (Italy) 34

35. First South Africa Grid Training in Catania
VOs Dedicated Services
VO specific services previously run by the INFNGRID Certification Testbed and now moved to production  DEVEL RELEASE
New DEVEL-INFNGRID-3.1 WMS and LB are coming soon as VO dedicated services into production (atlas, cms, cdf, lhcb)
A total of 18 VO dedicated services that will become 25 with the introduction of the 3.1 WMS and LB
First South Africa Grid Training in Catania
16-26 June 2008, Catania (Italy) 35

36. First South Africa Grid Training in Catania
FTS channels and VOs
Installed and fully managed via Quattor-Yaim;
3 hosts as frontend, 1 backend oracle cluster;
Not only LHC VOs
PAMELA
VIRGO
Full standard T1-T1 + T1-T2 + STAR channels
51 channel agents;
7 VO agents;
(A prototype of) Monitoring tool available
Agent and Tomcat log file parsing and saved in a mysql db
Web interface:
Support:
Dedicated department team for Tickets;
Mailing list: fts-support<at>cnaf.infn.it
First South Africa Grid Training in Catania
16-26 June 2008, Catania (Italy) 36

37. First South Africa Grid Training in Catania
FTS transfer overview
First South Africa Grid Training in Catania
16-26 June 2008, Catania (Italy) 37

38. Monitoring and Accounting
Accounting Tools used by the ROC
First South Africa Grid Training in Catania
16-26 June 2008, Catania (Italy) 38

39. First South Africa Grid Training in Catania
Monitoring
GridICE:
Developed by INFN
Several servers with different scopes are installed and maintained by the IT-ROC
First South Africa Grid Training in Catania
16-26 June 2008, Catania (Italy) 39

40. Monitoring GSTAT: http://goc.grid.sinica.edu.tw/gstat//Italy.html
Developed out of INFN
A GSTAT server is maintained by the IT-ROC
GSTAT queries the Information System every 5 minutes
The sites and nodes checked are those registered in the GOC DB
The inconsistency of the information published and the eventual missing of a service that a site should publish are reported as an error
First South Africa Grid Training in Catania
16-26 June 2008, Catania (Italy) 40

41. First South Africa Grid Training in Catania
Monitoring
SAM:
SAM-ADMIN:
Is the CERN-EGEE official testing tool, tests are performed by jobs submitted to sites. Submission is triggered by an admin web interface. A mirror of the web interface is hosted at CNAF and maintained by the IT-ROC.
First South Africa Grid Training in Catania
16-26 June 2008, Catania (Italy) 41

42. First South Africa Grid Training in Catania
Accounting
Web interface to obtain aggregated Grid usage data. Two versions exists:
Data taken from the GridiceDB
Data taken from DGAS HLR DB – a new interface is being released
ROCRep && HLRMON:
(Data about all VOs, all sites, T1 excluded)
First South Africa Grid Training in Catania
16-26 June 2008, Catania (Italy) 42

43. First South Africa Grid Training in Catania
Accounting
Data from the HLR server are accounted into the GOC system through the dgas2apel tool
GOC ACCOUNITNG SYSTEM:
First South Africa Grid Training in Catania
16-26 June 2008, Catania (Italy) 43

44. Users and Sites Support First South Africa Grid Training in Catania
16-26 June 2008, Catania (Italy) 44

45. First South Africa Grid Training in Catania
Support
The IT-ROC offers a number of grid services and controls their correct operation. But not only….
The IT-ROC also continuously monitors the status of the sites inside the ROC itself and in case of problems helps site managers or users to find a solution.
As a parallel activity the IT-ROC is also involved in the monitoring and support of the entire EGEE infrastructure (TPM and COD) – The same support activity to users and sites given to the INFNGRID is given to the LCG/EGEE Grid on a round robin manner among the ROCs
First South Africa Grid Training in Catania
16-26 June 2008, Catania (Italy) 45

46. Users and sites support
The main tools to give support to users are the ticketing systems:
EGEE make use of the GGUS (Global Grid User Support) ticketing system (
Each ROC uses different tools interfaced to GGUS in a bidirectional Way.
By means of Web services, it is possible to:
Transfer tickets from the global to regional system
Transfer tickets from the regional to the global system
Once tickets are logged they are assigned to a proper support unit either in GGUS either in the regional systems
The IT-ROC ticketing system is based on XOOPS/xHelp
First South Africa Grid Training in Catania
16-26 June 2008, Catania (Italy) 46

47. Ticket assignment to ROC-1 First South Africa Grid Training in Catania
Interface to GGUS
GGUS System
ROC-1 Helpdesk
ROC-1 Interface
SU-1
Ticket assignment to ROC-1
SU-2
Ticket solved
GGUS/TPM
Ticket re-assigned
Web Portal
SU-N
ROC-X Helpdesk
ROC-X Interface
SU-1
SU-2
SU-N
First South Africa Grid Training in Catania
16-26 June 2008, Catania (Italy)

48. First South Africa Grid Training in Catania
Interface to GGUS
A new ticket arrives from GGUS
We assign the ticket to
the site concerning it
First South Africa Grid Training in Catania
16-26 June 2008, Catania (Italy)

49. First South Africa Grid Training in Catania
Interface to GGUS
The site reassigns the
ticket to GGUS…
…and adds a response
First South Africa Grid Training in Catania
16-26 June 2008, Catania (Italy)

50. First South Africa Grid Training in Catania
IT-ROC Control Shifts
About 20 supporters perform a monitoring activity composed by 2 shifts per day,
from Monday to Friday, with 2 persons per shift. At the end of the shift a report is produced.
During the shift the supporters:
Check the Grid status and try to discover problems before the users. In case of problems open tickets to the interested department in order to find a solution. If he/she is able suggests a possible solution.
Perform sites certification during the deployment phases
Check the status of tickets and urges experts or site-managers to give answers and solutions to them
First South Africa Grid Training in Catania
16-26 June 2008, Catania (Italy) 50

51. First South Africa Grid Training in Catania
IT-ROC Shifts ISSUES
The ROC monitoring is oriented to the infrastructure and not to the VOs
The active monitoring done via test jobs (i.e. the SAM tool) uses 3 VOs dedicated to infrastructure testing: dteam, ops and infngrid that in general have greater priority on sites  the side effect of this is that VO specific problems are not observed. Passive controls (i.e. gstat and gridice) are not affected by this problem.
The infrastructure test can be ok, but users can experience problems as well.
The actual control shift organization seems to be insufficient for the VOs needs and the LHC VOs are already performing their own tests (VO dashboards) in order to face this situation.
First South Africa Grid Training in Catania
16-26 June 2008, Catania (Italy) 51

52. First South Africa Grid Training in Catania
IT-ROC Shifts ISSUES
Both the Italian and the European experiences in Grid monitoring show that it is necessary to integrate the infrastructure oriented monitoring with a more VO specific monitoring  But just in INFNGRID we have about 40 VOs !!
Collaboration between the ROC and the people involved in the VO dashboards is desirable, at least to define a set of controls that are important for the VOs, but still not performed by the ROC
First South Africa Grid Training in Catania
16-26 June 2008, Catania (Italy) 52

53. First South Africa Grid Training in Catania
TPM and COD
The Italian ROC is involved also in the monitoring and support of the entire LCG/EGEE infrastructure. It participates to the TPM and COD activities.
TPM (Ticket Process Manager): is responsible of the right ticket assignment in the central GGUS system. When a ticket is logged it is automatically assigned to the TPM group that routes the ticket to the proper support unit or, if able, proposes a solution. The whole ticket life is under the control of the TPM that can at any time modify the ticket urging for an answer or solution.
Each ROC performs 1 week shift on a round robin cycle.
COD (CIC On Duty): the same monitoring done for the INFNGRID infrastructure is done for the EGEE infrastructure using the same tools (i.e. GSTAT, SAM, GRIDICE, GGUS) and some COD specific tools (i.e. COD dashboard)
First South Africa Grid Training in Catania
16-26 June 2008, Catania (Italy) 53

54. First South Africa Grid Training in Catania
Useful links…
THANK YOU
INFN GRID project:
Italian Production grid:
SAM:
CIC Portal:
GSTAT:
GridICE:
GOC Accounting:
First South Africa Grid Training in Catania
16-26 June 2008, Catania (Italy) 54