1. www.ccr.infn.it http://grid.infn.it/ gLite Information System: General Information, GRIS, GIIS, BDII, Info provider, Sensors, Configuration and Debugging Giuseppe Misurelli INFN-CNAF giuseppe.misurelli cnaf.infn.it I Corso di formazione INFN per amministratori di siti Grid Martina Franca, 5-9 Novembre 2007

2. www.ccr.infn.it http://grid.infn.it/ Disclaimer This presentation is based on materials provided and authorized by the EGEE project and is freely available to download and use according to the terms of the following license: http://creativecommons.org/licenses/by-nc-sa/2.5/

3. www.ccr.infn.it http://grid.infn.it/ Outline Introduction –Grid resources discovery and monitoring The gLite Information System (IS) –Architecture –Site components –Configuration and Debugging

4. www.ccr.infn.it http://grid.infn.it/ Introduction /1 Grid technologies enable wide-spread and coordinated use of networked resources Networked resources on a Grid are higly dynamic End-users need to be supported on the resources and service discovery –Direct discovery –Transparent discovery

5. www.ccr.infn.it http://grid.infn.it/ Introduction /2 Grid resources have to be monitored At a site level Grid info generation is a crucial service The quality of info gathered affects the performance and the behaviour of other Grid services –Meta-scheduling

6. www.ccr.infn.it http://grid.infn.it/ Introduction /3 Different info systems have been designed to accomplish discovery and monitoring processes. Each of the them with their own –Performance –Scalability –Query response EGEE solutions –gLite middleware implements two different IS  Berkley Database Information Index (BDII) Improvements of Globus Meta Data Structure  Relational Grid Monitoring Architecture (RGMA) Based on the GMA proposed by the Open Grid Forum It will not be discussed in this talk

7. www.ccr.infn.it http://grid.infn.it/ The gLite Information System (IS) –Architecture –Site components –Configuration and Debuging

8. www.ccr.infn.it http://grid.infn.it/ gLIte IS – Overview gLite adopts a combination of two components in a hierarchical system –Globus MDS  Works at the lowest level of the IS  Implements Grid Information Security (GSI) credentials and caching system  Hierarchical system composed by Grid Resource Information Service (GRIS) Grid Index information Service (GIIS) oReplaced by the Site BDII module –BDII  Introduced to overcome some Globus MDS scalability issues  Works at the highest level of the IS  Used by the Resource Broker for the matchmaking process  Can be configured by each VO  Queries underlying systems periodically (2 minutes)

9. www.ccr.infn.it http://grid.infn.it/ BDII at a glance BDII –Developed within the context of LCG project –LDAP database that is updated by a process –More than one DBs is used to separate read and write processes –A port forwarder is used internally to select the correct DB 2171 LDAP 2172 LDAP 2173 LDAP 2170 Port Fwd Update DB & Modify DB 2170 Port Fwd Swap DBs

10. www.ccr.infn.it http://grid.infn.it/ Globus MDS at a glance Globus Meta Directory Server (MDS) –Based on LDAP servers –Clients can query the system at every level –GRISes are leaves of the tree –GIISes are intermediate nodes of the tree Grid Resource Information Service (GRIS) –One for each Grid Resource (CE or SE) –Collects info on that resource –Static and dynamic info Grid Index Information Service (GIIS) –One for each site –Collects info from above GRISes –Caches info according to its validity time

11. www.ccr.infn.it http://grid.infn.it/ Info Collection Grid Site Level –Lower level: Grid Resource Information Server (GRIS)  Collects information on the state of a given resource  One GRIS on top of each resource  A set of scripts and sensor that try to extract useful info on the resource –Medium level: Site BDII  Collects information on resources of a given site  One Site BDII for each site Grid Core Service level –Higher level: BDII  Collects information on resources of a given VO  One BDII for each VO (suggested solution)

12. www.ccr.infn.it http://grid.infn.it/ Info Workflow Clients can query Top BDII as well as Site BDII with its GRISes Top BDII collects Site BDIIs’ info Each GRIS sends info to the Site BDII Specific nodes on a Grid Site (es. CE, SE, MON) run GRIS service to publish info Grid Site Local GRIS Site BDII TOP BDII At a site level, what about? Info generation Info modeling

13. www.ccr.infn.it http://grid.infn.it/ The gLite Information System (IS) –Architecture –Site components –Configuration and Debuging

14. www.ccr.infn.it http://grid.infn.it/ Info generation: Grid Info Provider Generic Information Provider (GIP) –Provides LDIF information about a grid service in accordance to the data model adopted (Glue schema) –It can be used to produce any kind of information for use with LDAP based grid information systems. –An information provider is a script that prints an LDIF file to standard output –Dynamic information (es. Running jobs, Free CPUs) is updated by means of GIP plug-ins GIP Provider Plugin Config File LDIF File Cache

15. www.ccr.infn.it http://grid.infn.it/ GIP: config files /opt/lcg/etc/lcg-info-generic.conf –Collects the info for the local set-up –Consists of a set of Key/Value pair  temp_dir - the directory to use as the GIPs cache  plugin_dir - the directory where to put links to the dynamic plugins  static_dir - the directory where to put links to the static ldif files  provider_dir - the directory where to put links to the information providers  freshness - how long to use the cache before running the dynamic plugins again  cache_ttl - how long the cache is valid  response - how long the GIP will wait for dynamic plugins before continuing  timeout - timeout for the dynamic plugins in seconds temp_dir = /opt/lcg/var/gip/tmp plugin_dir = /opt/lcg/var/gip/plugin static_dir = /opt/lcg/var/gip/ldif provider_dir = /opt/lcg/var/gip/provider freshness = 60 cache_ttl = 300 response = 110 timeout = 150

16. www.ccr.infn.it http://grid.infn.it/ GIP: main components /1 Generic script –performs the set up referring to the config file  /opt/lcg/bin/lcg-info-generic Ldif files –where the static ldif informations will be created  /opt/lcg/var/gip/ldif/ Wrapper script –used to give a clean interface between the GIP and the information system  /opt/lcg/libexec/lcg-info-wrapper

17. www.ccr.infn.it http://grid.infn.it/ GIP: main components /2 Dynamic scripts –will override the default information contained in the static ldif file when the GIP is run  /opt/lcg/libexec/lcg-info-dynamic-* Provider script –enables the GIP to be even more extensible by being able to run already existing information providers  /opt/lcg/libexec/lcg-info-provider-software Template –used to create the static ldif file, can be considered as example entries for the information that is to be produced  /opt/lcg/etc/Glue*.template

18. www.ccr.infn.it http://grid.infn.it/ Info modeling: Glue schema Grid Laboratory Uniform Environment –Conceptual model of Grid resources and services to be used in monitoring and discovery processes –Two main components of the GLUE Schema describe the attributes and the value of Site information  The computing resource  The storage resource

19. www.ccr.infn.it http://grid.infn.it/ Glue schema: Computing resources The computing power is typically offered by cluster systems divided in Grid sites Requests are typically staged into queues for an efficent stystem usage Queue polices enable service differentation (es. dedicated CPUs vs. shared CPUs assignement)

20. www.ccr.infn.it http://grid.infn.it/ Glue scema: Storage resources Storage Service: –Manages disk and tape resources in term of Storage Spaces –Hardware details are hidden –the Storage Service performs file transfer in or out of its Storage Spaces using a specified set of data access protocols (e.g. GridFTP, rfio, nfs) –files are managed in respect of the lifetime policy specified for the Storage Space where they are kept (e.g., in SRMv2, volatile, permanent and durable) Storage Space: –Assigned to a Virtual Organization –Associated to a directory of the underlying file system (es. /permanent/CMS) –has a set of policies (MaxFileSize, MinFileSize, MaxData, MaxNumFiles, MaxPinDuration, Quota) –has a set of access control base rules (to be used to publish rules to discover who can access what) –has a state (available space, used space)

21. www.ccr.infn.it http://grid.infn.it/ The gLite Information System (IS) –Architecture –Site components –Configuration and Debugging

22. www.ccr.infn.it http://grid.infn.it/ gLite IS: Configuration and Debugging Top-down approach for troubleshooting –From Site-BDII to GIP Common problems –Not exhaustive list Post-installation checks –Is the information published by my site consistent?

23. www.ccr.infn.it http://grid.infn.it/ Top-down approach Check that the information is in the site-BDII –ldapsearch -x -h site-bdii.fqdn -p 2170 -b mds-vo- name=SITENAME,o=grid  Service /etc/init.d/bdii on port 2170 Check that the information is in the GRISes –ldapsearch -x -h gris.fqdn -p 2135 -b mds-vo- name=local,o=grid  Service /etc/init.d/globus-mds on port 213 Check that information is returned by the GIP –/opt/lcg/libexec/lcg-info-wrapper  Output has to be a coherent ldif file containing a proper set of Glue attributes for your site

24. www.ccr.infn.it http://grid.infn.it/ Common problems /1 Cannot connect to the site-BDII and/or GRISes –Restart the related service Some GRIS fails to start –Error displayed by starting the GRIS by hand with debugging turned on  /opt/globus/libexec/slapd -h ldap://localhost:2135 -f /opt/globus/etc/grid-info-slapd.conf -d 255 -u edginfo –Possible stale slapd processes being left around  kill -9 slapd  /etc/rc.d/init.d/globus-mds restart

25. www.ccr.infn.it http://grid.infn.it/ Common problems /2 No information was produced by the information provider –Check that the static ldif file has been created  Defined in /opt/lcg/etc/lcg-info-generic.conf  Default location is /opt/lcg/var/gip/ldif Default values displayed instead of dynamic values –The dynamic plug-in has a problem  Error displayed by running /opt/lcg/var/gip/plugin scripts by hand New values not shown in GRIS –Possible stale slapd processes being left around  kill -9 slapd  /etc/rc.d/init.d/globus-mds restart

26. www.ccr.infn.it http://grid.infn.it/ Post installation checks Checking relevant glue attributes for a site-BDII –site-info.def variables settings (see next talks about Grid nodes installation) ldapsearch -LLL -x -H ldap://my_CE:2170 -b "mds-vo-name=my_site_name,o=grid“ GlueSiteSysAdminContact: mailto: my_support_contact (SITE_SUPPORT_SITE) GlueSiteLocation: my_site_location (SITE_LOC) GlueSiteLatitude: my_site_latitude (SITE_LAT) GlueSiteLongitude: my_site_longitude (SITE_LONG) … GlueCEAccessControlBaseRule: the application VOs plus the certification VOs (VOS) … GlueCEInfoTotalCPUs: compatible with WNs enabled for CE queues (wn.list file) GlueCEInfoLRMSType: “pbs” instead of “torque” to enabling mpi jobs submission (CE_BATCH_SYS) … GlueHostApplicationSoftwareRunTimeEnvironment: LCG current version, MPICH tag for mpi jobs, applications software tags (mandatory) (CE_RUNTIMEENV)

27. www.ccr.infn.it http://grid.infn.it/ References gLite general documentation –http://glite.web.cern.ch/glite/documentation/default.asp CERN Twiki – doc on gLite Middleware Support –https://twiki.cern.ch/twiki/bin/view/EGEE/EGEEMiddlewareSupport Articles –Information Quality Evaluation for Grid Information Services  10th IEEE International Symposium on High Performance Distributed Computing (HPDC-10 '01) p. 0181 –Grid Information Services for Distributed Resource Sharing  K. Czajkowskiy, S. Fitzgeraldz, I. Foster, C. Kesselman. Proc. 10th IEEE (HPDC-10), IEEE Press, 2001 –Grid information services (MDS)  GlobusToolkit Developer Tutorial. The GlobusProject, ArgonneNational Laboratory. USC Information Sciences Institute