AcrossGrids Conference – Santiago 13 of February 2003 First Prototype of the CrossGrid Testbed Jorge Gomes (LIP) On behalf of X# WP4

AcrossGrids Conference – Santiago 13 of February 2003 CrossGrid testbed –The CrossGrid international testbed includes 16 sites across 9 European countries. –The site list includes: Small computing facilities in Universities and research centers. Large computing centers. –CrossGrid offers an ideal mixture of sites to test the possibilities of the grid technologies. –It is expected that more sites will join in.

AcrossGrids Conference – Santiago 13 of February 2003 CrossGrid testbed sites The testbed sites seen from the CrossGrid Mapcenter. Mapcenter was developed by DataGrid. The CrossGrid Mapcenter is maintained by LIP.

AcrossGrids Conference – Santiago 13 of February 2003 Testbed middleware –One of the aims of CrossGrid is to extend the grid coverage in Europe hence it needs to: be compatible with other testbeds namely DataGrid. develop and build upon existing middleware such as Globus and EDG. –Since the CrossGrid middleware is still being developed the initial testbed was based entirely in EDG and Globus middleware.

AcrossGrids Conference – Santiago 13 of February 2003 CrossGrid testbeds –The CrossGrid initial testbed deployment started in May 2002 with 4 sites. –The testbed was setup in the context of the quality assurance activities and was used to test and validate EDG 1.2.x. –The initial testbed has grown and was recently divided in two. Initial testbed EDG 1.2.2/3 Production testbed EDG 1.2.2/3 Validation testbed EDG 1.4.3

AcrossGrids Conference – Santiago 13 of February 2003 CrossGrid testbeds Production testbed –Production testbed Used to run applications. –Validation testbed Used to test new production middleware. –Development testbed Used to support the development of middleware, applications and integration of new testbed releases. –In the future three testbeds will coexist. Release process Validation testbed Development testbed

AcrossGrids Conference – Santiago 13 of February 2003 CrossGrid testbed resources Production testbedValidation testbed Computing Elements14Computing Elements3 Worker Nodes69Worker Nodes4 CPUs110CPUs5 Storage Elements14Storage Elements3 Storage capacity2.7TBStorage capacity1.2TB –Of the 16 sites foreseen: 10 are fully available. 2 are deployed and being tested. 2 are currently in the validation testbed. 2 are deployed but not yet available (not tested). –The X# testbeds already offer considerable computing and storage resources.

AcrossGrids Conference – Santiago 13 of February 2003 Networking Spain Portugal Poland Germany Greece Cyprus Ireland Slovakia Netherlands Austria –CrossGrid uses the Géant backbone for international connectivity. Géant

AcrossGrids Conference – Santiago 13 of February 2003 Networking –Site connectivity is provided by the National Research Networks. UCY Nikosia DEMO Athens Auth Thessaloniki CYFRONET Cracow ICM & IPJ Warsaw PSNC Poznan CSIC IFIC Valencia UAB Barcelona CSIC-UC IFCA Santander CSIC RedIris Madrid LIP Lisbon USC Santiago TCD Dublin UvA Amsterdam FZK Karlsruhe II SAS Bratislava Géant

AcrossGrids Conference – Santiago 13 of February 2003 X# Production sites status

AcrossGrids Conference – Santiago 13 of February 2003 The X# production Resource Broker JDL Job request Resource broker Any CrossGrid User Interface JSS Central site Lisbon 1. 2.Jobs are sent to the RB located atLIP 3.The RB submits the job to a CE using GRAM 1.Job requests are submitted from remote UIs 2.Jobs are sent to the RB located at LIP 4.4.The RB submits the job to a CE using GRAM AnyX# Remote site Cracow Lisbon Valencia Bratislava Poznan Barcelona Warsaw Thessaloniki Santiago Karlsruhe (not available) Athens (not available) Dublin I 3. The RB uses site information in the matchmaking 3

AcrossGrids Conference – Santiago 13 of February 2003 Central X# production services Central services Monitoring LCFG VO RC RB MyProxy L I P Network Portuguese Research Network GÉANT Network MyProxy RB RC VO UI Monitoring : Certification proxy : Resource broker : Replica catalogue : Virtual organisation server : User interface : Grid monitoring The CrossGrid production central services are located in Lisbon and maintained by LIP.

AcrossGrids Conference – Santiago 13 of February 2003 Central X# production services (2) –Resource Broker Matchmaking and load balancing scheduler. Performs load sharing across X# sites. –Certificate Proxy server Short lived certificates for long lived processes. Used by the application portals and by the RB. –Virtual Organizations server Database for user authentication. Is used to build the authorization databases of all X# sites. –Replica Catalogue Database for physical replica file location. Central service for finding files in SEs. –Monitoring Early detection of problems in the X# testbed.

AcrossGrids Conference – Santiago 13 of February 2003 The X# production Replica Catalogue –The production RC: Basically an LDAP server. Hosted at lngrid08.lip.pt port Used by the RB and RM. VOCollectionDescription crossgridcgtst0CrossGrid collection wpsixwpsixtst0Used in tests atlasatlastst0Used in tests cmscmstst0Not used URL: ldap://lngrid08.lip.pt:9980 /rc=CrossGridReplicaCatalogue,dc=lngrid08,dc=lip,dc=pt

AcrossGrids Conference – Santiago 13 of February 2003 X# validation sites status –The validation testbed was created in the context of the task 4.4 testbed quality assurance. –Currently EDG is being tested. –All three sites have been successfully deployed. –The central services for the validation testbed have been successfully deployed at LIP.

AcrossGrids Conference – Santiago 13 of February 2003 The X# validation Resource Broker JDL Job request Resource Broker Any CrossGrid User Interface JSS Central site Lisbon 1. 2.Jobs are sent to the RB located atLIP 3.The RB submits the job to a CE using GRAM 1.Job requests are submitted from remote UIs 2.Jobs are sent to the RB located at LIP 4.4.The RB submits the job to a CE using GRAM AnyX# Remote site Lisbon Karlsruhe Athens Information Index 3. The RB uses site information in the matchmaking 3 Self registration New server

AcrossGrids Conference – Santiago 13 of February 2003 Central X# validation services Central services Monitoring LCFG VO RC RB MyProxy L I P Network Portuguese Research Network GEANT Network MyProxy RB RC VO UI Monitoring II : Certification proxy : Resource broker : Replica catalogue : Virtual organisation server : User interface : Grid monitoring : Information Index The CrossGrid validation central services are located in Lisbon and maintained by LIP. I

AcrossGrids Conference – Santiago 13 of February 2003 Central X# validation services (2) –Information Index TOP MDS information server contains pointers to the site information servers. –Resource Broker Matchmaking and load balancing scheduler. Performs load sharing across X# sites. –Replica Catalogue Database for physical replica file location. Central service to find the location of files in SEs. –Certificate Proxy server Shared with the production testbed. –Virtual Organizations server Shared with the production testbed. –Monitoring Shared with the production testbed.

AcrossGrids Conference – Santiago 13 of February 2003 The X# validation Replica Catalogue –The production RC: Basically an LDAP server. Hosted at rc01.lip.pt port Used by the RB and RM. VOCollectionDescription crossgridcgCrossGrid collection URL: ldap://rc01.lip.pt:9980/rc=CG Replica Catalog,dc=rc01,dc=lip,dc=pt

AcrossGrids Conference – Santiago 13 of February 2003 Production and validation systems hosted at LIP Gatekeeper (lngrid02) Production Test and Validation Shared Local Resources Cent ral Services WN (...) SE (lngrid03) UI (lngrid05) RB (lngrid06) RC (lngrid08) RB (rb01) RC (rc01) II (ii01) Gatekeeper (ce01) WN (...) SE (se01) UI (ui01) LCFG (lngrid01) CA (OFFLINE) MyProxy (lngrid07) VO (lnnet05) Monitoring (lnnet07)

AcrossGrids Conference – Santiago 13 of February 2003 Virtual Organizations –CrossGrid has a dedicated VO server The VO server is used to build the authorization databases of the X# testbed systems. Currently is an LDAP server. Hosted at grid-vo.lip.pt port users are registered in the crossgrid VO. VOGroupDescription crossgridtestbed1All CrossGrid users cgTValphaTest and validation experts cgTVbetaTest and validation users gdmpserversapptbAll production GDMP servers gdmpserverstvtbAll validation GDMP servers gdmpserversdevtbNot used

AcrossGrids Conference – Santiago 13 of February 2003 Certification Authorities –Five new CAs were created and are now recognized by CrossGrid. –All CAs are operational issuing certs and CRLs. –All CAs are recognized by the DataGrid Certification Authorities Task force (Cyprus is currently finishing the acceptance process).

AcrossGrids Conference – Santiago 13 of February 2003 Certification Authorities (2) –However the work is not yet complete. Sometimes CRLs expire causing denial of service. A tool to monitor the CRL issuance is being developed. Possibly the same will happen with the issued certificates since they have 1 year of lifetime. A tool to monitor the validity of the host certificates is also being developed. The new Cyprus CA is not installed everywhere. Security: policies and procedures to deal with certificate compromise are required. A draft was recently written. A manual explaining the theory behind certificates and how they should be used is required.

AcrossGrids Conference – Santiago 13 of February 2003 Testbed support –The CrossGrid helpdesk application is being tested and is almost ready. –The current sources of support are: –The support for the central services is currently provided by LIP.

AcrossGrids Conference – Santiago 13 of February 2003 Testbed support (2) –A support knowledge database with solutions for common problems has been adapted and is being tested: Web access (PHP + SQL). Users will be able to send questions. Questions will be routed to the right expert. User and administrator level almost finished. –Unified CrossGrid/DataGrid helpdesk One support DB for both projects. Extend the user support team. Some helpdesk guidelines already agreed.

AcrossGrids Conference – Santiago 13 of February 2003 Software repository –A CVS repository has been established at FZK in Karlsruhe. –A web portal based on GNU Savannah was deployed to interface with the repository. –Savannah is based on SourceForge 2.0. –Savannah was customized to the X# needs. –The repository is now in production and is being used by the whole project.

AcrossGrids Conference – Santiago 13 of February 2003 Software repository (2)

AcrossGrids Conference – Santiago 13 of February 2003 Monitoring and verification –Grid and network monitoring services have been deployed at LIP to monitor the X# testbed. –An installation and verification tool was developed at LIP to verify the X# testbed sites. services/site_check

AcrossGrids Conference – Santiago 13 of February 2003 Testbed monitoring Mapcenter grid monitoring framework. Mapcenter was developed by DataGrid. The CrossGrid Mapcenter is maintained by LIP. Excellent tool to monitor the availability of sites and services

AcrossGrids Conference – Santiago 13 of February 2003 X# host check tool Host Check grid host checker. Host Check was developed to support the CrossGrid testbed deployment. Host Check produces a detailed report for each testbed CE and SE.

AcrossGrids Conference – Santiago 13 of February 2003 Production RB statistics –The peak usage of the RB was between last November and December. –Since the RB doesn’t support parallel jobs, most job submissions pass unnoticed to the RB. Total users33 Jobs submitted1943 Jobs accepted1904 Jobs with good match1799 Jobs submitted by JSS1781 Jobs run1620 Jobs done1070 Data from end of January

AcrossGrids Conference – Santiago 13 of February 2003 Validation RB statistics –The test and validation RB has been established recently. –The validation RB also doesn’t support parallel applications. 163 matching failures 3 not submitted 43 jobs aborted 219 jobs lost 94.8% success Data from end of January Total users8 Jobs submitted4173 Jobs accepted4173 Jobs with good match4010 Jobs submitted by JSS4007 Jobs run3964 Jobs done3954

AcrossGrids Conference – Santiago 13 of February 2003 Production CEs statistics SitesConnec tions PingsJobs OK Failed Jobs LCASCRL expJobmanGSS LIP IFIC Cyfronet II SAS FZK Demo ICM CESGA UAB INS PSNC TCD AUTH TOTAL

AcrossGrids Conference – Santiago 13 of February 2003 Validation CEs statistics –The validation testbed has been heavily exercised. –More than jobs have been submitted since the end of November. SitesConnec tions PingsJobs OK Failed Jobs LCASCRL expJobmanGSS LIP FZK Demo TOTAL

AcrossGrids Conference – Santiago 13 of February 2003 Collaborative tools –Travelling is expensive and wastes time. –Frequent meetings are required to coordinate the activities (2-3 per month). –VRVS was selected as the main tool for videoconferencing. Inexpensive (uses the Internet). Supports several platforms. Supports a wide range of AV equipment. – and discussion lists are also extremely used.

AcrossGrids Conference – Santiago 13 of February 2003 Test of X# applications –Application prototypes are being tested in the current testbeds. Prototype of a X# HEP application: Distributed training of a neural network. Tested by IFCA, LIP, Demokritos Requiring: MPICH-G2 Lowest latency possible (QoS will be important) MPI traffic across sites Other MPI applications following Air pollution modelling. Meteorological downscaling. Flooding control. … MPI test programs.

AcrossGrids Conference – Santiago 13 of February 2003 Test of X# applications (2) –The tests of the X# HEP application prototype using MPICH-G2 started in November. Skeleton of the application. The first application prototype. –Test were performed: Using dedicated systems with Globus (IFCA). Using the CrossGrid production testbed (LIP, Demokritos). –The tests over the testbed have shown that: Its possible to run MPI jobs in the testbed. MPI across sites with MPICH-G2 works. However problems were detected in sites using private IP addresses.

AcrossGrids Conference – Santiago 13 of February 2003 Test of X# applications (3) –It was possible to run the application using processors in up to seven sites simultaneously. –The application was compiled statically. –Both PBS and FORK job managers were used in the tests. –Issues: There isn’t support for parallel jobs in the RB (yet), matchmaking must be performed by the user. Check that the user is authorized at the testbed sites. Check that there are free CPUs available. PBS jobs may end up waiting in a queue. Sometimes processes don’t die when they should. Sometimes the execution hangs. Problems with invalid IP addresses. Possible problems with firewalls.

AcrossGrids Conference – Santiago 13 of February 2003 Integration –The first CrossGrid software integration started last week. –The main goals were achieved: Integration of tools. Integration of middleware components. Integration of application portals. –A demonstration of the middleware and applications was performed yesterday. –The integration work is not yet finished and will continue in the next days. –The first #X software release will be available in the coming weeks.

AcrossGrids Conference – Santiago 13 of February 2003 IST Demonstration –CrossGrid has participated in the World grid demonstration involving European and US sites from CrossGrid, DataGrid, GriPhyN and PPDG. –It took place in November –It was the largest grid testbed in the world. –Applications from the CERN/LHC experiments CMS and Atlas were used. –CrossGrid participated with 3 sites: LIP- Lisbon FZK- Karlsruhe IFIC- Valencia

AcrossGrids Conference – Santiago 13 of February 2003 Near future Integration of the first X# middleware release has started. Monitoring tools. Development tools. Migrating desktop. Remote access server. Portals. Parallel scheduler. Test and validation period. First production release. Deployment in the production testbed. The first X# middleware will use EDG and Globus 2.

AcrossGrids Conference – Santiago 13 of February 2003 Next 12 months –Support the extension of the testbed to new sites. More sites internal to the project. Possible external sites and users (policy needed). –Establish a development testbed. –Support clusters already running other Linux flavours. Light installation. –Prepare the test and possible migration to EDG 2.x and Linux 7.x (RH 6.2 is the current OS). –Study the usage of QoS in CrossGrid. Create a QoS test infrastructure. –Start the security group activities. Policies, guidelines, tracking of problems, patches. –Stress testing of the infrastructure.

AcrossGrids Conference – Santiago 13 of February 2003 END LIP IFIC IFCA FKZ