1. INFSO-RI-508833 Enabling Grids for E-sciencE www.eu-egee.org User and Virtual Organisation Support in EGEE Flavia Donno, CERN Torsten Antoni, FZK Alistair Mills, CERN Ian Bird, CERN

2. Enabling Grids for E-sciencE INFSO-RI-508833 F. Donno et al, EGEE User Forum, CERN, 01-03 March 2006 2 User and VO Support in Grid User Support in a Grid environment –Distributed nature of the Grid : experts located everywhere, sometimes in specific centers; spread of resources and services; different policies and lows. –Variety of users : beginners, system administrators, operators, network specialists, Virtual Organization communities –Variety of applications : high energy physics, biomedical, earth observation, astrophysics, computational chemistry, etc.

3. Enabling Grids for E-sciencE INFSO-RI-508833 F. Donno et al, EGEE User Forum, CERN, 01-03 March 2006 3 Expected services A single access point for support A portal with a well structured sources of information and updated documentation concerning the VO or the set of services involved; EIS portal for HEP VO VO technical groups pages CERN EDMS docs Wiki pages …

4. Enabling Grids for E-sciencE INFSO-RI-508833 F. Donno et al, EGEE User Forum, CERN, 01-03 March 2006 4 Expected services Experts knowledgeable of the particular application in use and who can even discuss with the user to better understand what he/she is trying to achieve (hot-line); help integrating user applications with the grid middleware; Correct, complete and responsive support; LCG-ROLLOUT@LISTSERV.CCLRC.AC.UK support-eis@cern.ch fts-support@cern.ch service-challenge-tech@cern.ch

5. Enabling Grids for E-sciencE INFSO-RI-508833 F. Donno et al, EGEE User Forum, CERN, 01-03 March 2006 5 Expected services Tools to help resolve problems (search engines, monitoring applications, resources status, etc.); Examples, templates, specific distributions for software of interest; JDL examplesgridftp distribution LFC usage patternVOBOX DLI implementation

6. Enabling Grids for E-sciencE INFSO-RI-508833 F. Donno et al, EGEE User Forum, CERN, 01-03 March 2006 6 Expected services Integrated interfaces with other Grid infrastructure support systems; Connection with the grid developers and the deployment and operation teams; Assistance during production use of the grid infrastructure. Number of jobs per day Data Challenge Rome Production

7. Enabling Grids for E-sciencE INFSO-RI-508833 F. Donno et al, EGEE User Forum, CERN, 01-03 March 2006 7 A possible answer Global Grid User Support is the support infrastructure for Grid users, deployment and operation problems It offers a great variety of services to satisfy user needs at all levels It does not substitute but integrates existing infrastructures and coordinates support efforts http://www.ggus.org

8. Enabling Grids for E-sciencE INFSO-RI-508833 F. Donno et al, EGEE User Forum, CERN, 01-03 March 2006 8 GGUS starts in 2003 with a prototype support system result of the discussion in the LHC Computing Grid Deployment Board The plan was to offer coverage 24x7 by 3 teams in different time zones GGUS was conceived to be a Single Point Of Contact Strictly hierarchical structure in LCG (tier model) Transition to EGEE meant migration to a different model: the federative approach very good choice 9 Regional Operation Centres instead of one Grid Operation Centre Different approach was needed in user support also A little history

9. Enabling Grids for E-sciencE INFSO-RI-508833 F. Donno et al, EGEE User Forum, CERN, 01-03 March 2006 9 The Support Model The support model in EGEE can be captioned “Regional Support with Central Coordination" The ROCs and VOs and the other project wide groups such as the Core Infrastructure Center (CIC), middleware groups (JRA), network groups (NA), service groups (SA) areCICJRANA connected via a central integration platform provided by GGUS. ….. … Central Application (GGUS) Deployment Support RC 1RC X Middleware Support Network Support Operations Support TPM BIOMEDESR DS 1 DS 5 … MS 1 MS 8 … ROC 1 ROC 12 ROC… RC 1RC X… VO Support ALICE RC 1RC X… Interface Webportal Other Grids

10. Enabling Grids for E-sciencE INFSO-RI-508833 F. Donno et al, EGEE User Forum, CERN, 01-03 March 2006 10 The GGUS System

11. Enabling Grids for E-sciencE INFSO-RI-508833 F. Donno et al, EGEE User Forum, CERN, 01-03 March 2006 11 Coordination: ESC Chaired by Flavia Donno/Torsten Antoni/Alistair Mills 27 January 2005 ( Kick off meeting of ESC at Karlsruhe - 27 January 2005) Goal: To ensure an effective, efficient, scalable Grid User Support Service. It coordinates operations, follows/cures infrastructure problems, takes users/supporters input. Members: people from the ROCs and GGUS-FZK, representatives from VOs, NA3, NA4, other Grids (OSG and NorduGrid), ESC meets monthly to discuss organization issues and problems.

12. Enabling Grids for E-sciencE INFSO-RI-508833 F. Donno et al, EGEE User Forum, CERN, 01-03 March 2006 12 Mail to - user- support@ggus.org Central Application (GGUS) VO Support Units Middleware Support Units Deployment Support Units Operations Support ROC Support Units Network Support VO-specific TPM Grid+VO experts - Solves - Classifies - Monitors Automatic Ticket Creation Support Workflow For VO users and VO specific problems Other Grids

13. Enabling Grids for E-sciencE INFSO-RI-508833 F. Donno et al, EGEE User Forum, CERN, 01-03 March 2006 13 Support Workflow Local Helpdesk Central Application (GGUS) Automatic Ticket Creation VO Support Units Middleware Support Units Deployment Support Units Operations Support ROC Support Units Network Support - Solves - Classifies - Monitors TPM Local Helpdesk Local Problem? RC SavannahRemedyOthers Other Grids

14. Enabling Grids for E-sciencE INFSO-RI-508833 F. Donno et al, EGEE User Forum, CERN, 01-03 March 2006 14 The GGUS Supporters TPMSupport VO Support Ticket Processing Managers (TPM) Ticket Processing Managers (TPM) : Generic grid experts VO TPMs VO TPMs: First line supporters for Vos Teams like EIS can be here Specialized Support Specialized Support: Middleware, Deployment (13) specialized VO Support (14) ROCs (12) ROCs (12): local support and services ENOC ENOC: network support

15. Enabling Grids for E-sciencE INFSO-RI-508833 F. Donno et al, EGEE User Forum, CERN, 01-03 March 2006 15 GGUS Portal: User Services Browseable tickets Search through solved tickets Useful links (Wiki FAQ) Latest News GGUS Search Engine Updated documentation (Wiki FAQ)

16. Enabling Grids for E-sciencE INFSO-RI-508833 F. Donno et al, EGEE User Forum, CERN, 01-03 March 2006 16 GGUS Search Engine GGUSSearchEngine Ongoing work to make it faster and to search through a widerset of docs and DBs

17. Enabling Grids for E-sciencE INFSO-RI-508833 F. Donno et al, EGEE User Forum, CERN, 01-03 March 2006 17 Ticket Search

18. Enabling Grids for E-sciencE INFSO-RI-508833 F. Donno et al, EGEE User Forum, CERN, 01-03 March 2006 18 Documentation

19. Enabling Grids for E-sciencE INFSO-RI-508833 F. Donno et al, EGEE User Forum, CERN, 01-03 March 2006 19 Wiki page compilation

20. Enabling Grids for E-sciencE INFSO-RI-508833 F. Donno et al, EGEE User Forum, CERN, 01-03 March 2006 20 Supporters’ Broadcast Tools

21. Enabling Grids for E-sciencE INFSO-RI-508833 F. Donno et al, EGEE User Forum, CERN, 01-03 March 2006 21 VRVS Chat and hot-lines Hot-line

22. Enabling Grids for E-sciencE INFSO-RI-508833 F. Donno et al, EGEE User Forum, CERN, 01-03 March 2006 22 CIC Portal

23. Enabling Grids for E-sciencE INFSO-RI-508833 F. Donno et al, EGEE User Forum, CERN, 01-03 March 2006 23 CIC Portal FZK, Karlsruhe, GermanyIN2P3-CC, Lyon, France CIC PORTAL CIC-on-duty dashboard UK FRGERIT … Regional Support Units Operator on duty - Create() - Set(ticket) SOAP -Get(ticket) - Get_all() Ticket Central Helpdesk CIC Helpdesk WSDL GGUS Problem detection & reporting Ticket follow-up GGUS Interface

24. Enabling Grids for E-sciencE INFSO-RI-508833 F. Donno et al, EGEE User Forum, CERN, 01-03 March 2006 24 Some Statistics September October A peak of 200 tickets per day has been reached >>The system can handle at least 1400 tickets a week. November 2005: 315 tickets

25. Enabling Grids for E-sciencE INFSO-RI-508833 F. Donno et al, EGEE User Forum, CERN, 01-03 March 2006 25 Some Statistics: VO Usage

26. Enabling Grids for E-sciencE INFSO-RI-508833 F. Donno et al, EGEE User Forum, CERN, 01-03 March 2006 26 Meeting User Needs  GGUS provides a single entry point for reporting problems and dealing with the grid.  In collaboration with the EGEE EIS team, the EGEE User Information Group, NA3, and the entire EGEE infrastructure, GGUS offers a portal where users can find up-to-date documentation, and powerful search engines to find answers to resolved problems and examples.  Common solutions are stored in the GGUS knowledge database and Wiki pages are compiled for frequent or undocumented problems/features.  GGUS offers hot lines for users and supporters and a VRVS chat room to make the entire support infrastructure available on-line to users.  Special tools and grid middleware distributions are made available by the NA4/EIS team for GGUS users.  GGUS is interfaced with other grids’ support infrastructures such as in the case of OSG and NorduGrid.  GGUS is used for daily operations to monitor the grid and keep it healthy. Specific user problems can be directly communicated to the Grid Operation Centers and broadcasted to the entire grid community.  GGUS is used also to follow and track down problems during stress testing activities such as the HEP experiments production data challenges and the service challenges.

27. Enabling Grids for E-sciencE INFSO-RI-508833 F. Donno et al, EGEE User Forum, CERN, 01-03 March 2006 27 Open Issues  Small VOs do not have the resources to implement their part of the model. GGUS still provides support for the VO at generic Grid level.  Supporters are not dedicated to GGUS. Some times it is difficult to ensure responsiveness. Supporters are concentrated in a few locations. VO Support tends to exist in only a small number of locations, with a small number of supporters. Scalability is constrained by the availability of supporters. As the VOs become larger this will become a constraint to growth unless more supporters are found. Limited experience in handling a large number of tickets. At the moment GGUS has successfully handled 200 tickets per day. Limited engagement of existing VOs in the implementation of GGUS. The present VOs have found it difficult to provide people for involvement with this work. However more and more people are getting aware of the existence and good service provided by GGUS.

28. Enabling Grids for E-sciencE INFSO-RI-508833 F. Donno et al, EGEE User Forum, CERN, 01-03 March 2006 28 Conclusions  The GGUS system is now ready for duty.  Many more support units have been added in preparation for the next LCG Service Challenge.  During 2006, it is expected that there will be a large number of tickets passing through the system as the LHC VOs move from preparing for service to being in production.  It is also expected that the number of Virtual Organisations will grow as the work of EGEE-II proceeds.