1. Polish Infrastructure for Supporting Computational Science in the European Research Space Recent Advancements of National Grid Infrastructure in Poland ( PL-Grid ) A CC C yfronet AGH, Krakow, Poland Cracow Grid Workshop – CGW’10 11-13.10.2010, Krakow, Poland www.plgrid.pl/en Jacek Kitowski

2. 2 Acknowledgements  ACC Cyfronet AGH  Kazimierz Wiatr  Marian Bubak  Łukasz Dutka  Karol Krawentek  Andrzej Oziębło  Maciej Malawski  Zofia Mosurska  Robert Pająk  Marcin Radecki  Renata Słota  Mariusz Sterzel  Tomasz Szepieniec  Agnieszka Szymańska  Teresa Ozga  ICM  Piotr Bała  Maciej Filocha  PCSS  Norbert Meyer  Bartek Palak  Krzysztof Kurowski  Tomasz Piontek  Dawid Szejnfeld  Paweł Wolniewicz  WCSS  Jerzy Janyszek  Bartłomiej Balcerek  TASK  Rafał Tylman  Mścislaw Nakonieczny

3. 3 Outline  General Introduction to PL-Grid Project  Motivation, consortium and objectives  PL-Grid and EGI  Organization of the project  Current Status and Advancements  Status of hardware infrastructure  Operational Centre – tasks and services  e-Infrastructure software, software packages and user’s tools  Training and user support  Security aspects  Conclusions – Future work

4. 4 PL-Grid Consortium  Consortium Creation – agreement signed January 2007  a response to requirements of Polish scientists and due to ongoing Grid activities in Europe within the framework of EGI_DS.  Consortium members – make up of 5 Polish supercomputing and networking centres:  Academic Computer Centre CYFRONET AGH, Krakow (coordinator)  Interdisciplinary Centre for Mathematical and Computational Modelling, Warsaw University  Poznan Supercomputing and Networking Centre  Academic Computer Centre, Gdansk  Wroclaw Centre for Networking and Supercomputing  PL-Grid Project proposal got funded by the European Regional Development Fund as part of the Innovative Economy Program on March 2, 2009.  Result: First working NGI in Europe in the framework of EGI.eu

5. The Consortium consists of five High Performance Computing Polish Centres representing Communities, coordinated by ACC Cyfronet  Participation in international and national projects  35+ international projects FP5, FP6, FP7 on Grids (~50% common) (~45 currently, including EDA)  15+ Polish projects (~50% common)  Polish scientific communities  ~75% highly rated Polish publications in 5 Communities  Computational resources  Top500 list  European/Worldwide integration Activities  EGEE I-III, EGI_DS, EGI, e-IRG, PRACE, DEISA, OMII, EU Unit F3 „Research Infrastructure” Experts, Representatives in EGI Council, Members of EGI Executive Board  National Network Infrastructure available  Pionier National Project 5 Rationales behind PL-Grid Consortium RankSiteSystemCoresR max (TFlops)R peak (TFlops) 161 Cyfronet AGH KrakowCluster Platform 3000 BL2x220, L54xx 2.5 Ghz, Infiniband / 2010 Hewlett-Packard 614439.9355.54 181 Gdansk University of Technology, CI Task ACTION Cluster Xeon E5345 Infiniband / 2008, Action 533638.1749.73 444 PCSS PoznanCluster Platform 3000 BL 2x220, E5530 2.4 GHz, Infiniband GDR / 2010, Hewlett-Packard 345626.2231.24 TOP500 – June 2010

6. 6 PL-Grid Project - Basic Data  The Project is co-funded by the European Regional Development Fund as part of the Innovative Economy Program.  Total budget: 83 M PLN (~ 21 M EUR) Personel cost 27 M PLN (~7 M EUR) Equipment cost 33 M PLN (~8 M EUR) Other cost 23 M PLN (~6 M EUR)  Funding from the EC: 68 M PLN (~ 17 M EUR)  Project duration: 01 January 2009 – 31 December 2011  Beneficiary: Academic Computer Centre Cyfronet AGH, Krakow, Poland  Contract number: POIG.02.03.00-00-007/08  Project website: www.plgrid.pl/en

7. 7 Grid infrastructure (Grid services) PL-Grid Application Clusters High Performance ComputersData repositories National Computer Network PIONIER Domain Grid Advanced Service Platforms Domain Grid Main Objectives of PL-Grid  Polish Grid is developing a common base infrastructure – compatible and interoperable with European and Worldwide Grids  Capacity to construct specialized, domain Grid systems – including services and tools focused on specific types of applications  This approach should enable efficient use of available financial resources  Plans for HPC and Scalability Computing enabled with more focus on domain specific Grids Offer for the Users  Computing Power 215 Tflops  Storage 2500 TBytes  Support from PL-Grid staff on using advanced Grid tools  Support on porting legacy codes to Grid environment  Support on designing applications for PL-Grid environment

8. 8 PL-Grid Building Blocks PL-Grid software comprises:  user tools (portals, systems for applications management and monitoring, result visualization and other purposes, compatible with the lower-layer software used in PL-Grid)  software libraries  virtual organization systems: certificates, accounting, security, dynamic  data management systems: metadata catalogues, replica management, file transfer  resource management systems: job management, applications, grid services and infrastructure monitoring, license management, local resource management, monitoring Users National computer network Grid Application Programming Interface Virtual organizations and security systems Basic Grid services Grid services LCG/gLite (EGEE) UNICORE (DEISA) Other Grids systems Grid resources Distributed computational resources Grid portals, development tools Distributed data repositories Three Grid structures are maintained:  production  research  development / testing

9. 9  EGI.eu – European organization being developed to coordinate the European Grid Infrastructure, based on the federation of individual National Grid Initiatives (NGI), to support a multi-disciplinary user community.  PL-Grid tasks in EGI (planned/fixed)  Grid operation and oversight of the e-Infrastructure  Coordination of resource allocation and of brokering support for VOs from NGIs  Computational Chemistry – organization and management of Computational Chemistry and Material Science and Technology Specialized Support Centre and EGI liaisons  Development of unified middleware via European Middleware Initiative PL-Grid – first operational NGI in Europe Integration Activity in the framework of European Grid Infrastructure  Scientific application porting, especially concerning UNICORE architecture, within Application Porting SSC PL-Grid On 30 March 2010, Poland - as the first country in Europe - has initiated functioning of the National Grid Initiative (NGI)

10. 10 Organization of the PL-Grid project PLANNING AND DEVELOPMENT OF INFRASTRUCTURE P2 Coordination Strategic PlanningDissemination PROJECT MANAGEMENT P1 SECURITY CENTER P6 Training SUPPORT FOR VARIOUS DOMAIN GRIDS P5 P4 GRID SOFTWARE AND USERS TOOLS DEVELOPMENT EGEEDEISA …. OPERATIONS CENTER P3

11. 11 Partners’ Computing Resources TOP500 – June 2010 RankSiteSystemCoresR max (TFlops)R peak (TFlops) 161 Cyfronet AGH Krakow Cluster Platform 3000 BL2x220, L54xx 2.5 Ghz, Infiniband / 2010 Hewlett-Packard 614439.9355.54 181 Gdansk University of Technology, CI Task ACTION Cluster Xeon E5345 Infiniband / 2008 ACTION 533638.1749.73 444 PCSS Poznan Cluster Platform 3000 BL 2x220, E5530 2.4 GHz, Infiniband GDR / 2010 Hewlett-Packard 345626.2231.24

12. Current Status and Advancements 12

13. 13 Status of Hardware Infrastructure Computational power  CYFRONET: 29,3 Tflops  ICM: 13,1 Tflops  PCSS: 23,2 Tflops Disk storage  CYFRONET: 313 TBytes  ICM: 433 TBytes  PCSS: 300 TBytes Total: 65,6 Tflops Total: 1046 TBytes Plans until the end of 2010: Plans until the end of the Project (end of 2011): 185 Tflops 215 Tflops 1900 TBytes 2500 TBytes Current status (October 2010):

14. 14 Operational Center – just in operation Tasks  Coordination of operations  Management and accounting  Collaboration with EGI and PRACE/DEISA  Users’ requirements analysis for operational issues  Running infrastructure for:  Production  Developers  Research  Future consideration:  Computational Cloud  Data Cloud  Internal and External Clouds  Virtualization aspects

15. 15 Services of the Operational Center for Users  Operational Center aims at facilitating access to the infrastructure by simplifying the procedures and deployment of useful tools:  System of registration of account management of the PL-Grid user available at https://konto.plgrid.pl/ required entry in the Polish database of „People of Science” or confirmation of the scientific tutor grid access to PL-Grid resources  5 centers – gLite  1 center – UNICORE local access to the queue system  „zeus” cluster in ACC CYFRONET AGH ability of application for a grid certificate on-line application for access to computational services in other centers  Helpdesk system in PL-Grid enables reporting and tracking issues available at https://helpdesk.plgrid.pl (for registered users)https://helpdesk.plgrid.pl access also by e-mail: helpdesk@plgrid.pl manual: https://wiki.plgrid.pl/doku.php?id=pakiet5:publiczne:podrecznik_uzytkownika_pl-grid → System Pomocy Helpdesk

16. 16 Services of the Operational Center for Users (cont’d)  Operational Center cares for a proper functioning of the infrastructure for PL-Grid users by pro-active monitoring of the following infrastructure elements:  availability of the infrastructure services  software packages supported by PL-Grid  Provision of the conformity of the PL-Grid and European (EGI) infrastructures  software  operational procedures  security procedures  Advanced work on the „PL-Grid grants” idea  Integration of the data presentation concerning resources usage for a user  Work on provision of the integrated user portal

17. 17 Development of the e-Infrastructure Software and the User’s Tools  Close cooperation of 8 programming and testing groups, about 20 people  Installation and provision for testing purposes gLite, Unicore and QosCosGrid  About 30 various configurations of virtual machines with installed software used for development and testing of the tools for users – the choice of the technology made  Functional, conformity and efficiency tests of selected packages of the research software made in order to perform the deployment and support of the new tools and services on the production level

18. 18 Development of the e-Infrastructure Software and the User’s Tools (cont’d)  Direct contact with new users on the basis of a survey, available at: www.plgrid.pl/ankieta  Requirements of the Polish users (results ~190 surveys) considered in the new applications, tools and services developed and tested in the framework of the Package 4  Large group of users cooperating with the software and tools team  Department of Chemistry of the Jagiellonian University  Department of Bioinformatics and Telemedicine of the Collegium Medicum of the Jagiellonian University  University of Adam Mickiewicz  Poznan Technical University  Wrocław Technical University  Administrators of the computing centers  …

19. 19 Development of the e-Infrastructure Software and the User’s Tools (cont’d)  Test versions of tools for users and systems administrators: Grid Resource Bazaar, mobile access to the infrastructure, new security modules  Extending of the GridSpace2 platform with a set of new functions, support for new scripting languages and integration with new grid services  Integration of the Migrating Desktop and gEclipse tools with various PL-Grid middleware services  Extension and deployment of FiVO – a new tool for VO management and monitoring  Performance and functional tests of middleware service QosCosGrid and integration with gLite and Unicore infrastructure at the queue systems level  Implementation and provision of the advanced graphical interfaces, visualization and tasks and data management for selected user applications via the Vine Toolkit  Integration of the selected tools and web applications with Liferay portal framework and Nagios monitoring system

20. 20 Grid Resource Bazaar  Providing resources to users with required qualities of services  Required = specified in Service Level Agreement RU – Resource Users RP – Resource Providers

21. 21 Grid Resource Bazaar (cont’d) Resource Allocation-related Operation Model

22. 22 http://grid.cyfronet.pl/bazaar Grid Resource Bazaar (cont’d)

23. 23 GridSpace2 Experiment Workbench in the PL-Grid Virtual Laboratory  Use of distributed computational resources and data repositories  High-level tools offered for the user for in-silico experiments

24. 24 GridSpace2 - working with the Experiment Workbench The PL-Grid Virtual Laboratory provides an Experiment Workbench – a Web 2.0 interface supporting the development and execution of in silico experiments. Working with the Experiment Workbench:  Open the workbench in your browser (https://wl.plgrid.pl)https://wl.plgrid.pl  Log into one of the available servers with your PL- Grid account  Start your usual work. Your files are already there. Your code snippets may play the role of scripts (in Python, Perl, Ruby etc.), bash commands or input to external applications (such as Gaussian or Gnuplot);  All the files you generate may be viewed with visualization tools (such as Jmol)  Save your work – the experiment is now visible among your other files  Share the experiment with other members of your research team

25. 25 GridSpace2 - sample experiment in ViroLab environment  Patient’s data  Medical examination  HIV genetic sequence put into database  Experiment in-silico  Collect HIV genetic sequences from database  Perform sequence matching  Calculate virus resistance http://gs2.cyfronet.pl/ http://www.virolab.org

26. 26 Migrating Desktop Platform – the Framework for Grid Applications Main product features:  Simple integration with grid applications  Easy job defining, submission, monitoring and visualization of results  Support for batch and interactive jobs  Handling of sequential and parallel applications  Intuitive management of grid data  Easy extendable framework Migrating Desktop main window Description:  The Migrating Desktop Platform is a powerful and flexible user interface to Grid resources that gives a transparent user work environment and easy access to resources and network file systems independently of the system version and hardware  It allows the user to run applications and tools, manage data files, and store personal settings independently of the location or the terminal

27. 27 Migrating Desktop Platform – the Framework for Grid Applications (cont’d) Application support:  The key feature: the possibility of easy adding various tools, applications and supporting different visualization formats  The Migrating Desktop offers a framework that can be easily extended on the basis of a set of well ‑ defined plug-ins used for: accessing data, defining job parameters, pre-processing job parameters, and visualization of job results  Open architecture of the Migrating Desktop speeds up the application integration process and makes that product significantly more flexible than specialized tools (e.g. portals) designed only for a specific application Job defining and visualization of results (examples of Migrating Desktop plug-ins)

28. 28 g-Eclipse – to Access the Power of the Grid Benefits from using g-Eclipse:  Grid application users will benefit from the desktop-like access to Grid resources  Grid operators and resource providers will be able to reduce the time-to-service by using the Grid management and Grid site configuration tools  Grid application developers will reduce the time- to-market for new Grid applications by accelerating the development and deployment cycle Features:  The g-Eclipse framework provides tools to customize Grid users’ applications, to manage remote resources and to support the development cycle of new Grid applications  The framework consists of general Grid workbench tools that can be extended for many different Grid and Cloud middleware’s (such as gLite, UNICORE, Globus Toolkit, Amazon Cloud)

29. 29 g-Eclipse – to Access the Power of the Grid (cont’d) g-Eclipse elements (functionalities):  Grid Virtual Organisation management allows the dynamic creation and management of Virtual Organisations and their resources  Grid job management supports Grid users and developers in creating and managing Grid jobs independent of the middleware  Grid file management provides access and management of local and foreign resources which are seamlessly integrated into the Eclipse Workbench  Grid application deployment supports Grid application developers and Grid application users with the deployment of their applications on the Grid g-Eclipse provides API and User Interface modules for all above functionalities.  Grid visualisation tools offer functionalities that allow the visualisation of scientific and numerical calculations  Grid workflow builder supports the creation and management of middleware independent workflows by use of a provided graphical editor  Grid command console is a command line interface to access the Grid

30. 30 FiVO/QStorMan toolkit – the goals Provisioning PL-Grid users an appropriate quality of access to storage systems with the following means:  Definition of the non-functional requirements for the storage systems, e.g. expected transfer rate of the read/write operations or parameters related to expected availability level of the resource or security.  Monitoring of the storage systems parameters, e.g. a distributed file system.  Data localization management based on the provided requirements and the current state of an environment. QStorMan is a part of the Framework for Inteligent Virtual Organization (FiVO)

31. 31 FiVO/QStorMan toolkit - architecture and components Components of the toolkit:  Portal – a graphical user interface for defining requirements for storage systems  GOM (Grid Organizational Memory) – a knowledge base for storing semantic description of the storage systems along with defined requirements  SMED – a knowledge-supported monitoring system  SES (Storage Element Selection) library – a programming library for selecting storage resource based on provided requirements and current state of the environment SE – Storage Element

32. 32 QCG – QosCosGrid Middleware Description:  The QosCosGrid (QCG) middleware is an integrated e-infrastructure offering advanced job and resource management capabilities to deliver to end-users supercomputer-like performance and structure  By connecting many computing clusters together, QosCosGrid offers easy-to- use mapping, execution and monitoring capabilities for variety of applications, such as parameter sweep, workflows, MPI or hybrid MPI- OpenMP QCG architecture

33. 33 QCG – QosCosGrid Middleware (cont’d) Benefits:  for End Users – efficient and secure access to dynamically discovered computational resources located in the PL-Grid environment  for Cluster and Grid Administrators – a great opportunity to share and use their clusters more efficiently by scheduling local and Grid end users’ applications according to defined policies and constraints Several portal based and mobile applications have been developed and integrated with QCG infrastructure to give the users intuitive - experiment and domain oriented - access to resources. QCG mobile client

34. 34 Vine Toolkit – Distributed Resources at Your Fingertips for Developers and End Users Main features:  Single Sign-On – users log in once and use all resources  Support for different grid infrastructures  Advanced web applications (integration with Adobe Flex / BazeDS technology)  Extensible, module based open architecture based on plugins  Many applications included „out-of-the-box” such as: Job Manager, File Manager, Resource Manager, Credential Manager, Certificate Manager, GSI-SSHTerm applet  Support for scientific applications: ABINIT, Quantum Espresso (and others…)  Integration with portal environments Liferay and GridSphere installed on Tomcat server General description:  Vine Toolkit was designed as an environment to facilitate the development and integration of web-based applications with HPC resources, Grid services and various existing large-scale computing infrastructures managed by Grid middleware, such as gLite, Unicore, Globus, QosCosGrid and GRIA.  It is possible to create a computational web-based Science Gateway using a modular structure and existing Vine components.

35. 35 Vine Toolkit – Distributed Resources at Your Fingertips for Developers and End Users (cont’d) For whom:  web applications developers – a base for building advanced computational scientific portal gateways and applications with many ready to use components, integration layer for portal frameworks and advanced installation engine  administrators of grid infrastructures – possibility of deployment of a web portal “out-of-the-box” to access existing grid infrastructure  end users – many ready to use web applications for grid computations and data management in grid infrastructure  scientists – dedicated web applications for scientific applications for different scientific domains Example of the advanced web interface in the Nano-Science Gateway based on Vine Toolkit

36. 36 Scientific software packages  Access to software packages is provided to users through:  gLite  UNICORE  Examples of available packages in various fields:  biology: AutoDock, BLAST, ClustalW2, CPMD, Gromacs, NAMD  quantum chemistry: ADF, CFOUR, Dalton, GAMESS, Gaussian, Molcas, Molpro, MOPAC, NWChem, OpenBabel, Siesta, TURBOMOLE  physics: ANSYS FLUENT, Meep  numerical computations and simulation: Mathematica, MATLAB  other: Blender, POV-Ray  Users may report us their expectations through a survey available at: http://www.plgrid.pl/ankieta  The system of testing the software packages in the grid environment has been prepared and deployed  The correctness of functioning of the packages is monitored automatically in each of the centers

37. 37 Training and Users’ Support  Basic training on access to the PL-Grid infrastructure through gLite and UNICORE conducted in all centers participating in the project – in Gdańsk / Kraków / Poznań / Warszawa / Wrocław  More advanced training started  Similar (free) training may be conducted in other centers, if necessary  eLearning courses offered through the Blackboard system ( available for registered users of the PL-Grid infrastructure)  Helpdesk system implemented  it’s a novel support system for people using the Project resources  it involves the technical support and organization of the current users’ support by the experts (maintenance of trouble tickets)  tickets in Helpdesk may be created by sending an email to: helpdesk@plgrid.pl or through the online system, available at: https://helpdesk.plgrid.pl

38. 38 Security in PL-Grid  Provision of two CAs – PKI certification centers – for grid users  Project and implementation of the SimpleCA system, facilitating the users obtaining PKI certificates and their usage  Project and implementation of the secure configuration of the infrastructure, in conformity with the most actual security standards  Project of the system monitoring the conformance of the configuration deployed in the centers with the security policy  Creation of the group of experts from the field of security, in order to continuously monitor the environment, immediate react on incidents and support users and administrators  Prototype version of the system of correlation of information about the attacks on the infrastructure (ACARM-ng)  Audits of applications crucial for grid security

39. Conclusions  Good and promising partial results of the PL-Grid Project  Justified expectations to Project completion according to its goals and requirements  Generic services and generic applications in development  Futher developement needed, as identified currently, mainly on Domain Specific Grids  Request from the users’ communities  Capacity for organization of future development according to  Expertise and experience  Strong scientific potential of the users’ communities being represented by PL-Grid Partners  Wide international cooperation concerning the Consortium and individual Partners, good recognition worldwide  Good managerial capacity 39

40. 40 http://www.plgrid.pl/en