1 All-Hands Meeting 2-4 th Sept 2003 e-Science Centre The Data Portal Glen Drinkwater

2 All-Hands Meeting 2-4 th Sept 2003 e-Science Centre Motivation Council for the Central Laboratory of the Research Councils‘ has many large scale facilities. E.g. ISIS, a pulsed neutron source and Diamond, the UK’s new synchrotron light source. Data Portal Making existing scientific data resources accessible through a single interface. Acting as a broker between scientists, facilities and data. Benefits Repetition of experiments can be avoided. Collaborations can be built by identifying that someone else is working in a similar area. Data about a related material can be found and used to aid a new analysis. Data can be reanalysed when better analysis tools becomes available

3 All-Hands Meeting 2-4 th Sept 2003 e-Science Centre General Architecture Local data Local metadata Facility 1 Local data Local metadata Facility 1 Local data Local metadata Facility 1 Local data Local metadata Facility N Local data Local metadata Facility 6 Local data Local metadata Facility 4 Local data Local metadata Facility 5 User

4 All-Hands Meeting 2-4 th Sept 2003 e-Science Centre General Architecture Local data Local metadata Facility 2 Local data Local metadata Facility N Local data Local metadata Facility 1 CCLRC Data PortalOther Data Portal instances Xml Wrapper

5 All-Hands Meeting 2-4 th Sept 2003 e-Science Centre Core Modules Web Interface, Query and Reply, Lookup and Help. Important function grouped into modules, each modules with a web service interface, interface description in WSDL and communicate via SOAP.

6 All-Hands Meeting 2-4 th Sept 2003 e-Science Centre Additional Modules Access Control, Authentication and Authorisation, Data Transfer, Shopping Cart, User Administration, Facility Administration and Accounting. Functions grouped into modules, each with web services interface, interface description in WSDL, communication via soap. These Modules could be shared or exchanged with their own implementation.

7 All-Hands Meeting 2-4 th Sept 2003 e-Science Centre External Services XML-Wrapper, RasDaMan Insertion and Extraction, Specialised Search, HPCPortal, Visualisation Portal, SRB, other DataPortal Instances. Other services that are linked with the DataPortal, but are not integral part of it. Registered with the Portal and accessible via web services interface.

8 All-Hands Meeting 2-4 th Sept 2003 e-Science Centre Authentication Session Manager Access & Control FACILITY B Web Interface Authentication USER 1: get certificate(name, passphrase) Access & Control FACILITY A 2: getUserPrivileges (certificate) 3: startSession(certificate, permissions) OUTSIDE SERVICE START: Login(name, passphrase) 4: return SID START: Login(name, passphrase, lifetime) FACILITY A ACCESS & CONTROL DATABASE FACILITY A ACCESS & CONTROL DATABASE SESSION MANAGER DATABASE Set permissions in database for all facilities for duration of session Get permissions from database for facility A Session Manager 5: getPermissions(SID) MyProxy

9 All-Hands Meeting 2-4 th Sept 2003 e-Science Centre Basic Search Session Manager XML Wrapper FACILITY B Web Interface Query & Reply User OUTSIDE SERVICE START: basic query (list of facilities, discipline) XML Wrapper FACILITY A 2: getXML (discipline) FACILITY A metadata archive FACILITY B metadata archive SQL selects FINISH: XML results in CLRC scientific metadata format 1: getPermissions(SID) 2: getXML (discipline) FINISH: XML results in CLRC scientific metadata format

10 All-Hands Meeting 2-4 th Sept 2003 e-Science Centre Shopping Cart Web Interface Shopping Cart USER START: Add item to Shopping Cart SHOPPING CART database 1: addToCart(SID, item) FINISH: success/fail Session Manager 2: getCert(SID) 3: add item to database (DN, item) 4: success/fail

11 All-Hands Meeting 2-4 th Sept 2003 e-Science Centre HPC Portal Another e-Science Centre project to develop a Web portal to search for resources and submit HPC applications to a computational Grid. Uses Globus toolkit v2.2 Functionalities include: Resource Management: GRAM. Information Services: MDS. Data Management: GridFTP and GASS. All use GSI security protocol as the connection layer.

12 All-Hands Meeting 2-4 th Sept 2003 e-Science Centre Integrated Portals Visualisation DataPortal HPCPortal HPC Systems Data Systems Web Services Working with GGF Grid Computing Environments Research Group GridFTP GSI Globus

13 All-Hands Meeting 2-4 th Sept 2003 e-Science Centre Signal Sign on How do you have signal sign on? Both HPC and DataPortal have their own Session Managers which rely on Globus Proxy Certificates. Integrated session managers communicate over SSL using mutual authentication between the web servers. Allows certificates to be a transferred between portals allowing single sign on. The certificate can then be used for GSI authentication.

14 All-Hands Meeting 2-4 th Sept 2003 e-Science Centre Single Sign on DataPortalHPC Portal USER 1: Login(username,password,lifetime) Via Authentication Module Dataportal Session Manager 5: RequestCert(SID) 7: YES! 4: isValid(SID) 2: Dataportal SID 3: LoginHPC(SID) HPC Session Manager 6: Certificate 8: HPC Session id FINISH: User is sent to HPC front page to use its services START: Log on to DataPortal then to HPC Portal

15 All-Hands Meeting 2-4 th Sept 2003 e-Science Centre Scenario User logs on to Data Portal and searches for data. The data found is added to the persistent shopping cart. The user could then transfer the data to another machine using GSI FTP, either using the Data Portal or the HPC Portal. Using single sign on, the user could then go directly to the HPC, and the use a remote job run on the data that they have just transferred and using GSIFTP, transfer the results back to their machine for analysis.

16 All-Hands Meeting 2-4 th Sept 2003 e-Science Centre Current Work Converting to a new Xml Wrapper architecture Problems: DTD limits us to one investigation per study. Any new wrappers should be written using the new standard. Only allows one investigation per study (as using the old format). XSLT time consuming to write and must operate on the whole study. Need to transfer the whole studys – this can be large. Data Archive goes down we cannot access study even though it is cached.

17 All-Hands Meeting 2-4 th Sept 2003 e-Science Centre Old Xml Wrapper Architecture DataPortal Xmlwrapper cache Data Archive SQL ‘like’ where clause Studies in XML Csmd v1 format SQL statements and replies

18 All-Hands Meeting 2-4 th Sept 2003 e-Science Centre Solution DataPortal (via Q&R) XMLwrapper: Doc Selector XMLwrapper: Doc Builder Cache XML Document Repository Data Archive Xml Wrapper Framework XQuery XQuery & Result Returned Update Repository with New and changed CSMD XML docs SQL Selects and Results Mapping between Archive schema And CSMD format

19 All-Hands Meeting 2-4 th Sept 2003 e-Science Centre Advantages Xml Wrapper still operates if archive is down. Concurrency much better supported as selecting from the archive is essentially one thread. Only relevant data sent between xml wrapper and Data Portal – also XQuery can include formatting rules which can potentially do away with the need to XSLT.

20 All-Hands Meeting 2-4 th Sept 2003 e-Science Centre Questions?