1. OGSA-DAI Neil Chue Hong 29 th January 2007 OGF19, Chapel Hill

2. Overview l What is OGSA-DAI l What you can do with OGSA-DAI l What’s new in OGSA-DAI l Where you can get more information This is an overview – please get in touch if you have more questions N.ChueHong@epcc.ed.ac.uk

3. Data Integration Challenges Diversity Scale Ownership Security of data resource types, vendors, middleware, schema, metadata of collections, formats, geographical, political and social distance on individual, group, and organisation levels; intersecting yet independent for client, service and data owner; at many levels, with many tradeoffs

4. Use Cases for Data Services l Data Filtering: u Single source producing large amounts of data distributed to many sites downstream l Data Discovery: u many sources, many query entry points in a linked system l Data Translation: u source to sink, conversion of data model / structure l Data Federation: u many sources, linked to provide view as a single source l Data Replication u full or partial copies to improve throughput l Data Integration (model aggregation) u e.g. integration of time variant data, streams, files l Data Integration (knowledge expansion) u forming links between databases to increase knowledge

5. Data Service Spec Goals Make access transparent Make integration easy Make management simple Impose standard interfaces to:

6. Extensible Portable Easy to develop We provide the generic You develop the specific Diverse, independently curated data sources

7. Core features of OGSA-DAI l Data access, insert and update u Relational: MySQL, Oracle, DB2, SQL Server, Postgres u XML: eXist u Files – CSV, BinX, EMBL, OMIM, SWISSPROT,… l Data delivery u SOAP over HTTP u FTP; GridFTP u E-mail u Inter-service l Metadata extraction l Data transformation u XSLT u ZIP; GZIP u Projections l Security u X.509 certificate based security l Multi OS support u Java 1.4/1.5 based l Client API l Documentation/ Tutorials

8. Making data accessible Images from UNIDART and ConvertGRID projects Bringing together PUBLIC and PRIVATE data

9. Demographic forecasting Census DB Borders DB WFS JDBC OGSA-DAI SQL WFS GLS Join Feature Portrayal GLS Portal Map Server Receive ticket for results Retrieve annotated image Store image on server Send parameterised query FPS Call out to existing FP service Cache attributes Stream polygons Request attributes Request features Run algorithm Stream relevant annotated polygons Concentrate on algorithm Reuse generic functionality Utilise existing services Efficient delivery methods

10. OGSA-DAI workflows l Data-driven workflows to solve problems u Obtain scan data for scans since date d of embryos in stage s showing expression of gene g.

11. Other common patterns l Robust vertical integration (eDIAMOND, GeneAssociationAnalysis) u the more results you can get, the better confidence interval you have l Content-based retrieval (AHDS) u queries are based on performing some computation/querying within the initial result set, e.g. for images, videos

12. OGSA-DAI 3.0 l Top to bottom rewrite l New service and resource model l APIs to write new web service layers l Persistence module l New activity framework u new input and output types u invocation u iteration l Released Q2 2007

13. OD3: What does this mean? l You can: u Chain OGSA-DAI services together to create powerful data-driven workflows. u Create workflows that integrate and transform data from multiple data resources, including accessing multiple data resources from within the scope of a single OGSA-DAI request. u "Reskin" OGSA-DAI with application-specific presentation layers to fit particular domains (e.g. DAIS, OGC, etc). u Develop application-specific activities easily and without resorting to XML manipulation.

14. Distributed Query Processing l Higher level services building on OGSA- DAI l Queries mapped to algebraic expressions for evaluation l Parallelism represented by partitioning queries u Use exchange operators table_scan (protein) table_scan termID=S92 (proteinTerm) reduce hash_join (proteinId) op_call (Blast) reduce exchange 3,4 12

15. DQP architecture

16. MDS/GridFTP/GSI Integration l Can publish any OGSA-DAI resource property to a local MDS Index Service u e.g. databaseSchema, activityTypes u information published is on a per-resource basis, and can differ for each resource l Can transfer results via GridFTP rather than via SOAP l Can use X509 certificates to secure services u new fine grained security model in preparation

17. Further information l See more projects using OGSA-DAI: u http://www.ogsadai.org.uk/about/projects.php l And what they’ve been doing: u http://www.ogsadai.org.uk/about/success_stories/ l Learn to program OGSA-DAI: u http://www.ogsadai.org.uk/documentation/ogsadai- wsrf-2.2/doc/clients/clienttoolkit/index.html l See what’s coming up in OGSA_DAI 3.0: u http://www.ogsadai.org.uk/documentation/Design_d ocuments/

18. Further information l The OGSA-DAI Project Site: u http://www.ogsadai.org.uk l The DAIS-WG site: u http://forge.gridforum.org/projects/dais-wg/ l OGSA-DAI Users Mailing list u users@ogsadai.org.uk l Formal support for OGSA-DAI releases u http://bugs.ogsadai.org.uk l OGSA-DAI training courses (live and online)