gCube framework: the way to implement the D4Science vision Pasquale Pagano CNR-ISTI ICIS Requirements Gathering Meeting Rome, Italy, June 2008

2 D4Science vision calls for the realization of scientific e-Infrastructures that will remove all heterogeneity, sustainability, scalability, and other technical concerns from the minds of scientists, hide all related complexities from their perception, and enable them to focus on their science and collaborate on common research challenges gCube is a framework to manage distributed e-infrastructures where it is possible to define, host, and maintain dynamic Virtual Research Environments (VREs) capable to satisfy the collaboration needs of distributed Virtual Organizations (VOs)

3 e-Infrastructure An infrastructure is the basic physical and organizational structures and facilities (roads, power supplies,..) needed for the operation of a society or enterprise An e-Infrastructure provides support for effective consumption of shared resources:  hardware-bound resources (i.e. networks, storage, instruments, and computational resources),  system-level software resources (i.e. basic middleware services),  and application-level software resources (i.e. data sources and services). These e-Infrastructures offer mechanisms that concurrently exploit networks, grids and data in a seamless fashion, and will thus enable scientific communities to operate within a coherent model, regardless of the location of their research facilities.

4 Virtual Organization (VO) A Virtual Organization (VO) models sets of users and resources defining clearly and carefully  what is shared,  who is allowed to share,  and the conditions under which sharing occurs, usually based on an authentication and authorization policies. VOs may have a limited lifetime and they are dynamically created to satisfy transient needs of the constituent potentially heterogeneous actual Organizations.

5 Virtual Research Environment A Virtual Research Environment (VRE) provides a framework of applications, services and data sources dynamically identified to support the underlying processes of research/collaboration/cooperation. The purpose of a VRE is to help researchers* in all disciplines by managing the increasingly complex range of tasks involved in carrying out their activities. * Researcher has to be considered in the large, i.e. end-user, decision-makers, resource and data providers, etc.

6 gCube Infrastructure gHN RI service RI port-type RI VO VRE VO VRE ADMIN DESIGNER ADMIN USERS OWNER ADMIN

VRE Highlights

8 VRE Advantages gCube infrastructures creates new opportunities to change the VRE development model used by distributed and dynamic organisations and communities Using gCube empowered infrastructures, the organisations and communities are able to setup their own environment:  When and for the time they need it  Exploiting existing Grid-based services  Accessing to and handling of distributed multi-focused data and services  Orchestrating user defined services, with defined QoS (wrt. scalability, reliability)  Profiting from a shared storage and computational set of resources  Sharing data and services in a collaborative and efficient way

9 VRE Definition Steps

10 VRE Definition Steps

11 VRE Definition Steps

12 VRE Definition Steps

13 VRE Definition Steps

14 VRE Definition Steps

15 VRE Definition Steps

16 gCube Service Management: System Administrator Measured Effort SCOPEACTIONTIME Infrastructure install collective layer< 1 day install portal< 1 day VO install 1 DHN< 10 min register resources (DHN, data)‏< 1 min approve resource (DHN, data)‏< 1 min data publishing (metadata, indexes)‏hours manage users< 10 min VRE define VRE< 10 min approve VRE< 1 min deploy VRE< 2 hour modify VRE< 1 hour

Content Management

18 Content & Storage Management: Challenges Store large volumes of digital content in the Grid But there is much more: Metadata for each object MER_RR__2P MER RR 2P MERIS ___ALGAL_1 ALGAL World World Europe Bigger_Europe Smaller_Europe Mediterranean Iberia North_Atlantic Africa North_Africa Middle_East Portugal MER_RR__2P MER RR 2P MERIS ___ALGAL_1 ALGAL World World Europe Bigger_Europe Smaller_Europe Mediterranean Iberia North_Atlantic Africa North_Africa Middle_East Portugal MER_RR__2P MER RR 2P MERIS ___ALGAL_1 ALGAL World World Europe Bigger_Europe Smaller_Europe Mediterranean Iberia North_Atlantic Africa ENVI... MER_RR__2P MER RR 2P MERIS ___ALGAL_1 ALGAL World World Europe Bigger_Europe Mediterranean Iberia North_Atlantic Africa North_Africa Middle_East ENVI... Automatically extracted features per object (e.g., color histo- grams for images) Storage properties (e.g., size, etc.) … and all that highly inter- connected

19 gCube Data Management gCube Data Management supports researchers by providing means for  Persistently storing and physically structuring of content  Logical grouping of content in collections  Logical sharing of content among several collections  Sharing of collections in a VRE and among several VREs through shared workspace  Management of complex content consisting of several parts and having multiple representations  Storage of structured and heterogeneous metadata compliant with different formats and schemas  Programmatic/manual annotation of content via text & images, e.g. data provenance  Content linking  Definition of ‘composite documents’ template

20 gCube Data Management [cont.] gCube Data Management supports application designers/providers by providing means for  Bulk upload and update of data and metadata  Manipulation of metadata and data through a powerful Data Transformation Engine (gDTS).  Metadata can be cleaned, enriched, and transformed in different formats by exploiting mapping schema, controlled vocabulary, thesauri, and ontology to facilitate data integration and discovery  Data can be transformed to offer different views of the some goods  Replication and partition of data  Subscription and notification  Generation and publishing of new data through workflow definition, optimisation, and execution gCube exploits Grid technology file-system-like functionalities to manage data storage

21 Document Model All entities in a VRE are information objects: collections, documents, metadata Any information object can be stored or fetched independently of what it represents Information Object Storage_Property Object Reference comprise Reference_Source Reference_Target Reference_Role Reference_Propagation Info_Object_ID Property_Name Property_Type Property_Value BLOB object File object ISA

22 Content example (Earth Observation) Storage Properties Satellite Image Image Features Image Features Image Features MER_RR__2P MER RR 2P MERIS ___ALGAL_1 ALGAL World World Europe Bigger_Europe Smaller_Europe Mediterranean Iberia North_Atlantic Africa North_Africa Middle_East Portugal ENVI... Metadata as XML Document Metadata Management Content & Storage Management Feature Extraction Storage Properties

23 Data Management Architecture Content Management Layer Base Layer Storage Management Layer Content Management Service Collection Management Service Notification Service Archive Import Service Storage Management Service Any off-the-shelf DBMS, e.g. MySQL Any SRM enabled SE, e.g. DPM JDBC GFAL Replication Management Service Properties & Relations Raw File Content Cataloguing API Storage API LFC SRM

24 gCube Storage Model Content Management Layer Base Layer Storage Management Layer RDBMSgLite SE ISA BLOB object File object Content GUID Information Object Document

25 Bridging Data Sources Hosted on the infrastructure Data Sources are interfaced through.. The bridges are managed by.... the infrastructure

26 Managing Data Source Heterogeneity Mapping rules

27 gCube Infrastructure Managing Data Import DS import MR VRE 2VRE 1VRE 3VRE 4

Search Management

29 gCube Search Engine Features An open, feature-rich distributed Search Engine  Composed out of diverse, autonomous, pluggable elements.  Capturing complex application scenarios by combining information retrieval and data processing procedures Maximization of resources placed at the disposal of VRE managers and users  Ease of sharing of resources, avoiding mis-utilization and misuse  Reduction of cost of ownership and use

30 Functional overview Search types  Structured data (fielded search / xml search)  Semi structured data (xml search)  Geospatial / temporal data (R-Tree)  Content based search  Full text search  Image similarity search Access  XML-based Query Language  Web user interface (portal / search portlets)  Command line UI Retrieval  Incremental result delivery  Automatic caching  Result persistence

31 Functional Overview [cont.] Automatic description of the Content Sources To cope with  Several content sources  Potentially thematically focused  Different ranking estimation  Different structures for managing metadata / information retrieval  Different content By offering the  Selection of the appropriate sources  Merging or fusing (re-ranking) the results in meaningful lists

32 Search in action

33 Search in action

34 Search in action

35 Search in action

Process Management

37 Process Management Rationale VREs are large-scale collections of resources that provide dedicated services to manage and access digital data sources VRE applications require to integrate these (distributed) resources into a coherent whole = „Programming in the Large“ (composition of services into processes) Query Reformulation Index Access (Query Execution) Result Filtering

38 Service Process Management: Example Search Similarity search over multimedia documents 1. Query 2. Extract Features 3. Query Index 5. Present Results Index Access Content & Create Result Set Process

39 Processes in gCube Following the SOA paradigm, processes are the first choice in gCube to define and execute applications on the basis of available services gCube’s approach to Process Management on the Grid consists of three main phases  Process Design and Verification  Through a graphical user interface for specification of processes  Process Execution and Reliability  Distribute process support in the infrastructure  Dynamic allocation of resources  Sophisticated failure handling  Process Optimization  Structural process modifications to maximise parallelism

40 … Sample Application Process: Meris Global Vegetation Index (MGVI)

41 Process Monitoring A monitoring interface allows to keep track of running process instances

42 Conclusion gCube reduces the costs to manage any complex multi- domain e-Infrastructure. gCube offers an horizontal solution to manage and enrich on-demand created VREs on complex e-Infrastructures. gCube is equipped with data and metadata management facilities that allows to make interoperable heterogeneous data sources. gCube is compliant with consolidated and emerging standards. gCube offers an open family of frameworks that can be easily customised

Thank you.