From Digital Objects to Content across eInfrastructures Content and Storage Management in gCube Pasquale Pagano CNR –ISTI on behalf of Heiko Schuldt Dept. of Computer Science University of Basel Switzerland

EGEE Conference, Budapest2 Content & Storage Management: Challenges  Store large volumes of digital content in the Grid  But there is much more:  Metadata for each object MER_RR__2P MER RR 2P MERIS ___ALGAL_1 ALGAL World World Europe Bigger_Europe Smaller_Europe Mediterranean Iberia North_Atlantic Africa North_Africa Middle_East Portugal MER_RR__2P MER RR 2P MERIS ___ALGAL_1 ALGAL World World Europe Bigger_Europe Smaller_Europe Mediterranean Iberia North_Atlantic Africa North_Africa Middle_East Portugal MER_RR__2P MER RR 2P MERIS ___ALGAL_1 ALGAL World World Europe Bigger_Europe Smaller_Europe Mediterranean Iberia North_Atlantic Africa ENVI... MER_RR__2P MER RR 2P MERIS ___ALGAL_1 ALGAL World World Europe Bigger_Europe Mediterranean Iberia North_Atlantic Africa North_Africa Middle_East ENVI...  Automatically extracted features per object (e.g., color histo- grams for images)  Storage properties (e.g., size, etc.)  … and all that highly inter- connected

EGEE Conference, Budapest3 Introduction  gCube Data Management provides means for  Persistently storing and physically structuring of content.  Logical grouping of content in collections – independent of physical locations.  Logical sharing of content among several collections.  Complex content consisting of several parts and having multiple representations.  Replication and partition of content.  Subscription and notification  Populating VREs by importing/linking pre-existing content.  gCube exploits Grid technology file-system-like functionalities to manage content storage

EGEE Conference, Budapest4 Storage Properties Content example (Earth Observation) Satellite Image Image Features Image Features Image Features MER_RR__2P MER RR 2P MERIS ___ALGAL_1 ALGAL World World Europe Bigger_Europe Smaller_Europe Mediterranean Iberia North_Atlantic Africa North_Africa Middle_East Portugal ENVI... Metadata as XML Document Metadata Management Content & Storage Management Feature Extraction Storage Properties

EGEE Conference, Budapest5 Document Model  All entities in the VRE are information objects  Collections, documents, metadata  Any information object can be stored or fetched independently of what it represents Information Object Storage_Property Object Reference comprise Reference_Source Reference_Target Reference_Role Reference_Propagation Info_Object_ID Property_Name Property_Type Property_Value BLOB object File object ISA

EGEE Conference, Budapest6 Data Management Architecture Content Management Layer Base Layer Storage Management Layer Content Management Service Collection Management Service Notification Service Archive Import Service Storage Management Service Any off-the-shelf DBMS, e.g. MySQL Any SRM enabled SE, e.g. DPM JDBC GFAL Replication Management Service Properties & Relations Raw File Content Cataloguing API Storage API LFC SRM

EGEE Conference, Budapest7 gCube Storage Model Content Management Layer Base Layer Storage Management Layer RDBMSgLite SE ISA BLOB object File object Content GUID Information Object Document

EGEE Conference, Budapest8 Where to Store What?  An information object is stored in the relational database management system (RDBMS) and in the Storage Element (SE)  The RDBMS is used to store  properties of documents  relationships between documents  Grid storage is used to store raw data

EGEE Conference, Budapest9 Replication  Replication  Fully or partially duplicates raw data among the nodes of a distributed system  Replication management: responsible for the maintenance of replicas  Ensures consistency of multiple copies of the same data object  Identification of master site (original –and updateable– copy) and slave sites (replica)  Basic approaches for maintaining replicas : eager vs. lazy  Eager replication: synchronizes replicas within the boundaries of the update transaction  Lazy replication: decouples synchronization from the updating transaction (replication done in separate transaction)

EGEE Conference, Budapest10 Replication Management Service  Replication Management service is an internal service of the Base Layer  Goal:  increase the degree of availability of content  Provides transparent access to data storage Get Document Store Document Content Management Storage Management DB SE DB 1 DB m SE 1 SE n...

EGEE Conference, Budapest11 Replication Management Service  Features  Ensure consistency of multiple copies of the same data object  Support data with different degrees of freshness  Guarantee consistent reads for all objects of a collection  Handle replication internally lazily, although being eager from the user‘s perspective  Adjust to changing load by dynamically creating new replicas

EGEE Conference, Budapest12 Advanced Replication in the Grid Update Sites Update Transactions Read-only Sites Middleware  Update and read-only storage nodes in the Grid  Query can be served by any node; changes only to be sent to update node  Many update nodes per object: Correct serialization and propagation  Specific features for replication in the Grid  Replicas subscribe for changes instead  Large number of heterogeneous nodes Update Transactions Read- only

EGEE Conference, Budapest13 Base Layer Replication Service Storage Management (SM) Layer Content Management (CM) Layer ColMCMAINS storage node 1 Replicator storage node 2 Replicator storage node n Replicator … Replication in gCube Metadata Management Index Management Change request to a document, e.g. update a document  Replication Service: internal service of the Base Layer  Provides transparent access to data stores Invoking appropriate SM operation Notification is generated to maintain metadata/indexes Update request is handled by the master site of the document Document is updated Update is being propagated to replicas 5.1 Notifications are consumed by external services 6 2

EGEE Conference, Budapest14 Content & Storage Mgmnt in gCube: Summary  Basic tools and protocols are in place but gCube Data Management orchestrates all of them in order to  Associate the different parts of an information object  Associate information objects and all its meta data  Transparently replicate information objects and their meta data

EGEE Conference, Budapest15 Grid (gLite) Storage  LFC: LCG File Catalog (LCG: LHC Computing Grid / LHC: Large Hadron Collider)  Centralized catalog for storing locations of files stored in the grid  Complete catalog can be replicated  SRM: Storage Resource Manager. Interface to  Copy a file on a storage element  Gather information about a file stored into a storage element (SE)  Remove a file from a SRM storage  Retrieve information about a SRM managed storage.  GFAL: Grid File Access Library  provides calls for catalog interaction, storage management and file access and can be very handy when an application requires access to some part  DPM: Disk Pool Manager  APIs for accessing local storage