San Diego Supercomputer Center National Partnership for Advanced Computational Infrastructure San Diego Supercomputer Center National Partnership for Advanced Computational Infrastructure University of Florida POSIX-like OGSA/SOAP Services Arun Jagatheesan Architect & Team Lead, SDSC Matrix San Diego Supercomputer Center GFS, Global Grid Forum-9 October 7, 2003, Chicago

San Diego Supercomputer Center National Partnership for Advanced Computational Infrastructure University of Florida 2 Talk Outline Grid File System The small big picture Need for Schema Need for Operation definitions Data Transport

San Diego Supercomputer Center National Partnership for Advanced Computational Infrastructure University of Florida 3 Grid File System Data Sources Grid File System Service (POSIX-like Interface) Data Services Virtual Directory Service (Management of virtualization) Coordinated with other groups Hierarchical Logical Name space, ACL, metadata Applications (Astronomy, Physics, Life Science, business apps,...) NFS/CIFS …

San Diego Supercomputer Center National Partnership for Advanced Computational Infrastructure University of Florida 4 The small big picture Data Sources Grid File System Service (POSIX-like Interface) Data Services Virtual Directory Service (Management of virtualization) NFS/CIFS … XML Schema for Collections, Data Sets OGSA/SOAP based interfaces for file operations NFS or other standard interface over the virtualized schema

San Diego Supercomputer Center National Partnership for Advanced Computational Infrastructure University of Florida 5 Grid Collection Schema XML Schema based Description for Collections or Virtual Directories Data Sets File System Meta-data (file size, date created, …) Application Specific Meta-data Access Permissions … Logical Name space Extensible Scalable (more federations) Dynamic Composition of the name space Import and Export

San Diego Supercomputer Center National Partnership for Advanced Computational Infrastructure University of Florida 6 Operations on Logical Namespace OGSA/SOAP based interfaces Grid File System operations Similar to traditional file systems operations / POSIX Open (= Get a GSR?), Read, Seek’n’Read, Seek’n’Write, … Simple Control (Context) Operations Management of Logical Namespace SOAP based bindings Bulk (Content) Operations Only SOAP bindings for data transport ??? (NOPE) Alternative mechanisms needed in standard

San Diego Supercomputer Center National Partnership for Advanced Computational Infrastructure University of Florida 7 How do we form the logical namespace?

San Diego Supercomputer Center National Partnership for Advanced Computational Infrastructure University of Florida 8 Logical Layers (bits,data,information,..) Storage Resource Transparency Storage Location Transparency E:\srbVault\image.jpg /users/srbVault/image.jpg Select … from srb.mdas.td where... Data Identifier Transparency image_0.jpg…image_100.jpg Data Replica Transparency image.sqlimage.cgiimage.wsdl Virtual Data Transparency Collections or Virtual Directories patientRecordsCollectionmyActiveNeuroCollection

San Diego Supercomputer Center National Partnership for Advanced Computational Infrastructure University of Florida 9 Storage Resource Transparency (1) Storage repository abstraction Archival systems, file systems, databases, FTP sites, … Logical resources Combine physical resources into a logical set of resources Hide the type and protocol of physical storage system Load balancing – based on access patterns Unlike DBMS, user is aware of logical resources Flexibility to changes in mass storage technology

San Diego Supercomputer Center National Partnership for Advanced Computational Infrastructure University of Florida 10 Storage Resource Transparency (2) Standard operations at storage repositories POSIX like operations on all resources Storage specific operations Databases - bulk metadata access Object ring buffers - object based access Hierarchical resource managers - status and staging requests

San Diego Supercomputer Center National Partnership for Advanced Computational Infrastructure University of Florida 11 Storage Location Transparency Support replication of data for performance Transparent access to physical location and physical resource Virtualization of distributed data resources Data naming managed by the data grid Redundancy for preservation Resource redundancy – “m of n” resources in list Location redundancy – replicate at multiple locations

San Diego Supercomputer Center National Partnership for Advanced Computational Infrastructure University of Florida 12 Data Identifier Transparency Four Types of Data Identifiers: 1. Unique name OID or handle 2. Descriptive name Descriptive attributes – meta data Semantic access to data 3. Collective name Logical name space of a collection of data sets Location independent 4. Physical name Physical location of resource and physical path of data

San Diego Supercomputer Center National Partnership for Advanced Computational Infrastructure University of Florida 13 Data Replica Transparency Replication Improve access time Improve reliability Provide disaster backup and preservation Physically or Semantically equivalent replicas Replica consistency Synchronization across replicas on writes Updates might use “m of n” or any other policy Distributed locking across multiple sites Versions of files Time-annotated snapshots of data

San Diego Supercomputer Center National Partnership for Advanced Computational Infrastructure University of Florida 14 Conclusion Lot of possibilities Need for a Standard Grid File Schema and Global Logical Namespace for virtualization Need for Standard description of Operations or Grid File System Service Call for Users, Projects Developers, Vendors It’s a stone’s throw away – together, we will do it.