1. San Diego Supercomputer Center University of California, San Diego
What is SRB Matrix?
Data Grid Automation
Arun Jagatheesan et al.,
San Diego Supercomputer Center
University of California, San Diego
VLDB Workshop on Data Management in Grids
Trondheim, Norway, 2-3 September 2005

2. Talk Outline Data grid Landscape Long-run data management processes
Data Grid ILM
Data Grid Triggers
Dataflow Pipelines
Execution Logic – Data Grid Language
End-to-End Infrastructure Deployment
API
User GUI
Service-oriented *Infrastructure*

3. Data Grid Landscape

4. The “Grid” Vision
complexity in getting electrical grid (transmission, fluctuation etc) but once done plug-n-play devices to grid; similary … grids in computing.

5. Data Grid Resource Providers
I start with by introducing the reality.
We have some data and storage on some disk/ hierarchical resource. IT can have its own directories or physical names for the data. These are Grid Resource providers from the GFS perspective.
Later these resource providers could be other sources like service registries or any thing dealing with a naming system or catalog.
It is highly tempting to say that GFS Resource providers are nothing but SRM – at least looks for me to be so. But please refrain from doing that. We need to have the GSM also agree to this picture.
Grid Resource Providers (GRP) providing content and/or storage
/txt3.txt
GRP
GRP

6. Data Grid Administrative Domain
Administrative domain with one or more GFS Resource Providers
Could include their data centers
Research Lab
Multiple such resource provides could be present in an organization or autonomous administrative domain as we called before.
/txt3.txt
GRP
GRP

7. Data Grid Administrative domains
University
data + storage (10)
Storage-R-Us Resource Providers
data + storage (50)
Research lab- Taiwan
data + storage (40)
There could be multiple administrative domains like this that makes up an virtual enterprise or what we call as the Grid
GRP
/txt3.txt
GRP
GRP
GRP
/…/text1.txt
/…//text2.txt

8. Data Grid (Enterprise Utility)
Physical Resources managed by autonomous administrative domains of the same enterprise (ABCZ.com)
IT Department US
IT Department Asia
3rd Party
ABCZ.com US
Data center
ABCZ.com Asia

9. Data Grid (Enterprise Utility)
Each project has a data grid instance consisting of
Logical Resources with different SLAs offered by IT department
Project 1
Project 2
IT Department US
IT Department Asia
3rd Party
ABCZ.com US
Data center
ABCZ.com Asia

10. Data Grid (Enterprise Utility)
Project1
Project2
Project3
Project4
IT Department US
IT Department Asia
3rd Party
ABCZ.com US
Data center
ABCZ.com Asia

11. Long-run Processes in Data Grid
Data Grid ILM
Data Grid Triggers
Data Gridflows

12. Data Grid ILM

13. Change is Constant Changes in access patterns Data Value
Based on number of users accessing a data
Domains which want to access data
Data Value
The value of data set (collections?) for a particular domain based on it business model and users’ access patterns
Each domain will have a different value based on its users and its role in a data grid

14. “Data Value” based on users
When more users access a project’ data, its data value increases, move that data to a faster storage type
Project1
Project2
Project3
Project4
IT Department US
IT Department Asia
3rd Party
ABCZ.com US
Data center
ABCZ.com Asia

15. “Data Value” based on domain
When more users from the same domain access the data, the data value for that particular data in that particular domain increases, so replicate the data to resources in that domain. (converse is also true)
Project1
Project2
Project3
Project4
IT Department US
IT Department Asia
3rd Party
ABCZ.com US
Data center
ABCZ.com Asia

16. “Data Value” based on role
The 3rd party data center – no users who use data, but is interested in having replica of any data (or deleted data) for long term preservation
Project1
Project2
Project3
Project4
IT Department US
IT Department Asia
3rd Party
ABCZ.com US
Data center
ABCZ.com Asia

17. Data Grid ILM ILM = Information Lifecycle Management
Dynamic re-orientation of data placement and data retention policies (rules)
Based on “business value of data” and storage cost
HSM = Hierarchical Storage Management, based on “data freshness”. ILM goes one step further
Applying this concept on Data Grid, very tricky as different autonomous domains have different business rules

18. Data Grid Triggers

19. Data Grid Triggers Similar to triggers in databases
Based on ECA concepts
Event
Condition
Action
Example
Event = Insert new file in collection (“/ourProject/data”)
Condition = (color= “blue” && galaxy = “Andromedia”)
Action = Run ( selectiveDataReplicator.dgl )

20. Data  Discovery New data Digital entities Meta-data Services State
updates relationships among
data in collections
Meta-data
Services invoked to analyze
new relationships
Services
DGMS applications get
notified of state updates
Generalizing the pipelines used in all these projects
Heavy use of Animation: Take care not to press <enter> or click to many times as it will skip the next slides
State

21. Data Gridflows

22. Gridflow in SCEC (data  information pipeline)
Metadata derivation
Pipeline could be triggered by input at data source or by a data request from user
Ingest Data
Ingest Metadata
Determine analysis pipeline
Initiate automated analysis
Use the optimal set of resources based on the task – on demand
Organize result data into distributed
data grid collections
All gridflow activities stored for data flow provenance

23. Data Grid Language (DGL)

24. Data Grid Language Requirement Analogy of SQL in relational databases
Data Grid ILM process
The long run process that has to be run is described in DGL
Data Grid Triggers
Action part of the ECA (Event-Condition-Action) logic
Data Gridflows
Step by step execution of long run process on Data Grid
Analogy of SQL in relational databases
Long-run process procedures stored and executed in Data Grid it self
Captures the “Infrastructure Execution Logic”

25. DGL Request Annotations about the Data Grid Request
Can be either a Flow or a Status Query

26. DGL Requests (2 types) Data Grid Flow Status Query
An XML Structure that describes the execution logic, associated procedural rules and DGL variables. Can be synchronous or asynchronous flow
Status Query
An XML Structure used to query the execution status any gridflow or a sub-flow at any granular level. Status Queries can be made for both synchronous and asynchronous flows

27. Flow Scoped Variables that can control the flow
Logic used by the sub-members
Sub-members that are the real execution statements

28. Flow Logic (How a flow executes)

29. …
<userDefinedRule name="beforeEntry">
<condition>
<simpleQuery>$numVar == 1</simpleQuery>
</condition>
<action name="true">
<actionString>SET var1 = 1</actionString>
</action>
<actionString>SET var2 = "foo"</actionString>
<action name="false">
<actionString>SET var1 = 0</actionString>
</userDefinedRule>

30. What is SRB Matrix? Matrix provides the SRB as a Web Service
Web Service based on Data Grid Language
SOA for Data Grid or Digital Library
Service oriented *infrastructure*
Asynchronous end-user facing applications
Long run operations presented to users as portlets
Data Grid Automation and ILM
File Triggers on unstructured data
Automated movement or management of data

31. Matrix Gridflow Server Architecture
JAXM Wrapper
WSDL Description
JMS Messaging Interface
Event Publish
Subscribe, Notification
SOAP Service for Matrix Clients
Matrix Data Grid Request Processor
Sangam P2P Gridflow Broker and Protocols
Transaction Handler
Status Query Handler
Workflow Query Processor
Flow Handler and
Execution Manager
XQuery
Processor
ECA rules Handler
Gridflow Meta data Manager
Persistence (Store) Abstraction
Matrix Agent Abstraction
SDSC SRB Agents
Other SDSC Data Services
Agents for java, WSDL and other grid executables
JDBC
In Memory Store

32. Conclusion Data Grids are evolving
Data Grid Automation of long-run processes essential
Need a language for Data Grid Automation
Data Grid Language is one such effort as part SRB Matrix Project
Open source project for anyone to use (or join)
(or