1. unibasel Christoph Langguth, Paola Ranaldi, Heiko Schuldt University of Basel, Database and Information Systems Group Bernoullistr 16, CH-4056, Basel, Switzerland {firstname.lastname}@unibas.ch This work has been partly supported by the Hasler Foundation within the project COSA (Compiling Optimized Service Architectures)‏ Towards Quality of Service For Scientific Workflows by using Advance Resource Reservations

2. unibasel 2 A motivating example: weather forecast Long-running and data-intensive And time-critical: must be finished by the time of evening news ADAS-ARPS Data Analysis Satellite Data Remapper L3 Radar Remapper L2 Radar Remapper Terrain Preprocessor Gridded Data Interpolator Surface Preprocessor ARPS to WRF Converter WRF Model WRF to ARPS Converter ARPS Plotting Preprocessing Misc.Transformation Analysis Legend (*) Simplified version of workflow from Droegemeier, Gannon, Reed, et al: Service-Oriented Environments for Dynamically Interacting with Mesoscale Weather

3. unibasel 3 QoS, and how to provide it – in a nutshell Individual services give guarantees about their execution time –Which are combined to QoS guarantees for the entire WF –users could specify: this WF must be executed as [fast, cheap, energy-efficient,...] as possible Contracts negotiated using WS-Agreement Needs predictable resource utilization on provider's side –Resources needed must be announced, and reserved for execution –Advance Reservations for every service call A bunch of metadata is required for planning the execution and setting up the reservations

4. unibasel 4 WF execution in DWARFS: the big picture A B DE F G H C Eng1 Eng3 Eng2 A B DE FGH C X:50GB T:40GB V:35GB W:23KB Z:10KB Y:15KB small amount of data Data dependencies large amount of data DWARFS: Distributed Workflow execution engine with Advance Reservation Functionality Support Partition the process for distributed execution: –Keeping large data transfers “local“, as far as possible –Inter-partition data transfers are handled by special DWARFS storage subsystem Start: 11:30 End: 13:07 CPU: 80% Storage: 10% Start: 17:04 End: 17:53 CPU: 25% Scheduling, Partitioning, Reservations are all interrelated And depend on the metadata that service providers make available

5. unibasel 5 Required Metadata for Reservations... CPU: storage: HW: Provider AProvider B I'll need to call Operation A w/INPUT: size=50GB, Class X OUTPUT: 25GB, Class Y timing: 50 % max. duration inv. prop. 30 % min. 10 % exactly 2:30 h 100 % max. duration inv. prop. 5 % min. 100 % exactly 3:50 h

6. unibasel 6...Required Metadata for Reservations Call of operation X: – Input characterization (size, class) – Resource requirements – Timing prediction, and resource ↔ timing relationship – Output characterization Also need providers' current resource allocation schedules, and cost functions Unified notion of resource „share of capacity“ Data characteristics metadata and duration prediction highly domain-specific – Only requirement for us: determine size of data

7. unibasel 7 Past, present and future Prototype of CPU share enforcement/timing prediction Formal model of DWARFS Workflow WS-Agreement with renegotiation support Planner implementation Storage Subsystem Bringing it all together :-)

8. unibasel 8 Thank you for your attention!

9. unibasel 9...CPU enforcement in action

10. unibasel 10 Required Metadata for reservations Call of service operation X with input characterized so and so, –How long will this take? –Which resources are needed? t, and how does modifying the share influence the timing? –Which output will this produce? Providers need to also make available their current resource allocation schedules, and the respective cost functions Any kind of resource can be represented using the unified notion of „share of local capacity“ Data characteristics metadata will be domain-specific. –Only requirement for DWARFS: be able to determine the size of it –(prediction of duration is also highly domain- and implementation-specific)