Performance Architecture within ICENI Dr Andrew Stephen M c Gough Laurie Young, Ali Afzal, Steven Newhouse and John Darlington London e-Science Centre Department of Computing, Imperial College London

2 Outline Overview of ICENI Performance Framework Example Conclusion

3 ICENI: Imperial College e-Science Network Infrastructure Collect and provide relevant Grid Meta-Data Pluggable architecture Test Architecture for Grid Research Foundation for higher-level Services and Autonomous Composition Integrated Grid Middleware Solution Interoperability between architectures, APIs Added value layer to other middleware Usability: Interactive Grid Workflows Role and policy driven security ICENI Open Source licence (extended SISSL) The Iceni, under Queen Boudicca, united the tribes of South-East England in a revolt against the occupying Roman forces in AD60.

4 Scheduling Workflows in ICENI Applications consist of a number of components linked together in a dataflow manner The abstract workflow needs to be mapped down to a set of component implementations which will run on resources Linear Equation Source Linear Equation Solver Display Vector Results General Equation generato r LU Factorisatio n Simple Vector Display

5 The Problem We have: –Multiple resources where components can run –Multiple implementations of components –The choice of one resource component mapping can affect the others User wants predictable performance How to choose the “best” mapping of workflow over resources to give user predictability?

6 The Solution We need to take into account: –Execution Times of components on Resources Performance Data –Inter-component effects of workflows Workflow aware Schedulers –Workload on resources, making sure they are free when we need them Reservation systems

7 The Architecture Scheduler Reservation Service Performance Store Launcher Application Service Reservation Engine Workflow

8 Performance Store - Persistent Performance storage Performance Store - Persistent Performance storage The Performance Repository Framework: Performance Framework Performance Processing - Conversion of raw event times into performance data Data Collector -Collecting data on currently running applications (event times) Performance Store - Persistent Performance storage

9 Collection of Performance Results Data Collector Linear Equation Source Linear Equation Solver Display Vector Results Time Event 12:00 Linear Equation Source Start 12:04 Send out Equations 12:03 Linear Equation Solver Start 12:05 Receive Equations 12:12 ……….. Event: Start Linear Equation Source Performance Processing Workflow Performance Store

10 Storing Performance Data Multiple stores can be used in one framework The stores may be data stores or analytical models All assumed to be persistent Allows requests for predictions to be made New Data can be added to the stores Store data is aggregated together –based upon reliability of store data Provided by the store Performance Store

11 Using Performance Data Scheduler Builds up workflow graph with timings requested from the Performance Repository Timings are based on component implementation, resource, co-allocation count and other properties defined by the component implementer As the store will not contain all possible combinations of these properties regression is used to provide estimates for the missing values –This is an area of ongoing research

12 Reservation Engine Reservation Engine Framework Reservations Database - Data about upcoming reservations,used to test if a new reservation can be made DRM Translation level -Communication with the underlying DRM system for supporting Reservations Listen out for requests - Launcher services wishing to make reservations

13 Reservation Service Reservation Service Framework Reservations Database - Data about upcoming reservations,used to test if a new reservation can be made Reservation orchestration -Attempts to reserve all resources for a given workflow. May shift when it runs Listen out for Services - Launcher with reservation - Scheduling Services

14 Reservations time → Reservation Service Scheduler Workflow Reserve (workflow) HOLD Conflict HOLD Reserve WS-Agreement Request interval Linear Equation Source Linear Equation Solver Display Vector Results Reservations not possible on Users Desktop

15 Example: Linear Equation Solver service listcomposition paneparameters

16 Inferring Workflow from Dataflow

17 Conclusion Better usage of resources. –Reservations of resources in the future –Determining if co-allocation of components will affect performance –Late Enactment of components Critical Path analysis – can schedule this appropriately Provides a framework for experimentation with –Different scheduling algorithms –Different performance models –Different reservation policies

18 Performance Trinity Performance Models Grid SchedulingReservation Fabric

19 The Architecture: Showing the Trinity Scheduler Reservation Service Performance Store Launcher Application Service Reservation Engine

20 Acknowledgements Director: Professor John Darlington Research Staff: –Nathalie Furmento, Stephen McGough, William Lee –Jeremy Cohen, Marko Krznaric, Murtaza Gulamali –Asif Saleem, Laurie Young, Jeffrey Hau –David McBride, Ali Afzal Support Staff: –Oliver Jevons, Sue Brookes, Glynn Cunin, Keith Sephton Alumni: –Steven Newhouse, Yong Xie, Gary Kong –James Stanton, Anthony Mayer, Angela O’Brien Contact: – 

21 Scheduling Architecture in ICENI Scheduler Globus Resource s Globus Launcher Single Resource Launcher Software Resource Repository Advertise Submit Query Request Reservation Performance Repository SGE Resource s Reservation Launcher Reservation Service Query Negotiate Reservation WS-Agreement