SmartGRID Decentralized, dynamic grid scheduling framework on swarm agent-based intelligence GCC'08, shenzhen, China. Oct. 26, 2008 Ye HUANG, Amos BROCCO Grid Group, Dept of Information and Communication Technologies, EIA-FR, Switzerland Pervasive Artificial Intelligence Group, Dept of Informatics, University of Fribourg, Switzerland

2 Outline  Aim  SmartGRID architecture  SmartGRID in depth  MaGate scheduler  Data warehouse interface  Agent-based swarm intelligence  Summary  Future work

3 Aim  Vision of Grid:  Large scale distributed resources  Decentralized  Different policies  Unstable, low reliability  SmartGRID: grid resource management framework  Utilizing different scheduling algorithms  Interoperation emphasized scheduler  Dynamic resource discovery by swarm intelligent algorithm

4 SmartGRID layered architecture  Loosely coupled layered architecture.  Two layers and one internal interface. Smart Resource Management Layer Data Warehouse Interface Smart Signaling Layer

5 SmartGRID Node (SG-Node)  SG-Node: the logical unit of SmartGRID framework  MaGate scheduler, DW interface, Ant nest MaGate Scheduler Nest Info. collector DW Interface

6 MaGate scheduler  MaGate stands for “Magnetic Gateway for scheduler”  Core of Smart Resource Management Layer (SRML)  Targets:  Open platform to different scheduling algorithms  Based on dynamic infrastructure information  Gateway between schedulers (MaGate & others, etc. PBS, MSS)  Allow heterogenous and dynamic Grid scheduling  Manage community of resources  Decentralized view of the Grid  Interface to external services  To deal with corresponding issues, e.g. network behavior analyzing  MaGate focuses on:  Decentralization & Interoperability

7 MaGate architecture  1. Self-management  2. Access to invoker  3. Community management  4. LRM utilization  5. External components Remote MaGate Local Resource Management Grid applications

8 MaGate behavior MaGate Community  Job executor  Interface to invoker  Router  Interface to external service  Full functional package

9 MaGate current roadmap  Community Component  Interoperation protocol & behavior, negotiation model  DRM Component  SAGA API  Interface Component  Application-Interface (App-I) to POP-C++  External Component Multi- scheduling algorithm adoption mechanism Utilizing agent-based dynamic resource discovery

10 MaGate technical timetable for current roadmap  Simulation  Basic on GridSim, integrated with Alea, GSSIM (Done)  DataWarehouse Interface prototype  Communication channel between SRML and SSL (Ongoing)  Candidate standard specification  Scheduler Interoperability best practice, WS-Agreement, JSDL, CSG (Ongoing)  First standalone prototype

11 Smart Signaling Layer  What do we need? A network and remote resources information source » network monitoring » resource discovery  Other Requirements – adaptive, reliable, robust behavior over unstable network

12 Smart Signaling Layer  What do we need? A network and remote resources information source » network monitoring » resource discovery  Other Requirements – adaptive, reliable, robust behavior over unstable network Swarm Intelligence Algorithms

13 Ant-based swarm intelligence Swarm intelligence?  Artificial intelligence inspired by the behavior of swarms (of insects) ‏ » Typically used for optimization problems (Particle Swarm Optimization, Ant Colony Optimization) ‏... »...but also suitable for fully distributed algorithms Our approach  Use ants (lightweight mobile agents traveling across the network) to perform different tasks on the network.

14 Why swarm intelligence? Strenghts? – Fully distributed algorithm, asynchronous communication Ants can only access local resources on nodes Collaboration between individuals through indirect communication (stigmergy, pheromone trails) ‏ – Robust and fault tolerant algorithms loss of individuals can be tolerated – Adaptive behavior adapts to changing network environment

15 Datawarehouse interface – Why use a DWI? Loosely coupled communication between layers – Easier to adapt the framework to different scenarios – In detail... Persistent and cached Grid information storage – local information monitored by the scheduler – coordinated scheduling information negotiated by scheduler – network information gathered by ants – service request queries

16 Goals Mission: – Membership Management and Resource Discovery

17 Goals Mission: – Membership Management and Resource Discovery Idea: – Maintain an optimized topology across nodes: – lower TTL for resource discovery queries, lower communication overhead

18 Goals Mission: – Membership Management and Resource Discovery Idea: – Maintain an optimized topology across nodes: – lower TTL for resource discovery queries, lower communication overhead Implementation: – BlåtAnt algorithm

19 BlåtAnt Algorithm: introduction BlåtAnt – Fully distributed algorithm using ant colonies to construct and maintain a peer-to-peer overlay topology with bounded diameter – Pure peer-to-peer, unstructured networks: » simple membership management – Balanced link distributions: no large hubs – Fault resilient

20 BlåtAnt Algorithm: logic Goal – Ensure that the network diameter d is D ≤ d < 2D – 1 How? – Create and remove logical links: Connection Rule: two nodes are connected if their distance (number of hops) is greater than 2D – 1 Disconnection Rule: two adjacent nodes are disconnected if there exist an alternative path between them of length less than D

21 BlåtAnt Algorithm: connection rule (1) ‏ – Connection Rule: two nodes are connected if their distance (number of hops) is greater than 2D – 1 A B D = 4 2D – 1 = 7 d(A,B) = 8 > 2D - 1 = 7

22 BlåtAnt Algorithm: connection rule (2) ‏ – Connection Rule: two nodes are connected if their distance (number of hops) is greater than 2D – 1 A B D = 4 2D – 1 = 7

23 BlåtAnt Algorithm: disconnection rule (1) ‏ – Disconnection Rule: two adjacent nodes are disconnected if there exist an alternative path between them of length less than D A C B D = 4 2D – 1 = 7 Alternative path d(A,C) = 3 < D = 4

24 BlåtAnt Algorithm: disconnection rule (2) ‏ – Disconnection Rule: two adjacent nodes are disconnected if there exist an alternative path between them of length less than D A C B D = 4 2D – 1 = 7 Not necessary!

25 BlåtAnt Algorithm: disconnection rule (3) ‏ – Disconnection Rule: two adjacent nodes are disconnected if there exist an alternative path between them of length less than D A C B D = 4 2D – 1 = 7

26 BlåtAnt Algorithm: implementation Who does what? – Different species of ants with different tasks: Collect and spread information Connect / Disconnect peers

27 BlåtAnt Algorithm: implementation Who does what? – Different species of ants with different tasks: Collect and spread information Connect / Disconnect peers – Nodes run computation tasks: Discover nodes matching connection/disconnection rules based on a partial view of the network

28 BlåtAnt Algorithm: an example graph before augmentation graph after augmentation diameter = 4 (< 2D – 1, D=3) ‏ diameter = 19

29 Other details Where do ants live? – Solenopsis Framework: fully distributed platform to execute ant algorithms Sandboxed environment Extensible Support for strong migration of ant agents

30 Summary Smart Resource Management Layer – modular scheduler architecture – decentralized, distributed – open to existing local schedulers and external services Data Warehouse Interface Smart Signaling Layer – ant algorithms to provide network services – underlying peer-to-peer topology constructed using the BlåtAnt algorithm

31 Future work MaGate Scheduler – First prototype – SG-Node validation Data Warehouse – First development and integration Smart Signaling Layer (SSL) ‏ – Further research on resource discovery algorithms, round-trip time optimization, pro- active monitoring – Solenopsis 2.0 Middleware

32 Thanks! Questions?