Presentation is loading. Please wait.

Presentation is loading. Please wait.

Selected Project Highlights: An Overview of Gush Jeannie Albrecht Williams College

Similar presentations


Presentation on theme: "Selected Project Highlights: An Overview of Gush Jeannie Albrecht Williams College"— Presentation transcript:

1 Selected Project Highlights: An Overview of Gush Jeannie Albrecht Williams College http://gush.cs.williams.edu

2 Project Overview How do experimenters actually use GENI? Goal: Develop abstractions and tools for addressing the challenges of managing distributed applications on GENI Make it easy for a range of users to run experiments on GENI Strategy Make minimal assumptions about resources and allocations Leverage other services to locate resources/obtain credentials Interface with other user tools to improve functionality Hide complexity and use one common user interface to interact with different CFs (i.e., PlanetLab, ProtoGENI, ORCA)

3 Describe Application Acquire Resources Configure Resources Start Application Monitor Application Cleanup Gush A distributed application management infrastructure Designed to simplify deployment of distributed applications Provides abstractions for configuration and management Allows users to “remotely control” GENI resources running experiments Describe Application Acquire Resources Configure Resources Start Application Monitor Application Cleanup

4 Step 1: Describe Application Describe experiment using application “building blocks” Create customized control flow for distributed applications Application specification blocks are described using XML Application Block Component Block 1 SendersComponent Block 2 Receivers Process Block 1 Prepare Files Process Block 2 Join Network Process Block 3 Send Files Barrier Block 1 Phase 1 Barrier Process Block 1 Join Network Process Block 2 Receive Files Barrier Block 1 Phase 1 Barrier

5 Step 1: App Spec http://sysnet.cs.williams.edu/~jeannie/software.tar software.tar 20 cat software.txt SOFTWARE DEFINE RESOURCE POOL(S) DEFINE PROCESSES (EXECUTION)

6 I/O and Timers Communication Fabric Host Monitor Barriers File Manager Processe s Process Mon Resource Discovery and Acquisition Process Block Workflow Block Component Block Application Block Software Resource Manager User Interface Barrier Block Gush Internal Structure

7 Step 2: Acquire Resources Gush supports experimentation with PlanetLab, ProtoGENI, and ORCA resources How can we find “good” resources? We may want machines with specific characteristics Reserve machines a priori with specific characteristics on ProtoGENI and ORCA Use SWORD (http://sword.cs.williams.edu) on PlanetLab to find “good” machines (see Gush wiki for more info)http://sword.cs.williams.edu Describe Application Acquire Resources Configure Resources Start Application Monitor Application Cleanup

8 Gush Resource Directory PlanetLab ProtoGENI Note: ProtoGENI and ORCA slivers must be created BEFORE starting Gush! ORCA

9 Leveraging Omni When Gush controller starts, it parses resource directory Controller contacts (via XML-RPC) each aggregate manager (AM) specified using common GENI API Gush uses a modified omni client and the “sliverstatus” option Hides CF-specific nuances and differences from experimenters Parses (very different!) output from each AM and creates Gush resource pool…

10 Sliver at http://www.planet-lab.org:12346: {'geni_resources': [{'geni_error': '', 'geni_status': 'ready', 'geni_urn': 'urn:publicid:IDN+plc:williams+sliver+gush2:22629:10859:0', 'pl_boot_state': 'boot', 'pl_hostname': 'planetlab2.williams.edu', 'pl_last_contact': 'Mon Jul 25 21:03:26 2011'}, {'geni_error': '', 'geni_status': 'ready', 'geni_urn': 'urn:publicid:IDN+plc:williams+sliver+gush2:22629:11060:0', 'pl_boot_state': 'boot', 'pl_hostname': 'planetlab5.williams.edu', 'pl_last_contact': 'Mon Jul 25 21:01:31 2011'}, {'geni_error': '', 'geni_status': 'ready', 'geni_urn': 'urn:publicid:IDN+plc:williams+sliver+gush2:22629:10860:0', 'pl_boot_state': 'boot', 'pl_hostname': 'planetlab3.williams.edu', 'pl_last_contact': 'Mon Jul 25 20:57:30 2011'}, {'geni_error': '', 'geni_status': 'ready', 'geni_urn': 'urn:publicid:IDN+plc:williams+sliver+gush2:22629:11059:0', 'pl_boot_state': 'boot', 'pl_hostname': 'planetlab4.williams.edu', 'pl_last_contact': 'Mon Jul 25 21:01:59 2011'}, {'geni_error': '', 'geni_status': 'ready', 'geni_urn': 'urn:publicid:IDN+plc:williams+sliver+gush2:22629:10858:0', 'pl_boot_state': 'boot', 'pl_hostname': 'planetlab1.williams.edu', 'pl_last_contact': 'Mon Jul 25 12:30:39 2011'}], 'geni_status': 'ready', 'geni_urn': 'urn:publicid:IDN+plc:williams+slice+gush2', 'pl_expires': 'Thu Aug 25 13:41:44 2011', 'pl_login': 'williams_gush2'} PlanetLab

11 Sliver at https://www.emulab.net/protogeni/xmlrpc/am: {'geni_resources': [{'geni_error': '', 'geni_status': 'changing', 'geni_urn': 'urn:publicid:IDN+emulab.net+sliver+45994', 'pg_manifest': {'attributes': {'component_manager_urn': 'urn:publicid:IDN+emulab.net+authority+cm', 'component_manager_uuid': '28a10955-aa00-11dd-ad1f-001143e453fe', 'component_urn': 'urn:publicid:IDN+emulab.net+node+pc44', 'component_uuid': 'de9875c1-773e-102b-8eb4-001143e453fe', 'exclusive': 1, 'hostname': 'pc44.emulab.net', 'sliver_urn': 'urn:publicid:IDN+emulab.net+sliver+45994', 'sliver_uuid': '19a4884f-b723-11e0-b47a-001143e453fe', 'sshdport': 22, 'virtual_id': 'geni1', 'virtualization_subtype': 'raw', 'virtualization_type': 'emulab-vnode', 'xmlns': 'http://www.protogeni.net/resources/rspec/0.1'}, 'children': [{'attributes': {'name': 'urn:publicid:IDN+emulab.net+image+emulab-ops//FEDORA10-STD'}, 'children': [], 'name': 'disk_image'}, {'attributes': {}, 'children': [{'attributes': {'authentication': 'ssh-keys', 'hostname': 'pc44.emulab.net', 'port': 22, 'username': 'jeannie'}, 'children': [], 'name': 'login'}], 'name': 'services'}], 'name': 'node'}, 'pg_status': 'changing’}], 'geni_status': 'unknown', 'geni_urn': 'urn:publicid:IDN+emulab.net+slice+gush-jeannie', 'pg_expires': '2011-07-26 01:02:41', 'pg_status': 'changing'} ProtoGENI

12 Sliver at https://geni.renci.org:11443/orca/xmlrpc: ' ************************************************************* [ Reservation UID: b39ca6b3-e332-43d5-98fd-c96c8a5c8821 | Resource Type: renci.vm | Units: 1 | Status: Active | Notices: Reservation b39ca6b3-e332- 43d5-98fd-c96c8a5c8821 (Slice urn:publicid:IDN+emulab.net+slice+gush-orca-jeannie) is in state [Active,None]; ] [ Manifest properties: {unit.id = 61f548fe-41fe-45ad-a48e-e82e2553a87e} {unit.manage.port = 6002} {unit.manage.ip = 152.54.1.93} {unit.state = 2} {unit.domain = rencivmsite} {unit.actorid = cc748912-d46d-423a-a3d7-24062b81c596} {unit.cpu = 1.2} {unit.sliceid = b9a97509-b83a-489b-a514-2419b3b3bb63} {unit.resourceType = renci.vm} {unit.sequence = 2} {unit.rid = b39ca6b3-e332-43d5-98fd-c96c8a5c8821} {unit.ec2.instance = i-3C3A071F} {unit.hostname.url = http://geni-orca.renci.org/owl/rencivmsite.rdf#rencivmsite/Domain/vm} {unit.memory = 1.7} {unit.ndl.adomain = <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:j.0="http://geni-orca.renci.org/owl/compute.owl#" xmlns:j.2="http://geni-orca.renci.org/owl/ethernet.owl#" xmlns:j.1="http://geni-orca.renci.org/owl/layer.owl#" xmlns:owl="http://www.w3.org/2002/07/owl#" xmlns:xsd="http://www.w3.org/2001/XMLSchema#" xmlns:j.4="http://geni-orca.renci.org/owl/domain.owl#" xmlns:j.3="http://geni-orca.renci.org/owl/collections.owl#" xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#" xmlns:j.5="http://geni-orca.renci.org/owl/topology.owl#" > ORCA

13 10 1000000000 eth1 http://geni-orca.renci.org/owl/rencivmsite.rdf ge-0/0/[0-9] 1000000000 ORCA

14 ORCA

15 12 1.7 1.2 ORCA

16 http://geni-orca.renci.org/owl/rencivmsite.rdf true } ] Overall Sliver Status: ready' ORCA

17 Parsed Output 2011-08-25 13:41:44 2011-07-26 01:02:41 2011-07-28 12:52:43

18 Step 3: Configure Resources Connect to and configure selected resources Controller connects to and “remotely controls” the Gush clients on the experimenter’s behalf Controller SSHs to resource, installs Gush client and other software as needed, starts client, client makes TCP connection to controller Client Describe Application Acquire Resources Configure Resources Start Application Monitor Application Cleanup Controller

19 Client Step 4: Start Application Controller issues commands to clients telling them to start running applications/experiments Senders begin running sender processes Receivers begin running receiver processes Client Controller Describe Application Acquire Resources Configure Resources Start Application Monitor Application Cleanup Client

20 Step 5: Monitor Application We want to make sure the processes keep running Clients monitor experiment processes for failures If a failure is detected, client notifies controller Controller may tell client to restart failed process, or find new resource Describe Application Acquire Resources Configure Resources Start Application Monitor Application Cleanup Client Controller Process failed! Restart process/Find new host.

21 Client Step 6: Cleanup Gush clients make sure all programs exited cleanly Remove logs and software from remote machines (optional) Disconnect clients from controller Describe Application Acquire Resources Configure Resources Start Application Monitor Application Cleanup Controller

22 Gush in Action gush> load tests/simple.xml Project "simple" is selected. Experiment "simple" is selected. gush> run Starting experiment run. Running experiment simple... gush> The configuration matcher has finished matching. The resource allocator has finished successfully. williams_gush2@planetlab1.williams.edu:61500 has joined the mesh. The file transfer of Package to planetlab1.williams.edu has been completed. The software installation of Package on planetlab1.williams.edu was successful. williams_gush2@planetlab2.williams.edu:61500 has joined the mesh. The file transfer of Package to planetlab2.williams.edu has been completed. The software installation of Package on planetlab2.williams.edu was successful. williams_gush2@planetlab1.williams.edu:61500,24301: Hello World williams_gush2@planetlab2.williams.edu:61500,27717: Hello World The experiment has ended. gush> disconnect williams_gush2@planetlab1.williams.edu:61500 has decided to leave the mesh. williams_gush2@planetlab2.williams.edu:61500 has decided to leave the mesh. gush> quit Recall our simple app spec…

23 Nebula Nebula (GUI) allows users to describe, run, monitor, & visualize applications XML-RPC interface for managing applications programmatically

24 Gush XML-RPC Interface class GushXmlRpcServer extends XmlRpcServer { void GushAddNode(HashMap properties); void GushRemoveNode(string hostname); void GushCreateResources(); void GushLoadApp(string filename); void GushRunApp(); void GushDisconnectApp(string hostname); void GushQuit(); void GushFailHost(string hostname); void setXmlRpcClientUrl(string clientUrl); } XML-RPC API allows programmatic access to Gush functions External services (including SWORD) use XML-RPC to add and remove resources from Gush resource pool Other Gush user interfaces (including Nebula) run applications using XML-RPC

25 Thank you! For more info: http://gush.cs.williams.edu


Download ppt "Selected Project Highlights: An Overview of Gush Jeannie Albrecht Williams College"

Similar presentations


Ads by Google