Presentation is loading. Please wait.

Presentation is loading. Please wait.

System S Project | IBM Research © 2012 IBM Corporation – All Rights Reserved Building User-defined Runtime Adaptation Routines for Stream Processing Applications.

Similar presentations


Presentation on theme: "System S Project | IBM Research © 2012 IBM Corporation – All Rights Reserved Building User-defined Runtime Adaptation Routines for Stream Processing Applications."— Presentation transcript:

1 System S Project | IBM Research © 2012 IBM Corporation – All Rights Reserved Building User-defined Runtime Adaptation Routines for Stream Processing Applications Gabriela Jacques-Silva, Bugra Gedik, Rohit Wagle, Kun-Lung Wu, Vibhore Kumar

2 System S Project | IBM Research © 2012 IBM Corporation – All Rights Reserved Outline  Motivation  Challenge for orchestration  Building a management policy  Use case 2

3 System S Project | IBM Research © 2012 IBM Corporation – All Rights Reserved In System S, users build streaming applications as data flow graphs 3 operator stream connection composite processing element (PE)

4 System S Project | IBM Research © 2012 IBM Corporation – All Rights Reserved Streaming applications have single deployment mode 4 Application Manager start PEs submit application

5 System S Project | IBM Research © 2012 IBM Corporation – All Rights Reserved Users want adaptability, but currently this requires ad-hoc solutions 5 Application Manager submit application start PEs sleep, cat, grep, cut, …

6 System S Project | IBM Research © 2012 IBM Corporation – All Rights Reserved Orchestrator controls the life-cycle of an application via event handling 6 Application Manager submit orchestrator submit application onStart: submitJob(MyApp); onOperatorMetric: … onPEFailure: … onUserEvent: … orchestrator Use cases -Recovery coordination on operator failures -Operator adaptation due to metric measurement values -Incremental job deployment based on runtime learning Use cases -Recovery coordination on operator failures -Operator adaptation due to metric measurement values -Incremental job deployment based on runtime learning

7 System S Project | IBM Research © 2012 IBM Corporation – All Rights Reserved Events and actions occur at the physical level, but users are familiar with logical view 7 composite 1 composite 2 op 3’’ op 6’’ op 4’’ op 5’’ op 4’ op 5’ op 6’ op 3’ Logical view op 1 op 2 op 3’’ op 6’’ op 4’’ op 5’’ op 4’ op 5’ op 6’ op 3’ Physical view op 1 op 2 PE 1 PE 2 PE 3 PE 4 Host A Host B Orchestrator maintains an in-memory application representation to disambiguate logical and physical views when necessary.

8 System S Project | IBM Research © 2012 IBM Corporation – All Rights Reserved Developers register events of interest based on the logical view of the application 8 void MyOrca::handleOrcaStart( const OrcaStartContext & context) { PEFailureScope greenS(“green”); PEFailureScope blueS(“blue”); greenS.addCompositeInstanceFilter(“sample::greenComp”); _orca->registerEventScope(greenS); blueS.addOperatorInstanceFilter(“sample::blueOp”); _orca->registerEventScope(blueS); id = submitJob(sample::MyMainComp); }

9 System S Project | IBM Research © 2012 IBM Corporation – All Rights Reserved On event delivery, we provide a context to understand logical and physical views 9 void MyOrca::handlePEFailure( const PEFailureContext & context) if(context.hasScope(“green”)) { list pes = new list (); _orca->getPEsWithCompositeInstance( pes, “sample::greenComp”); _orca->restartPEs(pes); } else { _orca->executeCommand(“./myRecoveryHelper”); _orca->restartPE(context.getPEId()); } PE failure context: - Matched scopes - Application name - Job ID - PE ID - Reason PE failure context: - Matched scopes - Application name - Job ID - PE ID - Reason

10 System S Project | IBM Research © 2012 IBM Corporation – All Rights Reserved Orchestrator allows management of multiple streaming applications 10 -Dependency relations -Uptime requirements -Garbage collectable -Dependency relations -Uptime requirements -Garbage collectable

11 System S Project | IBM Research © 2012 IBM Corporation – All Rights Reserved Developer configures application requirements and specifies its dependency 11 void MyOrca::handleOrcaStart( const OrcaStartContext & context) { AppConfig fb(“Facebook”, “sample::Facebook”, true, 60.0); AppConfig tw(“Twitter”, “sample::Twitter”, false); AppConfig sc(“Social”, “sample::SNetwork”, true, 60.0); _orca->registerDependency(sc, fb, 10.0); _orca->registerDependency(sc, tw, 10.0); _orca->submitJob(sc); } T, 60 F, 0 T, 60 10.0

12 System S Project | IBM Research © 2012 IBM Corporation – All Rights Reserved Orchestrator starts applications according to their dependencies and uptime requirements 12 T, 60 F, 0 T, 60 10.0 Orchestrator log -10:00:00 AM – “Facebook” submitted -10:00:02 AM – “Twitter” submitted -10:00:12 AM – “Social” submitted Orchestrator log -10:00:00 AM – “Facebook” submitted -10:00:02 AM – “Twitter” submitted -10:00:12 AM – “Social” submitted void MyOrca::handleOrcaStart( const OrcaStartContext & context) { … _orca->registerDependency(sc, fb, 10.0); _orca->registerDependency(sc, tw, 10.0); _orca->submitJob(sc); }

13 System S Project | IBM Research © 2012 IBM Corporation – All Rights Reserved Unused applications are automatically cancelled according to their configurations and dependencies 13 void MyOrca::handleSomeEvent( const SomeEventContext & context) { … _orca->cancelJob(sc); } T, 60 F, 0 T, 60 10.0 Orchestrator log -11:00:00 AM – “Social” cancelled -11:01:00 AM – “Facebook” cancelled Orchestrator log -11:00:00 AM – “Social” cancelled -11:01:00 AM – “Facebook” cancelled

14 System S Project | IBM Research © 2012 IBM Corporation – All Rights Reserved Example demonstrator uses custom operator metrics to adapt a Sentiment Analysis applications 14 Sentiment Analysis Sentiment Analysis ProductCauseSentiment iPhonescreen- iPhoneAppStore+ iPhone?- camera- iPhone?- Model computation

15 System S Project | IBM Research © 2012 IBM Corporation – All Rights Reserved Orchestrator helped streaming application to adapt to the incoming data distribution 15 Actuation threshold new stream of tweets injected MapReduce job submitted New model loaded

16 System S Project | IBM Research © 2012 IBM Corporation – All Rights Reserved Summary  System S hides the complexity of parallel and distributed systems –Orchestrator must open this abstraction to allow appropriate event handling and actuation  APIs for event registration based on logical view  Event delivery includes context to understand physical view  Future work –Add orchestrator concepts to SPL –Use static analysis to automatically infer event scope 16

17 System S Project | IBM Research © 2012 IBM Corporation – All Rights Reserved Building User-defined Runtime Adaptation Routines for Stream Processing Applications Gabriela Jacques-Silva, Bugra Gedik, Rohit Wagle, Kun-Lung Wu, Vibhore Kumar


Download ppt "System S Project | IBM Research © 2012 IBM Corporation – All Rights Reserved Building User-defined Runtime Adaptation Routines for Stream Processing Applications."

Similar presentations


Ads by Google