4 th International Conference on Service Oriented Computing Adaptive Web Processes Using Value of Changed Information John Harney, Prashant Doshi LSDIS.

Slides:



Advertisements
Similar presentations
CSSSIA Workshop – WWW 2008 Speeding up Web Service Composition with Volatile External Information John Harney, Prashant Doshi LSDIS Lab, Dept. of Computer.
Advertisements

Decision Theory: Sequential Decisions Computer Science cpsc322, Lecture 34 (Textbook Chpt 9.3) Nov, 28, 2012.
Lirong Xia Reinforcement Learning (2) Tue, March 21, 2014.
Reinforcement Learning (II.) Exercise Solutions Ata Kaban School of Computer Science University of Birmingham 2003.
Probabilistic Planning (goal-oriented) Action Probabilistic Outcome Time 1 Time 2 Goal State 1 Action State Maximize Goal Achievement Dead End A1A2 I A1.
U NIVERSITY OF M ASSACHUSETTS, A MHERST Department of Computer Science Solving POMDPs Using Quadratically Constrained Linear Programs Christopher Amato.
Department of Computer Science Undergraduate Events More
Project Management Operations -- Prof. Juran. 2 Outline Definition of Project Management –Work Breakdown Structure –Project Control Charts –Structuring.
Use of Kalman filters in time and frequency analysis John Davis 1st May 2011.
CS 795 – Spring  “Software Systems are increasingly Situated in dynamic, mission critical settings ◦ Operational profile is dynamic, and depends.
1 Reinforcement Learning Introduction & Passive Learning Alan Fern * Based in part on slides by Daniel Weld.
All Hands Meeting, 2006 Title: Grid Workflow Scheduling in WOSE (Workflow Optimisation Services for e- Science Applications) Authors: Yash Patel, Andrew.
A Hierarchical Framework for Composing Nested Web Processes Haibo Zhao, Prashant Doshi LSDIS Lab, Dept. of Computer Science, University of Georgia 4 th.
An Introduction to Markov Decision Processes Sarah Hickmott
Planning under Uncertainty
Decision Theory: Single Stage Decisions Computer Science cpsc322, Lecture 33 (Textbook Chpt 9.2) March, 30, 2009.
Nov 14 th  Homework 4 due  Project 4 due 11/26.
Planning in MDPs S&B: Sec 3.6; Ch. 4. Administrivia Reminder: Final project proposal due this Friday If you haven’t talked to me yet, you still have the.
Cooperative Q-Learning Lars Blackmore and Steve Block Expertness Based Cooperative Q-learning Ahmadabadi, M.N.; Asadpour, M IEEE Transactions on Systems,
Markov Decision Processes
Planning to learn. Progress report Last time: Transition functions & stochastic outcomes Markov chains MDPs defined Today: Exercise completed Value functions.
Algorithms For Inverse Reinforcement Learning Presented by Alp Sardağ.
More RL. MDPs defined A Markov decision process (MDP), M, is a model of a stochastic, dynamic, controllable, rewarding process given by: M = 〈 S, A,T,R.
CPSC 322, Lecture 35Slide 1 Value of Information and Control Computer Science cpsc322, Lecture 35 (Textbook Chpt 9.4) April, 14, 2010.
Making Decisions CSE 592 Winter 2003 Henry Kautz.
Exploration in Reinforcement Learning Jeremy Wyatt Intelligent Robotics Lab School of Computer Science University of Birmingham, UK
Dimitrios Konstantas, Evangelos Grigoroudis, Vassilis S. Kouikoglou and Stratos Ioannidis Department of Production Engineering and Management Technical.
MDP Reinforcement Learning. Markov Decision Process “Should you give money to charity?” “Would you contribute?” “Should you give money to charity?” $
Deciding when to intervene: A Markov Decision Process approach Xiangjin Zou(Rho) Department of Computer Science Rice University [Paolo Magni, Silvana Quaglini,
Computational Stochastic Optimization: Bridging communities October 25, 2012 Warren Powell CASTLE Laboratory Princeton University
1 Endgame Logistics  Final Project Presentations  Tuesday, March 19, 3-5, KEC2057  Powerpoint suggested ( to me before class)  Can use your own.
Rohit Aggarwal, Kunal Verma, John Miller, Willie Milnor Large Scale Distributed Information Systems (LSDIS) Lab University of Georgia, Athens Presented.
20 October 2006Workflow Optimization in Distributed Environments Dynamic Workflow Management Using Performance Data David W. Walker, Yan Huang, Omer F.
Toward Optimal and Efficient Adaptation in Web Processes Prashant Doshi LSDIS Lab., Dept. of Computer Science, University of Georgia Joint work with: Kunal.
Decision Making Under Uncertainty and Risk 1 By Isuru Manawadu B.Sc in Accounting Sp. (USJP), ACA, AFM
Chapter 4 MODELING AND ANALYSIS. Model component Data component provides input data User interface displays solution It is the model component of a DSS.
Simulation is the process of studying the behavior of a real system by using a model that replicates the behavior of the system under different scenarios.
Haley: A Hierarchical Framework for Logical Composition of Web Services Haibo Zhao, Prashant Doshi LSDIS Lab, Dept. of Computer Science, University of.
FORS 8450 Advanced Forest Planning Lecture 5 Relatively Straightforward Stochastic Approach.
Cooperative Q-Learning Lars Blackmore and Steve Block Multi-Agent Reinforcement Learning: Independent vs. Cooperative Agents Tan, M Proceedings of the.
Learning to Navigate Through Crowded Environments Peter Henry 1, Christian Vollmer 2, Brian Ferris 1, Dieter Fox 1 Tuesday, May 4, University of.
MDPs (cont) & Reinforcement Learning
© English Matthews Brockman Business Planning in Personal Lines using DFA A Talk by Mike Brockman and Karl Murphy 2001 Joint GIRO/CAS Conference.
1 1 Slide Simulation Professor Ahmadi. 2 2 Slide Simulation Chapter Outline n Computer Simulation n Simulation Modeling n Random Variables and Pseudo-Random.
1 Chapter 17 2 nd Part Making Complex Decisions --- Decision-theoretic Agent Design Xin Lu 11/04/2002.
COMP 2208 Dr. Long Tran-Thanh University of Southampton Reinforcement Learning.
Smart Sleeping Policies for Wireless Sensor Networks Venu Veeravalli ECE Department & Coordinated Science Lab University of Illinois at Urbana-Champaign.
Generalized Point Based Value Iteration for Interactive POMDPs Prashant Doshi Dept. of Computer Science and AI Institute University of Georgia
On the Difficulty of Achieving Equilibrium in Interactive POMDPs Prashant Doshi Dept. of Computer Science University of Georgia Athens, GA Twenty.
Adaptable Approach to Estimating Thermal Effects in a Data Center Environment Corby Ziesman IMPACT Lab Arizona State University.
18 May 2006CCGrid2006 Dynamic Workflow Management Using Performance Data Lican Huang, David W. Walker, Yan Huang, and Omer F. Rana Cardiff School of Computer.
1 Passive Reinforcement Learning Ruti Glick Bar-Ilan university.
Introducing Information into RM to Model Market Behavior INFORMS 6th RM and Pricing Conference, Columbia University, NY Darius Walczak June 5, 2006.
Prof. Dr. Holger Schlingloff 1,2 Dr. Esteban Pavese 1
Reinforcement Learning (1)
ADVANTAGES OF SIMULATION
Task: It is necessary to choose the most suitable variant from some set of objects by those or other criteria.
16th International World Wide Web Conference Speeding up Adaptation of Web Service Compositions Using Expiration Times John Harney, Prashant Doshi LSDIS.
Optimal Electricity Supply Bidding by Markov Decision Process
Professor Arne Thesen, University of Wisconsin-Madison
Optimal Electricity Supply Bidding by Markov Decision Process
Project Management Operations -- Prof. Juran.
CASE − Cognitive Agents for Social Environments
CS 416 Artificial Intelligence
CS 416 Artificial Intelligence
Reinforcement Nisheeth 18th January 2019.
Intelligent Systems (AI-2) Computer Science cpsc422, Lecture 7
Dr. Arslan Ornek MATHEMATICAL MODELS
Reinforcement Learning (2)
Intelligent Systems (AI-2) Computer Science cpsc422, Lecture 7
Presentation transcript:

4 th International Conference on Service Oriented Computing Adaptive Web Processes Using Value of Changed Information John Harney, Prashant Doshi LSDIS Lab, Dept. of Computer Science, University of Georgia

Web Process Composition Traditional Web process compositions assume static environments Supply Chain Process Start Finish InvokeResponse Spot Market Service Rate of Order Satisfaction Preferred Supplier Service Rate of Order Satisfaction Other Supplier Service Rate of Order Satisfaction Inventory Service Rate of Order Satisfaction ResponseInvoke

Web Process Composition Many environments are dynamic Supply Chain Process Start Finish InvokeResponse Spot Market Service Rate of Order Satisfaction Preferred Supplier Service Rate of Order Satisfaction Other Supplier Service Rate of Order Satisfaction Inventory Service Rate of Order Satisfaction  ResponseInvoke Inventory satisfaction rate decreases Preferred Supplier may be better choice

Optimal Web Process Composition Underlying objective –Web process optimality Depends on how accurately the environment is captured Requires finding any changes that may have occurred

Motivating Scenario – Supply Chain

How does process environment change? –Example: Supply Chain (Inventory service) Rate of satisfaction of a supplier service –Eg Inventory satisfaction rate decreases or increases Cost of using a service –Cost of the Inventory service decreases or increases Other parameters (response time, QoS, etc)

Possible Adaptation Approaches Do Nothing (Ignore the changes) –Advantages Simple No additional cost or computational overhead of adaptation –Disadvantages Sub-optimal Web process –Web process can do better

Possible Adaptation Solutions Query a random provider for relevant information (eg. Inventory) –Advantages Up-to-date knowledge of queried service provider Performs no worse than “do nothing” strategy –Disadvantages Querying for information not free Paying for information that may not be useful –Information may not change Web process

Overview of Our Approach VOC – Value of Changed Information –Decides if obtaining information is: Useful –Will it induce a change in optimality of Web process? Cost-efficient –Is the information worth the cost of obtaining it? Extension of VOI (Value of Information)

Overview of Our Approach VOC –Measures how “badly” the current process is performing in changed environment –Defined as the difference between: Expected performance of the old process in the changed environment Expected performance of the best process in the changed environment

Web Process Composition Using MDPs Markov Decision Processes (MDP) (see JWSR 05) –Definition: M = ( S, A, T, C ) S : States, A : Actions, Actions may be non-deterministic T : Transition function, States are fully observable S x A  (S) C: Cost function S x A  Real Perform stochastic optimization using Dynamic Programming Value function heuristic : Optimal Policy  n : S  A –(Minimize expected cost)

Web Process Composition Using MDPs S : Feature-based state space using propositions –E.g. Mftg. Inventory Availability  Yes|No|Unknown A : WS invocations –E.g. Check Mftg. Inventory Status Check Preferred Supplier Status T : An estimate of the “ground truth” probabilities –E.g. T( Mftg. Inventory Avail = Yes | Check Mfg. Inventory Status, Mftg. Inventory Availability = Unknown ) = 0.33 C : Costs may be obtained through costing analysis Π * : Determines which service to invoke at a particular state

Formalizing VOC Supply Chain Example –Querying Transition function T (satisfaction rate of suppliers in supply chain) –Changed Transition function – T ’ (.|a,s ’ ) –Current Policy Value - V π (s|T ’ ) –Best Policy Value - V π* (s|T ’ )

Formalizing VOC Actual service parameters are not known –Must average over all possible revised parameters –We use a belief of revised values Could be learned over time

Manufacturer’s Beliefs Example - Beliefs of Order Satisfaction

Adaptive Web Process Composition … Prov 1Prov 2Prov n VOC Keep current policy Query Provider Re-solve policy if needed 1. Calculate VOC for each service provider involved in Web process 2. Find provider whose changed parameter induces the greatest change in policy (VOC*) 3. Compare VOC* to cost of querying VOC* < Cost of Querying VOC* > Cost of Querying *

Our Services Oriented Architecture

Empirical Results Simulated volatile Supply Chain & Patient Transfer scenarios for: –Do Nothing keeping the same process –Query random provider Obtaining information from one provider at each state Reformulate composition => Resolve policy –VOC VOC for determining if query is needed Reformulate composition if need be

Empirical Results Measured the average process cost over a range of query cost values –VOC queries selectively -- query random strategy cost grows at a larger rate than VOC –VOC performs no worse that the do nothing strategy Supply Chain Web ProcessPatient Transfer Web Process

Discussion Web Process environments are dynamic –Processes must adapt to changes in environment to remain optimal –Obtaining the revised information is crucial but may be costly VOC approach –Obtains revised information expected to be useful –Avoids unnecessary queries

Future Work VOC calculations are computationally expensive –Knowledge of service parameter guarantees may be used to eliminate unnecessary VOC calculations

Thank you Questions