1. 1 Optimizing Decisions over the Long-term in the Presence of Uncertain Response Edward Kambour

2. 2 Key Question  Given a set of choices (product offerings), which one is optimal? u For each choice s Probability of success (sale) s Revenue realized given a success u Revenue Management perspective s Which of the choices will yield the greatest revenue Elicits the best customer response

3. 3 Deterministic world  Customer response to each choice is known and constant u Plug in the known success rate and revenue for each choice and optimize

4. 4 Random World  Customer response is random with known parameter values u For example, success rate is 0.5 or realized revenue is Poisson(7)  How to handle random response u Risk neutral – Expected Response u Risk averse – Expected Utility u Minimax (Minimizes the worst possible outcome) u Maximin (Maximizes the best possible outcome)

5. 5 Random World  Typically don’t know precisely the demand parameters u Estimate the demand s Parametric estimators of unknown parameters s Robust estimators of unknown parameters s Non-parametrics Sample quantiles, Empirical CDF, …

6. 6 Real World  Random response u In either the success rate or the realized revenue  Uncertainty in the demand distribution u Based on earlier observations (expert opinion, similar processes, etc.) u The Bayesian approach quantifies this uncertainty in a directly

7. 7 Effect of Uncertainty  What effect does the uncertainty have with regard to making optimal decisions? u Assuming risk neutrality, should decisions be based only on expected revenue? u Is there potential value in exploring choices that do not yield the best expected revenue as of now? s Quantify value in obtaining additional information, i.e. reducing uncertainty in the demand response

8. 8 Simple Example  Choose either to place a $1 bet on a flip of a coin coming out heads or keep your $1  You estimate that the probability of a head is 0.4  Should you play? u Expected Revenue s Don’t play: 0 s Do play: 1(p) – 1(1-p) = 2p – 1 = 2(0.4) –1 = -0.2

9. 9 Simple Example Solution  Depends

10. 10 Scenario 1  There is no uncertainty u The probability of a head is exactly 0.4 u Assuming that you’re risk neutral, you should not play

11. 11 Scenario 2  You know that the coin is either a 2-headed coin or it is a 2-tailed coin u Pr[2-headed] = 0.4 u Pr[2-tailed] = 0.6 u If you play once s Observe head  2-headed Continue to play and make $1 every time s Observe tail  2-tailed Stop after losing $1

12. 12 Scenario 2 (cont.)  If you could play up to 10 times u Expected Revenue s Don’t play – 0 s Do play Observe the first flip »If it’s heads then play the next 9 times »If it’s tails then quit 0.4(10) - 0.6(1) = 3.4

13. 13 Scenario 3  You estimate that the probability of head is 0.4 plus or minus 0.2 u Playing once no longer yields perfect information s Playing any finite number of times will not yield perfect information u There is a possibility of long-term positive revenue from playing s Head probabilities greater than 0.5 are plausible u Should you play at all? s If so, under what circumstances should you stop playing?

14. 14 Scenario 3 (Bayesian Perspective)  You estimate that the probability of head is 0.4 with uncertainty of 0.1 u Prior mean = 0.4 u Prior standard deviation = 0.1 u Beta prior:  = 9.2,  = 13.8

15. 15 Scenario 3 (cont.)  Let X be the number of heads in any set of n flips of the coin and assume X is binomial distributed (iid Bernoulli trials)  Updated uncertainty in the head probability after the n trials u Beta u  * = 9.2 + X,  * = 13.8 + n – X u Expectation:  */(  *+  *) u Variance:  *  */[(  *+  *)(  *+  *)(  *+  *+1)] u Never know for sure

16. 16 Key Points  Short term expected revenue is not the only thing to consider  There is also value in obtaining additional information about the demand distribution because this information can be used to improve long-term revenue performance u How do we calculate the value of this information in terms of future revenue? u How much experimentation is optimal?

17. 17 One Approach  Evaluate any possible choice at every time u Completely reevaluate after every observation s Due to the prior uncertainty, lose independence because observations share the same unknown parameters s Enumerate all possible combinations of observations and strategies along with their associated probabilities and search for the best u Computationally intensive u Complications with continuous variables

18. 18 More Restrictive Approach  Define long-term in terms of a number of observations u For example, 10 flips of the coin  Allow for a single experiment with a single conclusion u For example, 3 flips of the coin s If 3 heads continue to play over the last 7 s Otherwise stop u Calculate the expected net return from an experiment s Expected Cost – Expected Revenue from additional observations

19. 19 Expected Cost of Experiment  Based on prior distribution on the unknown parameters and the conditional distribution of the customer response (given the parameters) u Calculate the marginal distribution for the data u Calculate the marginal expectation over the trials u Subtract the baseline revenue

20. 20 Expected Revenue from Trial  Expected revenue over the rest of the long-term period given the decision made after the trial u For each choice s Win implies the highest posterior expectation of revenue s Pr[win] s E[return|win] s For the rest of the period (n-m) Expected revenue given win is (n-m)E[return|win] u The expected revenue is then the weighted sum s Pr[Win](n-m)E[return|win] u Subtract the baseline revenue

21. 21 Scenario 3 (revisited)  Set n = 10  Investigate an experiment with m trials  Expected cost of the experiment u.2m  Assume we observed m 1 heads u Updated uncertainty s Beta(  = 9.2 + m 1,  = 13.8 + m – m 1 )

22. 22 Scenario 3 (cont.)  Continue to play if the new expectation on the probability of a head is greater than 0.5 u  * = 9.2 + m 1 > 13.8 + m – m 1 =  *  Expected return on the experiment u Pr[  * >  * ](10-m){2E[  */(  *+  *)|  * >  * ]-1]} + (1- Pr[  * >  * ])(10-m)0 u Note s Non-negative s If you let n   and Pr[  * >  * ] > 0, then the expected return for any finite experiment  

23. 23 Scenario 3 (Details)   * >  * u  9.2 + m 1 > 13.8 + m – m 1 u  2m 1 > 4.6 + 0.5m u  m 1 > 2.3 + 0.5m  m 1 | p ~ Bin(m,p)  m 1 ~BetaBinomial(m,9.2,13.8)   */(  *+  *) = (9.2 + m 1 ) / (23+m) u  E[  */(  *+  *)|  * >  * ] = (9.2 +E[m 1 | m 1 > 2.3 + 0.5m])/(23+m)

24. 24 Scenario 3 (Cont.)  If m = 5 u m 1 > 2.3 + 0.5m = 4.8  s m 1 = 5 s  * = 9.2 + 5 =14.2 s  * = 13.8 + 0 = 13.8 u Expected Return s Pr[  * >  * ](10-m){2E[  */(  *+  *)|  * >  * ]-1]} s Pr[m 1 = 5 ](10-5){2(14.2/28)-1} = 0.0175(5)0.0143 = 0.0012

25. 25 Scenario 3 (Cont.)  Value of the 5 flip experiment u 0.0012 – 0.2(5) = -0.9988  Value of each possible experiment

26. 26 Scenario 4  Equivalent to Scenario 3 except u  = 0.092,  = 0.138 u Uncertainty (prior standard deviation) = 1  Value of Experiment

27. 27 General Rules of Thumb  Larger values of n will lead to longer experiments u The longer the “long-term,” the greater the return on experiments  More uncertainty (larger prior standard deviations) will lead to more experimentation u The less you know, the greater the return on experiments

28. 28 Extensions to the Simple Example Method  There are multiple choices u Each with a current prior distribution for the success probability  The revenue achieved for a success may be random for each choice u The parameters of the distribution are not known

29. 29 Example  Two choices u Choice 1 and Choice 2 s The success probabilities are unknown for both choices s The realized revenue from a success is random Unknown mean and variance u Long term horizon (n) is 50 u Experiment s m 1 trials for choice 1 s m 2 trials for choice 2

30. 30 Choice 1: Current Opinion  Success Rate (Beta distributed) u Expectation: 0.50 u Uncertainty: 0.15  Mean revenue on a success (Normal given the variance) u Expectation: 8.00 u Uncertainty: 0.41  Variance in revenue on a success (Inverted Gamma) u Expectation: 2.07 u Uncertainty: 0.05

31. 31 Choice 2: Current Opinion  Success Rate (Beta distributed) u Expectation: 0.45 u Uncertainty: 0.28  Mean revenue on a success (Normal given the variance) u Expectation: 7.00 u Uncertainty: 2.11  Variance in revenue on a success (Inverted Gamma) u Expectation: 2.11 u Uncertainty: 0.08

32. 32 Results  Search for the best experiment in terms of net revenue over the 50 observations

33. 33 Results  The best trial consists of a sample of 6 observations u 4 with choice 1 u 2 with choice 2  After the trial the choice with the best posterior mean is used for the remaining 44

34. 34 Future Work  Develop efficient optimization routine u Search methods (concave envelope, gradients)  Extend the types of experimentation u Multiple experiments  Further investigate the asymptotics

35. 35