1. 1 Only Valuable Experts Can be Valued Moshe Babaioff, Microsoft Research, Silicon Valley Liad Blumrosen, Hebrew U, Dept. of Economics Nicolas Lambert, Stanford GSB Omer Reingold, Microsoft Research, Silicon Valley and Weizmann.

2. 2 Probabilities of Events Often, estimating probabilities of future events is important. Examples: –Weather: probability of rain tomorrow –Online advertising: what is the click probability of the next visitor on our web-site? –What % of Toyota cars is defective? –Many applications in financial markets.

3. 3 Contracts and Screening Uncertain about the probability of a future event. Claims he knows this probability. Averse to uncertainty, is willing to pay $$$ to reduce it. May be uninformed and pretend to be informed to get $$$… A decision maker (Alice)An expert (Bob)

4. 4 Contracts and Screening Goal: screen experts. That is, design contracts such that: –Informed experts will: 1.accept the contract 2.reveal the true probability. –Uninformed experts will reject the contract. Contracts can be based on outcomes only: True probabilities are never revealed.

5. 5 This work We characterize settings where Alice can separate good experts from bad experts. We discuss what is a “valuable” expert, and its relation to screening of experts.

6. 6 Outline Model With prior: –An easy impossibility result –A positive result No priors Extensions

7. 7 Model (1/4) Ω - Finite set of outcomes. p - A (true) distribution over Ω. –Unknown to Alice Φ - The set of possible distributions. –Φ may be restricted, examples to come… Bayesian assumptions: Prior f on Φ. f is known to Alice, Bob. No Prior on Φ. In the beginning of this talk Later….

8. 8 Model (2/4) Contract: π(q,ω) payment to Bob when reporting q when outcome is ω. –Bob is risk neutral. Bob’s expected payment depends on what he knows: –Informed: π(q,p) = E ω~p [ π(q,ω) ] –Uninformed: E p~f E ω~p [ π(q,ω) ] = π(q,E[p]) Reported probability Realized outcome Notation: E[payment] upon reporting q when the true probability is p.

9. 9 Model (3/4) Bob δ-accepts the contract if he has a report q with payment > δ –Otherwise, we say that Bob δ-rejects. For avoiding handling ties, we aim that for δ a > δ r : –an informed Bob will δ a -accept –an uninformed Bob will δ r –reject δaδa δrδr 0

10. 10 Model (4/4) We actually study a more general model: –experts are ε-informed. –Mixed strategies are allowed. This talk: perfectly informed, pure strategies.

11. 11 Example A binary event: Ω = {, } Pr( ) = α p  Φ = { (α, 1- α) | α  [0,1] } Alice does not know p. –Knows, however, that α ~ U[0,1].  Sees an a-priori probability of ½. Bob claims he knows the realization of p. Contract: Bob reports α. Is paid according to or.

12. 12 Main message The ability to screen experts closely relates to the structure of Φ –Roughly, on whether Φ is convex or not. If all possible experts are “valuable” to Alice, then screening is possible.

13. 13 An Easy Impossibility Reason: when the true probability is E[p] a true expert accepts the contract. Proof: o Informed experts always accept the contract π. That is, for all p, we have π(p,p) > δ a. o Then, an uninformed agent can get > δ a by reporting E[p]: E p~f E ω~p [π(E[p],ω)] =π(E[p],E[p]) > δ a Proposition: screening is impossible.

14. 14 Valuable Experts So screening is impossible when E[p]  Φ. But experts knowing that the true distribution is E[p] are not really valuable to Alice. In the binary-outcome example: −Alice’s prior is U[0,1], so she believes that Pr( ) = ½. −An expert knowing that p=½ is not that helpful… What if all experts are “valuable”?

15. 15 Possibility result Theorem: when E[p] is not in Φ (and Φ is closed), screening is possible. –When all experts are valuable, we can screen... Immediate questions: −Is non-convex Φ natural? −Can we expect Φ not to contain E[p]? E[p] Φ

16. 16 Non-convex Φ: examples Example 1: a coin which is either fair (p=1/2) or biased (p=3/4) –For any (non-trivial) prior, E[p] not in Φ. Example 2: many standard distributions are not closed under mixing. –E.g., uniform, normal, etc. 1/23/4

17. 17 Non-convex Φ: examples Example 3: The binary outcome example. But now, Alice observes two samples. –For example, we wish to know the failure rate in cars, and we thoroughly check 2 random Toyotas.

18. 18 Non-convex Φ: examples Ω = { (, ), (, ), (, ), (, ) } Φ = { ( α 2, α(1- α), (1- α)α, (1-α) 2 ) } α  [0,1] Example 3: The binary outcome example. But now, Alice observes two samples.  For every prior, E[p] is not in Φ, and thus screening is possible. Φ is not convex! −Moreover, for every p,p’, the convex combination of the above vectors is not in Φ.

19. 19 Possibility result Theorem: when E[p] is not in Φ (and Φ is closed), screening is possible. Φ One definition before proceeding to the proof…

20. 20 Scoring rules Scoring rules: –Contracts that elicit distributions from experts. S(q,ω) = payment for an expert reporting q when the realized outcome is ω. –A scoring rule is strictly proper if the expert is always strictly better off by reporting the true distribution. –Strictly proper scoring rules are known to exist [Brier ‘50,, Good ’52, Savage ‘71,…] We want, in addition, to screen good experts from bad.

21. 21 Possibility result Theorem: when E[p] is not in Φ (and Φ is closed), screening is possible. Φ Proof: Let s be some strictly proper scoring rule. −On the full probability space The following contracts screens experts: We need to show: 1. An informed expert δ a -accepts. 2. An uninformed expert δ r -rejects.

22. 22 Possibility result Theorem: when E[p] is not in Φ (and Φ is closed), screening is possible. Φ Proof: Let s be some strictly proper scoring rule. On the full probability space The following contracts screens experts: An informed expert reporting the truth p gains: ≥1

23. 23 Possibility result Theorem: when E[p] is not in Φ (and Φ is closed), screening is possible. Φ Proof: Let s be some strictly proper scoring rule. The following contracts screens experts: Since s is strictly proper, for every q: An uninformed expert will gain:

24. 24 Outline Model With prior: –An easy impossibility result –A positive result No priors Extensions

25. 25 Related work [Olszewski & Sandroni 2007] studied a similar model: –A binary event with unknown probability p. –No priors: An uninformed expert accepts a contract if it is good in the worst case. Theorem [O&S]: –Use min-max theorems. –The probability space is convex. All informed experts accept a contract An uninformed expert also accepts it

26. 26 Valuable Experts: No Prior We claim: invaluable agents are also behind this impossibility. –But what is a valuable agent without priors? What are Alice’s utility function and actions? –A(p) : Alice’s action when she knows p. –U( A(p),p ): utility maximizing actions. Theorem: Φ is convex (and closed) There exists p such that U(A(p),p)=U(A(“reject”),p) no prior on Φ Interpretation: if Φ is convex then some informed expert is not valuable.

27. 27 No prior: positive result We have an analogues positive result for the no-prior case: non-convex Φ  screening is possible. (we use the with-prior positive result in the proof)

28. 28 Outline Model With prior: –An easy impossibility result –A positive result No priors Extensions

29. 29 Extension: forecasting A related line of research is forecasting: –An unbounded sequence of events. –An expert provides a forecast before each event occurs. –Goal: test the expert.

30. 30 Forecasting: related work Negative results are known: –Informed experts pass the test  uninformed experts can do it too. [e.g., Foster & Vohra ‘97, Fudenberg & Levine ‘99] –When forecasting is possible, decisions can be delayed arbitrarily. [Olszewski & Sandroni ‘09] Some works around this impossibility: –[Olszewski & Sandroni ‘09] show a counter-example by constructing non-convex set of distributions. –[Al-Najjar & Sandroni & Smorodinsky & Weinstein ‘10] Describe a class of distribution such that decisions can be made in time. The relevant class of distributions also admits non-convexities.

31. 31 Extension: forecasting We extend our approach to forecasting settings. –In the works. We characterize conditions on the set of distributions that allow expert testing. Analysis is more involved, but the ideas are similar. –Results relate to the convexity of Φ. –For example: two samples at each period enable testing.

32. 32 Summary A decision maker want to hire an expert. –For learning the probability of some future event. –The expert may be a charlatan. Can the decision maker separate good experts from bad ones? We characterize the settings where such screening is possible. –With or without priors on Φ. We design screening contracts.

33. 33 Thanks!