1. © 2006 THE UNIVERSITY OF TEXAS AT AUSTIN 1 ART Testbed  Join the discussion group at: http://www.art-testbed.net

2. © 2006 THE UNIVERSITY OF TEXAS AT AUSTIN 2 ART Testbed Questions  Can agents request reputations about themselves?  Can an agent produce an appraisal without purchasing opinions?  Does the Testbed assume a common representation for reputations?  Does the Testbed prevent agents from winning via action-planning skills, as opposed to trust-modeling skills?  What if an agent can’t or won’t give a reputation value?  Why does it cost more to generate an accurate opinion than an inaccurate one?  Why not have a centralized reputation broker?  Isn’t it unrealistic to assume a true value of a painting can be known? Is art appraisal a realistic domain?  Why not design an incentive-compatible mechanism to enforce truth-telling?

3. © 2006 THE UNIVERSITY OF TEXAS AT AUSTIN 3 “Really Good” ART Testbed Questions  Is there a consensus on the definitions of “trustworthiness” and “reputation”?  How can collusion be avoided?  Is truth-telling a dominant strategy?  Will the system reach equilibrium, at which point reputations are no longer useful?  What happens if client fee (100), opinion cost (10), and reputation cost (1) are changed?  Do any equilibria exist?  What happens when agents enter or leave the system?  When will agents seek out reputations?  Space of experiments is underexplored—that’s a good thing!

4. © 2006 THE UNIVERSITY OF TEXAS AT AUSTIN 4 Questions about the Paper  What is a “trust model”?  How does q-learning work? How related to reinforcement learning? How do rewards tie in?  What is lambda?  How can experience- and reputation-based learning be combined to overcome the weaknesses of each (intermediate lambda values)?  What about different combinations of (more sophisticated) agents in a game?  Why the assumptions chosen? They seem too extreme.  Reputation decisions weren’t examined very well.

5. The Agent Reputation and Trust Testbed: Experimentation and Competition for Trust in Agent Societies Karen K. Fullam 1, Tomas B. Klos 2, Guillaume Muller 3, Jordi Sabater 4, Andreas Schlosser 5, Zvi Topol 6, K. Suzanne Barber 1, Jeffrey S. Rosenschein 6, Laurent Vercouter 3, and Marco Voss 5 1 Laboratory for Intelligent Processes and Systems, University of Texas at Austin, USA 2 Center for Mathematics and Computer Science (CWI), Amsterdam, The Netherlands 3 Ecole Nationale Superieure des Mines, Saint-Etienne, France 4 Institute of Cognitive Science and Technology (ISTC), National Research Council (CNR), Rome, Italy 5 IT Transfer Office, Darmstadt University of Technology, Darmstadt, Germany 6 Multiagent Systems Research Group—Critical MAS, Hebrew University, Jerusalem, Israel

6. The Agent Reputation and Trust Testbed, 2006 Appraiser Agent Client Client Share Opinions and Reputations Appraiser Agent Testbed Game Rules Agents function as art appraisers with varying expertise in different artistic eras. For a fixed price, clients ask appraisers to provide appraisals of paintings from various eras. If an appraiser is not very knowledgeable about a painting, it can purchase "opinions" from other appraisers. Appraisers can also buy and sell reputation information about other appraisers. Appraisers whose appraisals are more accurate receive larger shares of the client base in the future. Appraisers compete to achieve the highest earnings by the end of the game.

7. The Agent Reputation and Trust Testbed, 2006 Step 1: Client and Expertise Assignments n Appraisers receive clients who pay a fixed price to request appraisals n Client paintings are randomly distributed across eras n As game progresses, more accurate appraisers receive more clients (thus more profit)

8. The Agent Reputation and Trust Testbed, 2006 Step 2: Reputation Transactions n Appraisers know their own level of expertise for each era n Appraisers are not informed (by the simulation) of the expertise levels of other appraisers n Appraisers may purchase reputations, for a fixed fee, from other appraisers n Reputations are values between zero and one Might not correspond to appraiser’s internal trust model Serves as standardized format for inter-agent communication

9. The Agent Reputation and Trust Testbed, 2006 Step 2: Reputation Transactions Provider Requester Request Accept Payment Reputation Requester sends request message to a potential reputation provider, identifying appraiser whose reputation is requested Potential reputation provider sends “accept” message Requester sends fixed payment to the provider Provider sends reputation information, which may not be truthful

10. The Agent Reputation and Trust Testbed, 2006 Step 3: Opinion Transactions n For a single painting, an appraiser may request opinions (each at a fixed price) from as many other appraisers as desired n The simulation “generates” opinions about paintings for opinion-providing appraisers n Accuracy of opinion is proportional to opinion provider’s expertise for the era and cost it is willing to pay to generate opinion n Appraisers are not required to truthfully reveal opinions to requesting appraisers

11. The Agent Reputation and Trust Testbed, 2006 Step 3: Opinion Transactions Provider Requester Request Certainty Payment Opinion Requester sends request message to a potential opinion provider, identifying painting Potential provider sends a certainty assessment about the opinion it can provide - Real number (0 – 1) - Not required to truthfully report certainty assessment Requester sends fixed payment to the provider Provider sends opinion, which may not be truthful

12. The Agent Reputation and Trust Testbed, 2006 Step 4: Appraisal Calculation n Upon paying providers and before receiving opinions, requesting appraiser submits to simulation a weight (self- assessed reputation) for each other appraiser n Simulation collects opinions sent to appraiser (appraisers may not alter weights or received opinions) n Simulation calculates “final appraisal” as weighted average of received opinions n True value of painting and calculated final appraisal are revealed to appraiser n Appraiser may use revealed information to revise trust models of other appraisers

13. The Laboratory for Intelligent Processes and Systems Electrical and Computer Engineering The University of Texas at Austin http://www.lips.utexas.edu Karen K. Fullam 2006 ART Testbed Competition Results

14. © 2006 THE UNIVERSITY OF TEXAS AT AUSTIN 14 Competition Organization  “Practice” Competition Spanish Agent School, Madrid, April 2006 12 participants  International Competition AAMAS, Hakodate, May 2006 Preliminary Round  13 Participants  5 games each Final Round  5 Finalists  10 games with all finalists participating

15. © 2006 THE UNIVERSITY OF TEXAS AT AUSTIN 15 Bank Balances Iam achieves highest bank balances

16. © 2006 THE UNIVERSITY OF TEXAS AT AUSTIN 16 Opinion Purchases Joey and Neil do not purchases opinions Sabatini purchases the most opinions

17. © 2006 THE UNIVERSITY OF TEXAS AT AUSTIN 17 Opinion Earnings Sabatini and Iam provide the most opinions Neil and Frost do not provide many opinions

18. © 2006 THE UNIVERSITY OF TEXAS AT AUSTIN 18 Opinion Sensing Costs Iam invests the most in opinions it generates

19. © 2006 THE UNIVERSITY OF TEXAS AT AUSTIN 19 Expertise vs. Bank Balance Iam’s average expertise was not significantly higher than others’ Greater Expertise

20. The Laboratory for Intelligent Processes and Systems Electrical and Computer Engineering The University of Texas at Austin http://www.lips.utexas.edu Karen K. Fullam K. Suzanne Barber Learning Trust Strategies in Reputation Exchange Networks

21. © 2006 THE UNIVERSITY OF TEXAS AT AUSTIN 21 Trust Decisions in Reputation Exchange Networks  Agents perform transactions to obtain needed resources Transactions have risk because partners may be untrustworthy Agents must learn whom to trust and how trustworthy to be  When agents can exchange reputations Agents must also learn when to request reputations and what reputations to tell Agents’ trust decisions affect each other  Difficult to learn each decision independently Resources (goods, services, information) How trustworthy should I be? Reputations Which reputations should I listen to? What reputations should I tell? Should I trust? If I lie to others that C is bad, can I monopolize C’s interactions? If I cheat A, and A tells B, will it hurt my interactions with B?

22. © 2006 THE UNIVERSITY OF TEXAS AT AUSTIN 22 Enumerating Decisions in a Trust Strategy Truster Trustee Agent Role Transaction Fundamental Reputation How trustworthy should I be? Should I trust? Should I tell an accurate reputation? Should I believe this reputation? Truster Trustee Num agents = a Num transaction types = e Num choices/decision = n How to learn the best strategy with so many choices? If these decisions affect each other, there are possible strategies!

23. © 2006 THE UNIVERSITY OF TEXAS AT AUSTIN 23 Reinforcement Learning Select a strategy Strategy feedback influences expected reward Strategies with higher expected rewards are more likely to be selected Strategy Expected Reward A B C D

24. © 2006 THE UNIVERSITY OF TEXAS AT AUSTIN 24 Learning In Reputation Exchange Networks StrategyExpected Reward Tr(A),Tr(B),Tr(C)…... ⌐Tr(A),Tr(B),Tr(C)… Tr(A), ⌐ Tr(B),Tr(C)… ⌐ Tr(A), ⌐ Tr(B),Tr(C)… Tr(A),Tr(B), ⌐ Tr(C)… ⌐ Tr(A),Tr(B), ⌐ Tr(C)… ⌐ Tr(A), ⌐ Tr(B), ⌐ Tr(C)… Tr(A), ⌐ Tr(B), ⌐ Tr(C)… DecisionExpected Reward Tr(A) ⌐Tr(A) DecisionExpected Reward Tr(B) ⌐Tr(B) DecisionExpected Reward Tr(C) ⌐Tr(C) Removing interdepend- encies makes each decision in the strategy learnable Use the ART Testbed as a case study Because decisions are interdependent, there are. possible strategies!

25. © 2006 THE UNIVERSITY OF TEXAS AT AUSTIN 25 Many Interdependent Decisions Accuracy of Opinion Requester’s appraisals Reputation Requester’s reputation costs Opinion Requester’s client revenue Other Appraisers’ client revenue Number of requests received by Opinion Provider Opinion Provider’s opinion revenue Opinion Provider’s opinion order costs Opinion Requester’s opinion costs Accuracy of Reputation Requester’s trust models Number of requests received by Reputation Provider Reputation Provider’s reputation revenue Opinion Provider Opinion Requester Reputation Provider Reputation Requester When Reputation Requester is Opinion Requester

26. © 2006 THE UNIVERSITY OF TEXAS AT AUSTIN 26 Opinion Requester Feedback Opinion Requester’s client revenue Opinion Requester’s opinion costs Opinion Requester Make sure to bundle: average all decisions for same opinion provider and era Assume: Client revenue feedback is wholly attributed to Opinion Requester decision Divide revenue (client revenue) among opinions based on opinion accuracy Opinion Requester’s client revenue Opinion Requester’s opinion costs Reward = – Client Revenue Opinion Purchase Costs

27. © 2006 THE UNIVERSITY OF TEXAS AT AUSTIN 27 Opinion Provider Feedback  Bundle all n opinions by requesting agent and era (same c g decision)  Reward for this c g decision: Other Appraisers’ client revenue Opinion Provider’s opinion revenue Opinion Provider’s opinion order costs Opinion Provider Assume: Client revenue is not related to Opinion Provider decision Bundle all decisions for same requesting agent and era Reward = – Opinion Selling Revenue Opinion Generating Costs Opinion Provider’s opinion revenue Opinion Provider’s opinion order costs

28. © 2006 THE UNIVERSITY OF TEXAS AT AUSTIN 28 Reputation Provider Feedback  Parameterize reputation accuracy:  Reward for this  value: (n = 1 if requested; n = 0 for each timestep until reputation is requested again) Other Appraisers’ client revenue Reputation Provider’s reputation revenue Assume: Client revenue is not related to Reputation Provider decision  Identify reputations according to requesting agent, subject agent and era (there will be only 1 in each “bundle”) Parameterize reputation accuracy Reward = Reputation Selling Revenue Reputation Provider’s reputation revenue Reputation Provider

29. © 2006 THE UNIVERSITY OF TEXAS AT AUSTIN 29 Reputation Requester Reputation Requester Feedback Reputation Requester’s reputation costs Opinion Requester’s client revenue Opinion Requester’s opinion costs determines influence of: past experience vs. reputations in deciding to purchase opinions = 0: Past experience only  Opinion-requesting decision  No reward for requesting reputations = 1: Reputations only  Reputation-requesting decision  Full reward for requesting reputations Opinion Requester’s client revenue Opinion Requester’s opinion costs Opinion Requester: which opinions to purchase Opinion Requester Reward = – Opinion Requester Reward Reputation Purchase Costs ( )