1. Ben-Gurion University of the Negev Department of Computer Science Distributed Search by Agents with Personal Preferences Alon Grubshtein Lessons learnt from applying distributed constraint reasoning to realistic agents with personal preferences

2. Ben-Gurion University of the Negev Department of Computer Science Before we begin…

3. Ben-Gurion University of the Negev Department of Computer Science Multi Agent Systems Constraint Reasoning Distributed Computing Distributed Constraint Reasoning In this talk:

4. Ben-Gurion University of the Negev Department of Computer Science Sometime back in 2006 Check out this great phone I got I can use it to work on my calendar!!! Who needs a computer with such phones? You can even write programs for it… Lets write a distributed agent to automate meeting coordination

5. Ben-Gurion University of the Negev Department of Computer Science Constraint reasoning (centralized) A Constraint Reasoning problem: Variables Domains Constraints (relations) A solution concept (target objective)

6. Ben-Gurion University of the Negev Department of Computer Science Examples

7. Ben-Gurion University of the Negev Department of Computer Science Whats in a constraint? A satisfying assignment A minimal cost assignment

8. Ben-Gurion University of the Negev Department of Computer Science Constraint algorithms How do we find a solution? Enumerate feasible outcomes Backtracking / Branch and Bound Intelligent backtracking Pre processing, forward checking and heuristics Local search algorithms

9. Ben-Gurion University of the Negev Department of Computer Science From centralized to distributed problem The problem itself is distributed across computational nodes – agents: Privacy Difficulty

10. Ben-Gurion University of the Negev Department of Computer Science Constraint reasoning (distributed) Distributed Constraint Reasoning (DCR) problem: Agents Variables Domains Constraints (relations) DCSP / DCOP

11. Ben-Gurion University of the Negev Department of Computer Science From centralized to distributed Computation on separate entities Communication via messages knows only a small portion of the problem Each agent knows only a small portion of the problem Allows for parallel computation DISTRIBUTED =/= PARALLEL

12. Ben-Gurion University of the Negev Department of Computer Science DCR algorithms

13. Ben-Gurion University of the Negev Department of Computer Science Local Search for real problems Computationally hard Simplistic myopic algorithms (local search/adaptive heuristics) Example, DSA: 1.Pick a random assignment 2.While (stop condition): a.Send assignment to all neighbors (receive) b.If can improve local state by changing assignment: change with probability p

14. Ben-Gurion University of the Negev Department of Computer Science A simple MAS example Coordinating a meeting (e.g., seminar): ME Two alternatives: Morning or Evening More participants – better Prof. Lynn does not care when If students disagree - morning Alice prefers morning Anna prefers evening Prof. Lynn Alice Anna 51 02 M M E E Alice Anna 30 24 M M E E Alice Anna 5, 31, 0 0, 22, 4 M M E E Alice Anna

15. Ben-Gurion University of the Negev Department of Computer Science Solving as a DCOP AliceAliceAnnaAnna 5, 31, 0 0, 22, 4 M M E E Alice Bob 81 26 M M E E Alice Bob What if students cant/wont communicate preferences? Alice Anna M MM E EE Cost: 8126

16. Ben-Gurion University of the Negev Department of Computer Science Standard model solutions Easiest solution: Disclose preferences An alternate approach: Add unary constraints Can prove that this approach will fail on some instances Problem: Can prove that this approach will fail on some instances

17. Ben-Gurion University of the Negev Department of Computer Science How its done these days The PEAV formulation: A1A1 A2A2 x1x1 x2x2 x21x21 x12x12 xy a 36 b 75 xy a 41 b 28 mirror variables hard constraints x12x12 x2x2 x1x1 x21x21 Modified search space Modified search space Cant be used with many local search algorithm! Cant be used with many local search algorithm! Requires more space Requires more space

18. Ben-Gurion University of the Negev Department of Computer Science Introducing ADCOPs

19. Ben-Gurion University of the Negev Department of Computer Science ADCOPs ADCOPs: At least as expressive as existing model Succinct representation Used with existing local search algorithms Search can be improved by introducing cooperation/coordination

20. Ben-Gurion University of the Negev Department of Computer Science ADCOP Local Search (quality) DCOPADCOP

21. Ben-Gurion University of the Negev Department of Computer Science Multi Agent Systems Constraint Reasoning Distributed Computing Distributed Constraint Reasoning

22. Ben-Gurion University of the Negev Department of Computer Science Rethinking agents joint objective Difference in best and worst gains – Meeting Scheduling Problem

23. Ben-Gurion University of the Negev Department of Computer Science Agreeing on an outcome (what is a fair solution?)

24. Ben-Gurion University of the Negev Department of Computer Science Graphical Games ADCOPs are Games played on a Graph Graphical Games Closely related to Graphical Games ADCOPs: No knowledge assumed Agents are cooperative An even more succinct representation DCR techniques game theoretic Can use DCR techniques to solve a game theoretic multi agent problem! 5, 31, 0 0, 22, 4 M M E E Alice Anna

25. Ben-Gurion University of the Negev Department of Computer Science Asynchronous Nash BackTracking (ANT) Transform a MAS to a Distributed Constraint Problem A distributed, asynchronous, non- binary, asymmetric search Two symmetric constraints Three asymmetric constraints

26. Ben-Gurion University of the Negev Department of Computer Science ANT

27. Ben-Gurion University of the Negev Department of Computer Science Multi Agent Systems Constraint Reasoning Distributed Computing Distributed Constraint Reasoning

28. Ben-Gurion University of the Negev Department of Computer Science The quality of a stable solution A stable solution is not necessarily a good one… Why is that? Competitive solution for cooperative agents? A 2 \ A 1 CooperateDefect Cooperate 4,44,46,06,0 Defect 0,60,61,11,1

29. Ben-Gurion University of the Negev Department of Computer Science Agreeing on an outcome (what is a fair solution?) Utilitarian, Egalitarian, Leximin,… Stable points Stable points: Nash (pure/mixed), Bayesian, Strong, Correlated, …

30. Ben-Gurion University of the Negev Department of Computer Science A different approach assume cooperation but try to incentivize agents by examining personal goals Cost of Cooperation Baseline search

31. Ben-Gurion University of the Negev Department of Computer Science The Cost of Cooperation (CoC) criteria: The difference in an agents gain from the worst equilibrium (from its point of view) outcome and from cooperatively solving the problem Possible solutions U 2 (x) U 1 (x) Pareto front Optimal solution (max sum) Nash equilibrium solutions Non positive CoC solutions

32. Ben-Gurion University of the Negev Department of Computer Science u 1 =med C2C2C2C2 A simple P2P example a1a1 a1a1 a8a8 a8a8 a2a2 a2a2 a3a3 a3a3 a4a4 a4a4 a5a5 a5a5 a6a6 a6a6 a7a7 a7a7 C1C1C1C1 F F S S F F u 1 =low u 2 =high F F S S S S Agents only interact with neighbors (unknown topology) An agents gain is lowered when exerting resources on sharing ( S ) Gain is maximized if an agent can free ride the efforts of other agents ( F ) Gain is lowest if no one shares Agents only interact with neighbors (unknown topology) An agents gain is lowered when exerting resources on sharing ( S ) Gain is maximized if an agent can free ride the efforts of other agents ( F ) Gain is lowest if no one shares

33. Ben-Gurion University of the Negev Department of Computer Science Competitive and Cooperative solutions a1a1 a1a1 a8a8 a8a8 a2a2 a2a2 a3a3 a3a3 a4a4 a4a4 a5a5 a5a5 a6a6 a6a6 a7a7 a7a7 S S S S F F F F F F F F F F F F 0.30.3 11 1 11 1 Cooperative Solution a1a1 a1a1 a8a8 a8a8 a2a2 a2a2 a3a3 a3a3 a4a4 a4a4 a5a5 a5a5 a6a6 a6a6 a7a7 a7a7 S S S S F F F F F F F F F F F F 0.3 0.3 11 0 0 0 0 A Bayesian stable solution (possible)

34. Ben-Gurion University of the Negev Department of Computer Science Cost of Cooperation solution An improvement can be guaranteed (proved) for a set of interactions! a1a1 a1a1 a8a8 a8a8 a2a2 a2a2 a3a3 a3a3 a4a4 a4a4 a5a5 a5a5 a6a6 a6a6 a7a7 a7a7 S S S S F F F F F F F F 0.3 0.3 11 0 0 S S S S 0.3 0.3

35. Ben-Gurion University of the Negev Department of Computer Science Applied to network games 35 ADCOP (CoC) Maximizing utilities

36. Ben-Gurion University of the Negev Department of Computer Science Multi Agent Systems Constraint Reasoning Distributed Computing Distributed Constraint Reasoning

37. Ben-Gurion University of the Negev Department of Computer Science Limits of the CoC approach So far we have seen several solutions: Fully cooperative (Utilitarian) Stable (Epsilon Nash Equilibrium) A combination: Non positive Cost of Cooperation However… A 2 \ A 1 LeftRight Up 2,52,54,14,1 Down 6,16,10,30,3 Mixed NE: (1/2,1/3) Gain: (3, 7/3) NO Cost of Cooperation solution!

38. Ben-Gurion University of the Negev Department of Computer Science A framework for partial cooperation Agents gain is different Do not improve cooperatively Define cooperation with respect to some baseline solution Agents must agree on the baseline (may need to apply a simple search algorithm).

39. Ben-Gurion University of the Negev Department of Computer Science Modes of cooperation Define modes of cooperation within an Interaction Process: Non-Cooperative (NC) Non-Cooperative (NC) – agents are driven by their own goals and act rationally. Can serve as a baseline solution Guaranteed Personal Benefit (GPB) Guaranteed Personal Benefit (GPB) – agents seek an agreement and may take irrational steps. Pareto Guarantees a Pareto improvement λ -cooperation λ -cooperation – agents agree to a bounded loss from their NC gain (up to some predefined λ )

40. Ben-Gurion University of the Negev Department of Computer Science Local Search and Partial Cooperation Maintain threshold/guarantee: 1.Incorporate with distributed anytime Can use any LS algorithm Focus on exploration 2.Tailor an algorithm maintain invariant (begins in a legal state)

41. Ben-Gurion University of the Negev Department of Computer Science Evaluation Three key parameters: 1.Compromise levels (lambda) 2.Agents degree 3.Costs distribution

42. Ben-Gurion University of the Negev Department of Computer Science Multi Agent Systems Constraint Reasoning Distributed Computing Distributed Constraint Reasoning

43. Ben-Gurion University of the Negev Department of Computer Science SUMMARY & CONCLUSIONS

44. Ben-Gurion University of the Negev Department of Computer Science Summary Multi Agent Problem DCSP/DCOP Utilitarian (Minimal sum of costs) Asymmetric Constraints Stable ε- Nash Equilibrium Non positive Cost of Cooperation Partial Cooperation Representation Algorithm Algorithm Objective

45. Ben-Gurion University of the Negev Department of Computer Science Conclusions Three points (up and down the ladder of abstraction): 1.How to model the problem 2.How does the model effect the means to find a solution 3.What is a solution? Rethinking basic assumptions Applying well established models to simple realistic settings can reveal many of its shortcoming

46. Ben-Gurion University of the Negev Department of Computer Science Journal publications: Arnon Netzer, Alon Grubshtein and Amnon Meisels, Concurrent Forward Bounding, Artificial Intelligence, Vol. 193, pp. 186-216, 2012. Roie Zivan, Alon Grubshtein and Amnon Meisels, Hybrid Search for Dynamically changing CSPs, Constraints, special issue on constraint satisfaction for planning and Scheduling, Vol. 16, num. 3, pp. 228-249, 2011. Alon Grubshtein and Amnon Meisels, Cost of Cooperation for Scheduling Meetings, Journal of Computer Science and Information System (ComSIS), Vol. 7, num. 3, pp. 551-567, 2010. Conference and workshops publications : Alon Grubshtein and Amnon Meisels, Finding a Nash Equilibrium by Asynchronous Backtracking, 18 th Intl. Conf. on Principles and Practice of Constraint Programming (CP12), pp. 925-940, Quebec city, Canada, Oct. 2012. Alon Grubshtein, Roie Zivan and Amnon Meisels, Partial Cooperation in Multi Agent Local Search, 20 th European Conf. on Artificial Intelligence,pp.378-383, Montpellier France, Aug. 2012 Roie Zivan, Alon Grubshtein, Michal Friedman and Amnon Meisels, Partial Cooperation in Multi Agent Search, (Extended Abstract) Proc. 11 th intern. Conf. Autonom. Agents Multi agent Sys. (AAMAS12), Valencia, Spain. Alon Grubshtein and Amnon Meisels, A Distributed Cooperative Approach for Optimizing a Family of Network Games, Proc. of the 5 th Intern. Symp. on Intelligent Distributed Computing (IDC11), Delft, the Netherlands, pp. 49-62, October 2011. Alon Grubshtein and Amnon Meisels, A Distributed Cooperative Approach for Optimizing a Network Game, Proc. 13 th Intern. Workshop on Dist. Constraints Reasoning (DCR11), Barcelona, Spain, June 2011. Alon Grubshtein, Nir Herschorn, Arnon Netzer, Guy Rapaport, Guy Yaffe and Amnon Meisels, The Distributed Constraints (DisCo) Simulation Tool, Proc. 13 th Intern. Workshop on Dist. Constraints Reasoning (DCR11), Barcelona, Spain, June 2011. Alon Grubshtein and Amnon Meisels, Cooperation Mechanism for a Network Game, Proc. 3 rd Intern. Conf. Agents and AI (ICAART11), Rome, Italy, pp. 336-341, January 2011. Alon Grubshtein, Tal Grinshpoun, Amnon Meisels and Roie Zivan, Local Search for Distributed Asymmetric Optimization, Proc. 9 th intern. Conf. Autonom. Agents Multi agent Sys. (AAMAS10), Toronto, Canada, pp. 1015-1022, May 2010. Arnon Netzer, Amnon Meisels and Alon Grubshtein, Concurrent Forward Bounding for DCOPs, Proc. 12 th Intern. Workshop on Dist. Constraints Reasoning (DCR10) at AAMAS10, Toronto, May 2010. Alon Grubshtein, Nurit Gal-Oz, Tal Grinshpoun, Amnon Meisels and Roie Zivan, Manipulating Recommendation Lists by Global Considerations, Proc. 2 nd Intern. Conf. Agents and AI (ICAART10),pp. 135-142, Valencia, Spain, January 2010. Alon Grubshtein and Amnon Meisels, Cost of Cooperation for Scheduling Meetings, Proc. 3 rd Intern.Symp. Intell. Dist. Comp. (IDC09), Vol. 237, pp. 227-236, Ayia Napa, Cyprus, October 2009. Alon Grubshtein, Tal Grinshpoun, Amnon Meisels and Roie Zivan, Asymmetric Distributed Constraint Optimization, Proc. 11 th Intern. Workshop on Dist. Constraints Reasoning (DCR09) at IJCAI-09, Pasadena CA, July 2009. Ehud Gudes, Nurit Gal-Oz and Alon Grubshtein, Methods for Computing Trust and Reputation While Preserving Privacy, Proc. Data and App. Security XXIII, 23 rd Ann. IFIP WG 11.3 Working Conf. (DBSEC09), Vol. 5645, pp. 291-298, Montreal, Canada, July 2009. Amir Gershman, Alon Grubshtein, Amnon Meisels and Roie Zivan, Scheduling Meetings by Agents, Proc.7 th intern. Conf. Practice and Theory Auto. Timetabling (PATAT08), Montreal, August 2008.

47. Ben-Gurion University of the Negev Department of Computer Science PEAV search space The PEAV search space – Generates new local minima! (new PSNE) – Hard constraints prevent a single agent from performing any assignment replacement with positive reduction after reaching the first valid state Ill fitting for many existing local search algorithms Ill fitting for many existing local search algorithms A 1 \A 2 a,xa,yb,xb,y x,a'755 111 y,a'60710864 x,b'61108965 y,b'109565713

48. Ben-Gurion University of the Negev Department of Computer Science ADCOP Local Search (privacy)

49. Ben-Gurion University of the Negev Department of Computer Science Experimental evaluation: Number of agents with NP-CoC (n=30, k=1) 49

50. Ben-Gurion University of the Negev Department of Computer Science Graph coloring – simple example

51. Ben-Gurion University of the Negev Department of Computer Science DCR algorithms

52. Ben-Gurion University of the Negev Department of Computer Science Solving ADCOPs The naïve approach: Alice Anna M MM E EE Cost: 3024 Anna is ignorant of Alices cost function, and computes the value of the constraint based on her knowledge – i.e., updates the gain to 3 The end result is the assignment with a combined gain of 4. This result is both wrong (the combined gain of is 6) and sub optimal (the combined gain of is 8) 5, 31, 0 0, 22, 4 M M E E Alice Anna