Download presentation

Presentation is loading. Please wait.

Published byAustin Bartlett Modified over 4 years ago

1
Learning to Support Constraint Programmers Susan L. Epstein 1 Gene Freuder 2 and Rick Wallace 2 1 Department of Computer Science Hunter College and The Graduate Center of The City University of New York 2 Cork Constraint Computation Centre

2
Facts about ACE l Learns to solve constraint satisfaction problems l Learns search heuristics l Can transfer what it learns on simple problems to solve more difficult ones l Can export knowledge to ordinary constraint solvers l Both a learner and a test bed l Heuristic but complete: will find a solution, eventually, if one exists l Guarantees high-quality, not optimal, solutions l Begins with substantial domain knowledge

3
Outline l The task: constraint satisfaction l Performance results l Reasoning mechanism l Learning l Representations

4
l Constraint satisfaction problem l Solution: assign a value to every variable consistent with constraints l Many real-world problems can be represented and solved this way (design and configuration, planning and scheduling, diagnosis and testing) The Problem Space Domains A {1,2,3} B {1,2,4,5,6} C {1,2} D {1,3} Constraints A = B A > D C D Variables A, B, C, D BA CD (1 1) (2 2) (2 1) (3 1) (3 2) (1 3) (2 1) (2 3)

5
A Challenging Domain l Constraint solving is NP-hard l Problem class parameters: n = number of variables k = maximum domain size d = edge density (% of possible constraints) t = tightness (% of value pairs excluded) l Complexity peak: values for d and t that make problems hardest l Heavy-tailed distribution difficulty [Gomes et al., 2002] l Problem may have multiple or no solutions l Unexplored choices may be good

6
Finding a Path to a Solution l Sequence of decision pairs (select variable, assign value) l Optimal length: 2n for n variables l For n variables with domain size d, there are (d+1) n possible states Select a variable Assign a value Solution

7
B D=3 No C=2 A=2 … Solution Method Search from initial state to goal Domains A {1,2,3} B {1,2,4,5,6} C {1,2} D {1,3} No D D=1 No D D=1D=3 No Constraints A = B A > D C D BA CD (1 1) (2 2) (2 1) (3 1) (3 2) (1 3) (2 1) (2 3) B=1 A CD A A=1 CD C C=1 D

8
Consistency Maintenance l Some values may initially be inconsistent l Value assignment can restrict domains B=2 … A {1,2} C {1,2} D {1,3} No C {1,2} D No other possibilities Constraints A = B A > D C D B B=1 A A=1 Domains A {1,2,3} B {1,2,4,5,6} C {1,2} D {1,3} BA CD (1 1) (2 2) (2 1) (3 1) (3 2) (1 3) (2 1) (2 3)

9
When an inconsistency arises, a retraction method removes a value and returns to an earlier state Retraction Here! B=2 … A {1,2} C {1,2} D {1,3} No! C {1,2} D B B=1 A A=1 BA CD (1 1) (2 2) (2 1) (3 1) (3 2) (1 3) (2 1) (2 3) Domains A {1,2,3} B {1,2,4,5,6} C {1,2} D {1,3} Wheres the error?

10
… A=2 B {1,2} C {1,2} D {1,2} Variable Ordering l A good variable ordering can speed search A A=1 Domains A {1,2,3} B {1,2,4,5,6} C {1,2} D {1,3} B {1,2} C {1,2} D No BA CD (1 1) (2 2) (2 1) (3 1) (3 2) (1 3) (2 1) (2 3)

11
Value Ordering A good value ordering can speed search too A A=2 Domains A {1,2,3} B {1,2,4,5,6} C {1,2} D {1,3} B {1,2} C {1,2} D {1,3} D D=1 B {1,2} C {1,2} B B=2 C C=2 C {1,2} Solution: A=2, B=2, C=2, D=1 BA CD (1 1) (2 2) (2 1) (3 1) (3 2) (1 3) (2 1) (2 3)

12
Constraint Solvers Know… l Several consistency methods l Several retraction methods l Many variable ordering heuristics l Many value ordering heuristics … but the interactions among them are not well understood, nor is one combination best for all problem classes.

13
Goals of the ACE Project l Characterize problem classes l Learn to solve classes of problems well l Evaluate mixtures of known heuristics l Develop new heuristics l Explore the role of planning in solution

14
Outline l The task: constraint satisfaction l Performance results ACE l Reasoning mechanism l Learning l Representation

15
Experimental Design l Specify problem class, consistency and retraction methods l Average performance across 10 runs l Learn on L problems (halt at 10,000 steps) l To-completion testing on T new problems l During testing, use only heuristics judged accurate during learning l Evaluate performance on l Steps to solution l Constraint checks l Retractions l Elapsed time

16
ACE Learns to Solve Hard Problems l near the complexity peak l Learn on 80 problems l 10 runs, binned in sets of 10 learning problems l Discards 26 of 38 heuristics l Outperforms MinDomain, an off-the-shelf heuristic Steps to solution 2500 1500 1000 500 2000 1 2 3 4 5 6 7 8 Bin # Means in blue, medians in red

17
ACE Rediscovers Brélaz Heuristic l Graph coloring: assign different colors to adjacent nodes. l Graph coloring is a kind of constraint satisfaction problem. l Brélaz: Minimize dynamic domain, break ties with maximum forward degree. l ACE learned this consistently on different classes of graph coloring problems. [Epstein & Freuder, 2001] Color each vertex red, blue, or green so pair of adjacent vertices are different colors.

18
ACE Discovers a New Heuristic l Maximize the product of degree and forward degree at the top of the search tree l Exported to several traditional approaches: Min Domain Min Domain/Degree Min Domain + degree preorder l Learned on small problems but tested in 10 runs on n = 150, domain size 5, density.05, tightness.24 l Reduced search tree size by 25% – 96% [Epstein, Freuder, Wallace, Morozov, & Samuels 2002]

19
Outline l The task: constraint satisfaction l Performance results l Reasoning mechanism l Learning l Representation

20
Constraint-Solving Heuristic l Uses domain knowledge l What problem classes does it work well on? l Is it valid throughout a single solution? l Can its dual also be valid? l How can heuristics be combined? … and where do new heuristics come from?

21
FORR (For the Right Reasons) l General architecture for learning and problem solving l Multiple learning methods, multiple representations, multiple decision rationales l Specialized by domain knowledge l Learns useful knowledge to support reasoning l Specify whether a rationale is correct or heuristic l Learns to combine rationales to improve problem solving [Epstein 1992]

22
An Advisor Implements a Rationale l Class-independent action-selection rationale l Supports or opposes actions by comments l Expresses opinion direction by strengths l Limitedly-rational procedure current problem state Advisor actions

23
Advisor Categories l Tier 1: rationales that correctly select a single action l Tier 2: rationales produce a set of actions directed to a subgoal l Tier 3: heuristic rationales that select a single action

24
Choosing an Action take action yes Tier 1: Reaction from perfect knowledge VictoryT-11T-1n … Decision? begin plan yes no Tier 3: Heuristic reactions T-31T-32T-3m … … Voting take action Tier 2: Planning triggered by situation recognition no P-1P-2P-k … Decision? Current state Possible actions

25
ACEs Domain Knowledge l Consistency maintenance methods: forward checking, arc consistency l Backtracking methods: chronological l 21 variable ordering heuristics l 19 value ordering heuristics l 3 languages whose expressions have interpretations as heuristics l Graph theory knowledge, e.g., connected, acyclic l Constraint solving knowledge, e.g., only one arc consistency pass is required on a tree

26
An Overview of ACE l The task: constraint satisfaction l Performance results ACE l Reasoning mechanism l Learning l Representation

27
What ACE Learns l Weighted linear combination for comment strengths l For voting in tier 3 only l Includes only valuable heuristics l Indicates relative accuracy of valuable heuristics l New, learned heuristics l How to restructure tier 3 l When random choice is the right thing to do l Acquire knowledge that supports heuristics (e.g., typical solution path length)

28
l Learn from trace of each solved problem l Reward decisions on perfect solution path l Shorter paths reward variable ordering l Longer paths reward value ordering l Blame digression-producing decisions in proportion to error l Valuable Advisors weight > baselines Digression-based Weight Learning Select a variable Assign a value Solution digression error

29
Learning New Advisors l Advisor grammar on pairs of concerns l Maximize or minimize l Product or quotient l Stage l Monitor all expressions l Use good ones collectively l Use best ones individually

30
Outline l The task: constraint satisfaction l Performance results ACE l Reasoning mechanism l Learning l Representation

31
No Yes Representation of Experience l State describes variables and value assignments, impossible future values, prior state, connected components, constraint checks incurred, dynamic edges, trees l History of successful decisions l … plus other significant decisions become training examples Is Can beCannot be A12 B 2 C1,2 D1,3 Checks incurred: 4 1 acyclic component: A,C,D Dynamic edges: AD, CD

32
Representation of Learned Knowledge l Weights for Advisors l Solution size distribution l Latest error: greatest number of variables bound at retraction

33
ACEs Status Report l 41 Advisors in tiers 1 and 3 l 3 languages in which to express additional Advisors l 5 experimental planners l Problem classes: random, coloring, geometric, logic, n-queens, small world, and quasigroup (with and without holes) l Learns to solve hard problems l Learns new heuristics l Transfers to harder problems l Divides and conquers problems l Learns when not to reason

34
Current ACE Research l Further weight-learning refinements l Learn appropriate restart parameters l More problem classes, consistency methods, retraction methods, planners, and Advisor languages l Learn appropriate consistency checking methods l Learn appropriate backtracking methods l Learn to bias initial weights l Metaheuristics to reformulate the architecture l Modeling strategies … and, coming soon, ACE on the Web

35
Acknowledgements Continued thanks for their ideas and efforts go to: Diarmuid Grimes Mark Hennessey Tiziana Ligorio Anton Morozov Smiljana Petrovic Bruce Samuels Students of the FORR study group The Cork Constraint Computation Centre and, for their support, to: The National Science Foundation Science Foundation Ireland

36
Is ACE Reinforcement Learning? l Similarities: l Unsupervised learning through trial and error l Delayed rewards l Learns a policy l Primary differences: l Reinforcement learning learns a policy represented as the estimated values of states it has experienced repeatedly … but ACE is unlikely to revisit a state; instead it learns how to act in any state l Q-learning learns state-action preferences … but ACE learns a policy that combines action preferences

37
How is ACE like STAGGER? l STAGGERACE l LearnsBoolean classifier Search control preference function for a sequence of decisions in a class of problems l RepresentsWeighted booleans Weighted linear function l Supervised Yes No l New elementsFailure-drivenSuccess-driven l Initial bias Yes Under construction l Real attributes Yes No [Schlimmer 1987]

38
l Both learn search control from unsupervised experience, reinforce decisions on a successful path, gradually introduce new factors, specify a threshold, and transfer to harder problems, but… l SAGE.2ACE l Learns onSame task Different problems in a class l RepresentsSymbolic rulesWeighted linear function l ReinforcesRepeating rulesCorrect comments l Failure responseReviseReduce weight l Proportional to errorNo Yes l Compares statesYesNo l Random benchmarksNoYes l SubgoalsNoYes l Learns during search Yes No How is ACE like SAGE.2? [Langley 1985]

Similar presentations

OK

Constraint Satisfaction Problems Chapter 6. Review Agent, Environment, State Agent as search problem Uninformed search strategies Informed (heuristic.

Constraint Satisfaction Problems Chapter 6. Review Agent, Environment, State Agent as search problem Uninformed search strategies Informed (heuristic.

© 2018 SlidePlayer.com Inc.

All rights reserved.

To make this website work, we log user data and share it with processors. To use this website, you must agree to our Privacy Policy, including cookie policy.

Ads by Google