1. The TM System for Repairing Non-Theorems Alison Pease – University of Edinburgh Simon Colton – Imperial College, London

2. Infeasible but Illustrative… Child interested in ATP –Says: “All prime numbers are odd” –ATP replies: “No. Go away.” Much more intelligent to say: –“You’re not quite right. In fact, all primes except two are odd”

3. Possibly More Feasible 1 st year maths student: –“All groups are Abelian” ATP: –“No” Model Generator/Constraint Solver: –Here’s a non-Abelian group, you idiot Clever reasoning system: –“No, but self inverse groups are Abelian and have you looked at cyclic groups?”

4. Inspiration from Imre Lakatos Philosophy of maths –Fallibilistic approach, theory is fluid Important book: –Proofs and refutations Two strands –Methods for dealing with counterexamples –Social aspect to theory formation process Running example –Euler’s theorem

5. Motivations for TM Project 1.Implement Lakatos’s philosophy of maths 2.Integrate Reasoning systems 3.Improve ATP systems To be more robust/flexible Enable more organic growing of theories 4.Show the HR system working in ATP Shown effective for ML and CSPs Theorem proving seen as A starting point for a discovery session

6. PhD Project of Alison Pease Aims to (and achieves) –The automation of Lakatos’s methods for handling counterexamples The social aspect of theory formation Perspectives: –Computational philosophy –Scientific discovery –Improvement of AI techniques Automated Theory Formation, Automated Reasoning

7. Spin off from Alison’s work The TM System A system for handling non-theorems –By modifying them into theorems Using methods inspired by Lakatos –Less interested here in the social aspect TM is a wrapper for 3 rd party software –Otter (ATP), MACE (Model generator) –HR (Machine learning – see later) Used so far only in algebraic domains

8. Some of Lakatos’s Methods Counterexample barring: –Alter conjecture to explicitly exclude each counterexample Primes except 2 are odd Piecemeal exclusion: –Exclude an entire class of examples Primes except powers of 2 are odd Strategic withdrawal –Specialise to a subset with no counters Mersenne primes are odd See paper for formal description of these

9. The TM System Input & Output Input –Conjectures of the form A  C A are axioms, C is conjecture statement –Given in Otter format Axioms first, last line is conjecture Output –Proof of the original if true, or: –Modified theorems Of the form A  M  C Which are proved and probably not obvious

10. Our Inspiring Example Input non-theorem all a b c ((a*b)*c = a*(b*c)). exists id (all a (a*id = id *a = a)). all a exists b (a * b = b * a = id). -(all a b (a * b = b * a)). Output modified theorem all a b c ((a*b)*c = a*(b*c)). exists id (all a (a*id = id *a = a)). all a exists b (a * b = b * a = id). all a (a * a = id). -(all a b (a * b = b * a)).

11. The TM System Overview Five stages: –Preliminary checks Using Otter –Forming supporting and falsifying models Using MACE –Forming a theory Using HR –Extracting modifications and proving them TM does this using Otter –Flagging possibly obvious modifications TM does this

12. Stage 1: Preliminary Checks Otter is used to attempt to prove –(i) A  C No modification required –(ii) A  ¬C No specialisation will help –(iii) A  (Triv  C) True only for trivial algebras –(iv) A  (¬Triv  C) True only for non-trivial algebras Last two are inspired by Lakatos’s counterexample barring methods

13. Stage 2: Model Generation Falsifying examples generated –MACE given Axioms + ¬C Supporting examples generated –MACE given Axioms + C 10 seconds allowed –For each size 1 to 8 –User can alter these settings

14. Stage 3: Theory Formation Forming specialisations –Done inductively (not math. induction) Predictive learning task –Positive and negative examples of a concept Learn a definition for positives –We want plenty of answers So we want a descriptive rather than predictive system TM uses the HR program to form a theory –Using concepts from axioms as background –And examples as objects of interest

15. The HR Program (since 1997) Descriptive induction program –Works mostly in mathematics domain –Also, bioinformatics, vision, music recently Forms a scientific theory –Given a small amount of knowledge –E.g., how to divide two numbers, ring axioms Theories contain –Example, concepts, conjectures, proofs Main features: –Production rules, measures of interestingness, –Empirical conjecture making, –Using reasoning programs (Otter, MACE, …)

16. Stage 4: Formation of Modifications HR’s theories contain many specialisations –E.g., self-inverse, idempotent, Abelian, Some specialisation are true of –A subset of the supporting examples But no falsifying examples –Such specialisation are added as an axiom (Axioms + Specialisation)  Conjecture –This is strategic withdrawal And also piecemeal exclusion (HR’s negate rule) Otter is used to prove each modified theorem –Time allowed varied by the user

17. Stage 5 Identifying Potentially Dull Results It’s quite easy to modify a theorem –And make it trivially true Case 1: –Specialisation is the trivial algebra E.g., all trivial algebras are Abelian So TM checks whether A  (M  Triv) –Flags these as probably uninteresting Case 2: –Concept is re-definition of conjecture E.g., all Abelian groups are Abelian So TM checks whether –(a) M  C (b) M  C (c) A  (M  C) –Flags these as probably uninteresting

18. Experiments Difficult to get hold of suitable non-theorems –In the form A  C TPTP library –Most non-theorems are satisfiable axioms –Others show that 2 sets of axioms not equivalent –Looked in GRP, RNG, FLD, COL Found only 9 suitable examples –Please add your non-theorems to the library!!! Also produced 89 artificial non-theorems –By taking TPTP theorems and changing: –Axioms, variables, quantifiers, bracketing

19. Experimental Setup(s) Otter –10 seconds on every run MACE –10 seconds on every run, size 1 to 8 Preliminary tests showed that –Altering Otter and MACE settings Had little effect HR settings altered –Theory formation steps (1000 & 3000) –Allowed to use equivalence conjs (& not)

20. Results 1000 steps Equiv off

21. Some Artificial Examples Derived from GRP001: –Self inverse groups are Abelian Removed inverse and associative axioms –HR re-invented Abelian and TM discarded Derived from GRP011-4 –Left cancellation law Identity and inverse axioms removed –Five cautioned modifications generated Including one of the form M  C –  x y (x * (x * y) = y) implies left cancellation –True without mention of associativity (interesting)

22. Real examples from TPTP TM successfully modified 7 of 9 –3 out of 5 in COL (new domain) Nice example –First non-theorem from GRP is GRP024-4 –comm(x,y) = x*y*x -1 y -1 –comm is associative iff all commutators are in the centre of the group –Mace found no counters But found four groups supporting this –TM found that this is true for Self inverse groups (  a (a * a = id))

23. My Favourite Example RNG031-6 In rings, the following property holds: –  w x ((((w*w)*x)*(w*w))=id) –Has some history: JAR paper about them Mace: 7 supporting, 6 falsifying HR: a single specialisation was a pos sub: –¬(  b, c (b*b=c  ¬(b+b=c))) Tidied up: –In Rings, If (  b (b*b = b+b)) then (  w x ((((w*w)*x)*(w*w))=id)) Nice symmetry to it Otter proved this

24. Conclusions & Further Work We have shown that ATP can be flexible –Required induction, deduction and calculation Integration is so obviously the right direction for automated reasoning –Demonstrated the effectiveness of TM On a set of problems from algebra Future work: –Possibly apply to verification tasks –“Crack open the conjectures” E.g., alter the LHS or RHS to fix it –Use Progol rather than HR for discrimination Will be quicker, but produce fewer modifications

25. Off now… To be on an ESFOR ‘wish list’ panel Where I’ll ask whether they can do this deductively rather than inductively!