1. ICML-Tutorial, Banff, Canada, 2004 Kristian Kersting University of Freiburg Germany „Application of Probabilistic ILP II“, FP6-508861 www.aprill.orgwww.aprill.org James Cussens University of York UK Probabilistic Logic Learning Probability LogicLearning al and Relational

2. ICML-Tutorial, Banff, Canada, 2004 Special thanks to the APrIL II consortium „Application of Probabilistic ILP“ 3 years EU project 5 institutes www.aprill.org Heikki Mannila Stephen Muggleton, Mike Sternberg Subcontractor: James Cussens Luc De Raedt Subcontractor: Manfred Jaeger Paolo Frasconi François Fages

3. ICML-Tutorial, Banff, Canada, 2004... special thanks...... for discussions, materials, and collaborations to Alexandru Cocura, Uwe Dick, Pedro Domingos, Peter Flach, Thomas Gaertner, Lise Getoor, Martin Guetlein, Bernd Gutmann, Tapani Raiko, Reimund Renner, Richard Schmidt, Ingo Thon,...

4. ICML-Tutorial, Banff, Canada, 2004 Tutorial´s Aims Introductory survey Identification of important probabilistic, relational/logical and learning concepts

5. ICML-Tutorial, Banff, Canada, 2004 The integration of probabilistic reasoning with Objectives One of the key open questions of AI concerns Probabilistic Logic Learning: machine learning. first order / relational logic representations and Probabilitiy Learning Logic

6. ICML-Tutorial, Banff, Canada, 2004 Diagnosis Prediction Classification Decision-making Description Medicine Computational Biology Robotics Web Mining PLMs Economic Text Classification Computer troubleshooting Why do we need PLL? Let‘s look at an example

7. ICML-Tutorial, Banff, Canada, 2004 Web Mining / Linked Bibliographic Data / Recommendation Systems / … book author publisher Real World [illustration inspired by Lise Getoor] publisher book author

8. ICML-Tutorial, Banff, Canada, 2004 Web Mining / Linked Bibliographic Data / Recommendation Systems / … B1 B2 B3 B4 P1 books authors publishers series author-of publisher-of Real World Fantasy Science Fiction P2 A2 A1

9. ICML-Tutorial, Banff, Canada, 2004 Why do we need PLL? Real World Applications Let‘s look at some more examples Structured Domains Not flat but structured representations: Multi-relational, heterogeneous and semi-structured Uncertainty Dealing with noisy data, missing data and hidden variables Machine Learning Knowledge Acquisition Bottleneck, Data cheap

10. ICML-Tutorial, Banff, Canada, 2004 Blood Type / Genetics/ Breeding 2 Alleles: A and a Probability of Genotypes AA, Aa, aa ? CEPH Genotype DB,http://www.cephb.fr/ AA Aa AA Aa aa Aa aa Aa AA Aa aa AA aa Aa Prior for founders FatherMother Offspring

11. ICML-Tutorial, Banff, Canada, 2004 Bongard´s Problems Noise? Some objects are opaque? (e.g. in relation is not always observed)

12. ICML-Tutorial, Banff, Canada, 2004 Bongard´s Problems Clustering? Noise? Some objects are opaque? (e.g. in relation is not always observed)

13. ICML-Tutorial, Banff, Canada, 2004 Others Protein Secondary Structure Metabolic PathwaysPhylogenetic Trees Scene interpretation Social Networks Data Cleaning ?

14. ICML-Tutorial, Banff, Canada, 2004 Why do we need PLL ? Real World Applications Uncertainty Machine Learning Structured Domains Statistical Learning (SL) Probabilistic Logics Inductive Logic Programming (ILP) Multi-Relational Data Mining (MRDM) - attribute-value representations: some learning problems cannot (elegantly) be described using attribute value representations + soft reasoning, learning - no learning: to expensive to handcraft models + soft reasoning, expressivity - crisp reasoning: some learning problems cannot (elegantly) be described without explicit handling of uncertainty + expressivity, learning

15. ICML-Tutorial, Banff, Canada, 2004 Why do we need PLL? Rich Probabilistic Models Comprehensibility Generalization (similar situations/individuals) Knowledge sharing Parameter Reduction / Compression Learning –Reuse of experience (training one RV might improve prediction at other RV) –More robust –Speed-up

16. ICML-Tutorial, Banff, Canada, 2004 When to apply PLL ? When it is impossible to elegantly represent your problem in attribute value form –variable number of ‘objects’ in examples –relations among objects are important Background knowledge can be defined intensionally : –define ‘benzene rings’ as view predicates

17. ICML-Tutorial, Banff, Canada, 2004 Why stressing Learning? Knowledge acquisition bottleneck / data is cheap General purpose systems Combining domain expert knowledge with data Logical structure provides insight into domain Handling missing data bt(luc)=? Database Model Learning Algorithm

18. ICML-Tutorial, Banff, Canada, 2004 LAPD: Bruynooghe Vennekens,Verbaeten Markov Logic: Domingos, Richardson CLP(BN): Cussens,Page, Qazi,Santos Costa (Incomplete) Historical Sketch 2003 Present PRMs: Friedman,Getoor,Koller, Pfeffer,Segal,Taskar ´03 ´96 SLPs: Cussens,Muggleton ´90´95 First KBMC approaches: Bresse, Bacchus, Charniak, Glesner, Goldman, Koller, Poole, Wellmann ´00 BLPs: Kersting, De Raedt RMMs: Anderson,Domingos, Weld LOHMMs: De Raedt, Kersting, Raiko [names in alphabetical order] many more... Future Prob. CLP: Eisele, Riezler ´02 PRISM: Kameya, Sato ´94 PLP: Haddawy, Ngo ´97 ´93 Prob. Horn Abduction: Poole ´99 1BC(2): Flach, Lachiche Logical Bayesian Networks: Blockeel,Bruynooghe, Fierens,Ramon,

19. ICML-Tutorial, Banff, Canada, 2004 Overview 1.Introduction to PLL 2.Foundations of PLL –Logic Programming, Bayesian Networks, Hidden Markov Models, Stochastic Grammars 3.Frameworks of PLL –Independent Choice Logic,Stochastic Logic Programs, PRISM, –Bayesian Logic Programs, Probabilistic Logic Programs,Probabilistic Relational Models –Logical Hidden Markov Models 4.Applications