1. Credibility: Evaluating what’s been learned This Lecture based on Ch 5 of Witten & Frank Plan for this week 3 classes before Midterm Paper and Survey discussion

2. Plan for this week Monday –Performance prediction of learning schemes –Cross-validation, Leave-one-out, bootstrap Wednesday –Loss function, cost-sensitive learning –Evaluating numeric prediction –Minimum description length (MDL) principle –Comparing data mining schemes

3. 3 Classes before Midterm Concept review exercises – in class Less homework More in class discussion Example: What is the design idea for basic association rule mining algorithm Apriori?

4. Discussions on paper and Survey areas one-page progress Report Due 3/24 Research paper selection (title, authors, source) Project proposal (problem definition and proposed solution design, dataset, algorithms, evaluation methods, or survey topic …)

5. Concept Questions Class 1 What are the design motivations and ideas for the following evaluation methods: –Cross-validation –Leave-one-out –0.632 bootstrap What is the final error estimate in each of the above method?

6. Concept Questions class 2 When do we care about accuracy of prediction probabilities? What role does loss function play in evaluation of learning scheme? Which loss function should we use if you are gambling on a particular event coming up? What is MDL principle? Give an application of MDL in data mining. Why lift chart is a valuable tool in marketing?