1. Collective Intelligence Week 1
Old Dominion University
Department of Computer Science
CS 795/895 Spring 2009
Michael L. Nelson
1/14/09

2. Goals We will: learn to program in Python
survey mathematical techniques for mining the web and be able to use them in actual applications
cover each chapter in the text book
note that each chapter is a survey; we could spend a semester or more on every topic…

3. Administrivia This is a programming class!
You should be able to teach yourself a new programming language from the book examples
You will work in teams of one or two people
mixes (g/u, g/g, u/u) ok
assignments are the same regardless of group size

4. Administrivia 2 Pick teams wisely
teams will exist by mutual consent only
at any time, teams can split up, but no new teams will be formed after the first assignment is due
no team member swaps
ex-team members will have access to their shared code base

5. Administrivia 3 Important URLs Class homepage:
Class homepage:
readings are posted
demo days are posted

6. Grading 10 problems, 10 points each Days of in class demo are posted
1 problem each from chapters 2-11
points recvd for each problem at the discretion of the instructor
each group must to the class list which problems they are planning to cover
instructor has the right to prohibit exercises deemed “too easy” and to approve exercises not included in book
since len(groups) > len(problems), overlap is ok (but variety is encouraged)
Days of in class demo are posted
demo day 1: chapters 2,3,4
demo day 2: chapters 5,6,7
demo day 3: chapters 8,9,10,11

7. Development Environment
Assignments must be demoed on the machine mln-web.cs.odu.edu
same uid/passwd as departmental unix machines; I will have accounts set up shortly
Assignments will be written in Python

8. Extra Credit 10 points available for each person (not group) for:
asking a question on the list
answering a question on the list
sharing tips
Only “significant” messages will be counted (instructor’s discretion)
e.g., “how do I login?” will not count

9. Why do we care about mining the Web?
People in aggregate can be very smart…
sometimes referred to as the “wisdom of crowds”
…and sometimes not.
pagerank image from:

10. Chapter 2: Recommendations

11. Comparing Reviews of 2 Movies

12. Comparing 2 Reviewers

13. 2 Highly Correlated Reviewers

14. Correlation
A measure of a linear relationship between two independent variables
from:

15. Correlation
In the book, we use “Pearson’s Product Moment Correlation Coefficient”
Many, many, many other coefficients possible
e.g, for rank data, we often use:
Spearman’s Rho
Kendall’s Tau

16. Look at data first!!!
Pearson’s only works for linear (i.e., normal distributions)
non parametric correlation methods needed for non-normal distributions
Example: Anscombe’s Quartet

17. Recommendations Based on Past Rankings
Note not all reviewers have seen all movies!

18. “If you like X, you probably won’t like Y.”

19. Item Based Recommendations (i.e., movies instead of critics)

20. To Do for Next Time… Subscribe to the class email list
Submit group info to class list
I’ll to the list when accounts on mln-web.cs.odu.edu are set up