1. Special Topics in Educational Data Mining HUDK5199 Spring term, 2013 February 18, 2013

2. Today’s Class Classification And then some discussion of features in Excel between end of class and 5pm – We will start today, and continue in future classes as needed

3. Types of EDM method (Baker & Siemens, under review) Prediction – Classification – Regression – Latent Knowledge Estimation Structure Discovery – Clustering – Factor Analysis – Domain Structure Discovery – Network Analysis Relationship mining – Association rule mining – Correlation mining – Sequential pattern mining – Causal data mining Distillation of data for human judgment Discovery with models 3

4. We have already studied Prediction – Classification – Regression – Latent Knowledge Estimation Structure Discovery – Clustering – Factor Analysis – Domain Structure Discovery – Network Analysis Relationship mining – Association rule mining – Correlation mining – Sequential pattern mining – Causal data mining Distillation of data for human judgment Discovery with models 4

5. Today’s Class Prediction – Classification – Regression – Latent Knowledge Estimation Structure Discovery – Clustering – Factor Analysis – Domain Structure Discovery – Network Analysis Relationship mining – Association rule mining – Correlation mining – Sequential pattern mining – Causal data mining Distillation of data for human judgment Discovery with models 5

6. Prediction Pretty much what it says A student is using a tutor right now. Is he gaming the system or not? A student has used the tutor for the last half hour. How likely is it that she knows the skill in the next step? A student has completed three years of high school. What will be her score on the college entrance exam?

7. Classification There is something you want to predict (“the label”) The thing you want to predict is categorical – The answer is one of a set of categories, not a number – CORRECT/WRONG (sometimes expressed as 0,1) This is what we used in Latent Knowledge Estimation – HELP REQUEST/WORKED EXAMPLE REQUEST/ATTEMPT TO SOLVE – WILL DROP OUT/WON’T DROP OUT – WILL SELECT PROBLEM A,B,C,D,E,F, or G

8. Where do those labels come from? Field observations Text replays Post-test data Tutor performance Survey data School records Where else?

9. Classification Associated with each label are a set of “features”, which maybe you can use to predict the label Skillpknowtimetotalactionsright ENTERINGGIVEN0.70491WRONG ENTERINGGIVEN0.502102RIGHT USEDIFFNUM0.04961WRONG ENTERINGGIVEN0.96773RIGHT REMOVECOEFF0.792161WRONG REMOVECOEFF0.792132RIGHT USEDIFFNUM0.07352RIGHT ….

10. Classification The basic idea of a classifier is to determine which features, in which combination, can predict the label Skillpknowtimetotalactionsright ENTERINGGIVEN0.70491WRONG ENTERINGGIVEN0.502102RIGHT USEDIFFNUM0.04961WRONG ENTERINGGIVEN0.96773RIGHT REMOVECOEFF0.792161WRONG REMOVECOEFF0.792132RIGHT USEDIFFNUM0.07352RIGHT ….

11. Classification Of course, usually there are more than 4 features And more than 7 actions/data points These days, 800,000 student actions, and 26 features, would be a medium-sized data set

12. Classifiers There are literally hundreds of classification algorithms that have been proposed/published/tried out A good data mining package will have many implementations – RapidMiner – SAS Enterprise Miner – Weka – KEEL

13. Domain-Specificity Specific algorithms work better for specific domains and problems We often have hunches for why that is But it’s more in the realm of “lore” than really “engineering”

14. Some algorithms you probably don’t want to use Support Vector Machines – Conducts dimensionality reduction on data space and then fits hyperplane which splits classes – Creates very sophisticated models – Great for text mining – Great for sensor data – Usually pretty lousy for educational log data

15. Some algorithms you probably don’t want to use Genetic Algorithms – Uses mutation, combination, and natural selection to search space of possible models – Obtains a different answer every time (usually) – Seems really awesome – Usually doesn’t produce the best answer

16. Some algorithms you probably don’t want to use Neural Networks – Composes extremely complex relationships through combining “perceptrons” – Usually over-fits for educational log data

17. Note Support Vector Machines and Neural Networks are great for some problems I just haven’t seen them be the best solution for educational log data

18. Some algorithms you might find useful Step Regression Logistic Regression J48/C4.5 Decision Trees JRip Decision Rules K* Instance-Based Classifier There are many others!

19. Logistic Regression

20. Already discussed in class Fits logistic function to data to find out the frequency/odds of a specific value of the dependent variable Given a specific set of values of predictor variables

21. Logistic Regression m = a0 + a1v1 + a2v2 + a3v3 + a4v4…

22. Logistic Regression

23. Parameters fit Through Expectation Maximization

24. Relatively conservative Thanks to simple functional form, is a relatively conservative algorithm – Less tendency to over-fit

25. Good for Cases where changes in value of predictor variables have predictable effects on probability of predictor variable class

26. Good when multi-level interactions are not particularly common Can be given interaction effects through automated feature distillation – We’ll look at this later But is not particularly optimal for this

27. Note RapidMiner and Weka do not actually choose features for you You have to select features by hand Or in java code that calls RapidMiner or Weka Easy to implement step-wise regression by hand, painful to implement other feature selection algorithms

28. Step Regression

29. Fits a linear regression function – (discussed in detail in a later class) – with an arbitrary cut-off Selects parameters Assigns a weight to each parameter Computes a numerical value Then all values below 0.5 are treated as 0, and all values >= 0.5 are treated as 1

30. Example Y= 0.5a + 0.7b – 0.2c + 0.4d + 0.3 Cut-off 0.5 abcd 1111 0000 13

31. Parameters fit Through Iterative Gradient Descent This is a simple enough model that this approach actually works…

32. Most conservative This is the most conservative classifier except for the fabled “0R” – More on that in a minute

33. Good for Cases where relationships between predictor and predicted variables are relatively linear

34. Good when multi-level interactions are not particularly common Can be given interaction effects through automated feature distillation – We’ll look at this later But is not particularly optimal for this

35. Feature Selection Greedy – simplest model M5’ – in between None – most complex model

36. Greedy Also called Forward Selection – Even simpler than Stepwise Regression 1.Start with empty model 2.Which remaining feature best predicts the data when added to current model 3.If improvement to model is over threshold (in terms of SSR or statistical significance) 4.Then Add feature to model, and go to step 2 5.Else Quit

37. M5’ Will be discussed in detail in regression lecture

38. 0R

39. Always say 0

40. Decision Trees

41. Decision Tree PKNOW TIMETOTALACTIONS RIGHT WRONG <0.5>=0.5 <6s.>=6s.<4>=4 Skillpknowtimetotalactionsright COMPUTESLOPE0.54491?

42. Decision Tree Algorithms There are several I usually use J48, which is an open-source re- implementation of C4.5 (Quinlan, 1993)

43. J48/C4.5 Can handle both numerical and categorical predictor variables – Tries to find optimal split in numerical variable Repeatedly looks for variable which best splits the data in terms of predictive power for each variable (using information gain) Later prunes out branches that turn out to have low predictive power Note that different branches can have different features!

44. Can be adjusted… To split based on more or less evidence To prune based on more or less predictive power

45. Relatively conservative Thanks to pruning step, is a relatively conservative algorithm – Less tendency to over-fit

46. Good when data has natural splits

47. Good when multi-level interactions are common

48. Good when same construct can be arrived at in multiple ways A student is likely to drop out of college when he – Starts assignments early but lacks prerequisites OR when he – Starts assignments the day they’re due

49. Decision Rules

50. Many Algorithms Differences are in terms of what metric is used and how rules are generated Most popular subcategory (including JRip and PART) repeatedly creates decision trees and distills best rules

51. Generating Rules from Decision Tree 1.Create Decision Tree 2.If there is at least one path that is worth keeping, go to 3 else go to 6 3.Take the “Best” single path from root to leaf and make that path a rule 4.Remove all data points classified by that rule from data set 5.Go to step 1 6.Take all remaining data points 7.Find the most common value for those data points 8.Make an “otherwise” rule using that

52. Relatively conservative Leads to simpler models than most decision trees – Less tendency to over-fit

53. Very interpretable model Unlike most other approaches

54. Example (Baker & Clarke-Midura, under review) 1. IF the student spent at least 66 seconds reading the parasite information page, THEN the student will obtain the correct final conclusion (confidence = 81.5%) 2. IF the student spent at least 12 seconds reading the parasite information page AND the student read the parasite information page at least twice AND the student spent no more than 51 seconds reading the pesticides information page, THEN the student will obtain the correct final conclusion (confidence = 75.0%) 3. IF the student spent at least 44 seconds reading the parasite information page AND the student spent under 56 seconds reading the pollution information page, THEN the student will obtain the correct final conclusion (confidence = 68.8%) 4. OTHERWISE the student will not obtain the correct final conclusion (confidence = 89.0%)

55. Good when multi-level interactions are common

56. Good when same construct can be arrived at in multiple ways A student is likely to drop out of college when he – Starts assignments early but lacks prerequisites OR when he – Starts assignments the day they’re due

57. K*

58. Instance-Based Classifier Takes a data point to predict Finds the K closest points to that data point, by Euclidean distance – K often equals 3 Gives data point whichever class is most common among those 3 points

59. Good when data is very divergent Lots of different processes can lead to the same result Impossible to find general rules But data points that are similar tend to be from the same class

60. Big Drawback To use the model, you need to have the whole data set

61. Big Advantage Sometimes works when nothing else works Has been useful for my group in affect detection

62. Comments? Questions?

63. Generating Confidences Each of these approaches gives not just a final answer, but a confidence (or pseudo- confidence) Many applications of confidences – we’ll discuss in detail in next lecture

64. Generating Confidences Step Regression – raw value of regression Logistic Regression – p(m) Jrip/J48 – ratio of correct classifications to incorrect classifications in each leaf – I will show you an example of this in a minute

65. Hands-On Activity Running algorithm in RapidMiner 4.6 Using some made-up data, just to see how things work

66. Open RapidMiner And open classifier.xml Let’s go through this script

67. Use RapidMiner Run JRip without cross-validation Run JRip with action-level cross-validation Run Jrip with batch-level cross-validation – Set to implement student-level cross-validation – We’ll discuss this more in the next lecture

68. Confidences How to interpret model confidence from RapidMiner output

69. Let’s look at the data Are there any features we shouldn’t be using?

70. Remove those features Run the model again

71. Run other algorithms J48

72. Run other algorithms J48 K*

73. Run other algorithms J48 K* Step Regression

74. Run other algorithms J48 K* Step Regression – Why didn’t Step Regression work? – How can we fix it?

75. Comments? Questions?

76. Comments or Questions About Assignment 3?

77. Next Class Wednesday, February 20 Behavior Detection Assignment Due: 3. Behavior Detection

78. Excel Plan is to go as far as we can by 5pm We will continue after next class session Vote on which topics you most want to hear about

79. Topics Using average, count, sum, stdev (asgn. 4 data set) Relative and absolute referencing (made up data) Copy and paste values only (made up data) Using sort, filter (asgn. 4 data set) Making pivot table (asgn. 4 data set) Using vlookup (Jan. 28 class data set) Using countif (asgn. 4 data set) Making scatterplot (Jan. 28 class data set) Making histogram (asgn. 4 data set) Equation Solver (Jan. 28 class data set) Z-test (made up data) 2-sample t-test (made up data) Other topics?

80. The End