1. Big Data, Education, and Society
February 28, 2018

2. Assignment
Any questions about the expectations for commenting on your classmates’ ideas?
Please do your best, your classmates are counting on your formative support!

3. Generalizability
Does your model remain predictive when used in a new data set?
Knowing the context the model will be used in drives what kinds of generalization you should study

4. Ecological Validity
Do your findings apply to real-life situations outside of research settings?
For example, if you build a detector of student behavior in lab settings, will it work in real classrooms?

5. Construct Validity
Does your model actually measure what it was intended to measure?

6. Construct Validity
Does your model actually measure what it was intended to measure?
(Does it map to a theory about the construct?)
(Do your model features plausibly measure what you are trying to detect?)

7. Predictive Validity
Does your model predict not just the present, but the future as well?

8. Substantive Validity Do your results matter?
Are you modeling a construct that matters?
If you model X, what kind of scientific findings or impacts on practice will this model drive?
Can be demonstrated by predicting future things that matter

9. Substantive Validity
For example, we know that boredom correlates strongly with
Disengagement
Learning Outcomes
Standardized Exam Scores
Attending College Years Later
By comparsion, whether someone prefers visual or verbal learning materials doesn’t even seem to predict very reliably whether they learn better from visual or verbal learning materials (See lit review in Pashler et al., 2008)

10. Consequential Validity
Does the use of the measure produce desired/desirable consequences?

11. Content Validity
From testing; does the test cover the full domain it is meant to cover?
For behavior modeling, an analogy would be, does the model cover the full range of behavior it’s intended to?
A model of gaming the system that only captured systematic guessing but not hint abuse (cf. Baker et al, 2004; my first model of this)
Would have lower content validity than a model which captured both (cf. Baker et al., 2008)

12. Conclusion Validity
Are your conclusions justified based on the evidence?

13. Other types of validity you want to discuss?

14. Relative Importance? Which of these do you want to optimize?
Which of these do you want to satisfice?
Can any be safely ignored completely? (at least in some cases)

15. Exercise
In groups of 3
Write the advertisement or elevator pitch for the least valid learning analytics based system ever

16. Any group want to share?

17. Hand’s perspective A statistician writing about data mining
"Statistics as a discipline has a poor record for timely recognition of important ideas... statisticians have later made very significant advances in all of these fields, but the fact that the perceived natural home of these areas lies not in statistics but in other areas is demonstrated by the key journals for these areas -- they are not statistical journals. Data mining seems to be following this pattern."

18. Hand’s perspective
Hand claims that selection bias is a particularly big problem for data used in data mining. Is this true? And if so, what are the consequences and how could they be addressed or mitigated?

19. Hand’s perspective
Hand et al claim that any pattern which cannot be explained should be treated as suspect. What are the benefits and drawbacks to this perspective?

20. Hand’s perspective
Hand et al claims that data mining can find common patterns, but their value and meaningfulness can only be determined by a domain expert. Do you agree?

21. Other thoughts and concerns
About validity

22. Questions? Comments?

23. Upcoming office hours Meetings by appointment next week (spring break)
March 14 10am-11am