1. Dimensional Standard Alignment in K-12 Digital Libraries Assessment of Self- found vs. Recommended Curriculum Byron Marshall René Reitsma Oregon State University Malinda Zarske University of Colorado

2. http://www.teachengineering.org/ Oregon State University Zzz... The Goal: Lessons For Standards AAAS NCMA 60,000+ constantly changing US Science and Math Standards! Just Call me “Doc” TX Edu Stds Oregon CO Providers of educational resources use experts to “align” educational resources (docs) to educational standards. These expert mappings are either used directly or used to train retrieval systems. CO MA NC

3. http://www.teachengineering.org/ Oregon State University The Problem: How Do We Know a Resource Aligns with a Standard? Alignment Considerations: Concepts! Grade Level Affect (Kids like it!) Extras (exercises, worksheets, examples, links) Cost CO Std. No Yes Do they agree? No. But are they thinking the same thing?

4. http://www.teachengineering.org/ Oregon State University Collecting Nuanced Training Data Binary (Yes/No) and singular (one dimensional) judgments produce low IRR. Instead, we asked: See Saracevic (2007) in JASIST for more on these match categories strongly disagree disagree N/A somewhat disagree somewhat agree agree strongly agree

5. http://www.teachengineering.org/ Oregon State University Differentiating Judgments by Task Self-Found Please Rate the Relevance of a Self-Found Doc/Std Pair Recommended What do you think of a given Doc/Std Pair?

6. http://www.teachengineering.org/ Oregon State University Question: Do People Evaluate Self- Found Items Differently than Recommended ones? Experiment: ◦43 Teachers, 934 standard/doc pairs ◦71 educational standards Two tasks: ◦Find docs for a std and rate them (Self-Found) ◦Rate doc/std pairs suggested by others (Recommended)

7. http://www.teachengineering.org/ Oregon State University Yes, Self-found Pairs are Scored Differently than Recommend Ones People judged differently when asked to find docs for stds and rate the alignment of as opposed to when asked to rate provided doc/std pairs ◦Two solid but different linear regression models emerged ◦User behavior as observed in the Web logs was different

8. http://www.teachengineering.org/ Oregon State University Different Models of Relevance Dependent variable: “Overall Alignment” Independent variables are our dimensions Situational Match Object Match Content Match Both Models Have ‘Good’ Explanatory Power All coefficients statistically significant at p<.01 Activities and Attachments did not Influence the Decision for Recommended Pairs Content Still Dominates

9. http://www.teachengineering.org/ Oregon State University User Behavior When Searching and Evaluating vs. Evaluating only DL Activity Log indications: ◦Searchers opened 1.97.pdfs,.docs, & worksheets when rating self-found document/std pairs ◦Raters viewed 1.37 when deciding on a provided pair Scores on one dimension “bled-over” more to other dimensions for recommended pairs (that is, we saw lower correlation between dimensions on self-found items)

10. http://www.teachengineering.org/ Oregon State University So What? Self-Found Vs. Recommended Resources Adjust user interfaces? Incorporate training data differently

11. http://www.teachengineering.org/ Oregon State University Thanks! Questions?