1. Standards or Semantics for Curriculum Search? Byron Marshall and René Reitsma Oregon State University Martha N. Cyr Worcester Polytechnic Institute JCDL 2007 TeachEngineering is funded by the National Science, Technology, Engineering, and Mathematics Education Digital Library (NSDL) program, under National Science Foundation grant nos. 0226322 and 0532709.

2. Standards or Semantics for Curriculum Search? 2 Agenda TeachEngineering The Goal: Help teachers find curricula that support assigned local educational standards Standards Alignment: –What is being done and why? –Two problems: granularity and sparseness Standards + Semantics: –Network-based approach –Preliminary results

3. Standards or Semantics for Curriculum Search? 3 TeachEngineering Collaborative project: –Colorado School of Mines, Duke, Oregon State University, University of Colorado at Boulder & Worcester Polytechnic Institute –American Society for Engineering Education –NSF National Science Digital Library Engineering curriculum (FIPSE and GK-12) Free, Web-based search http://www.teachengineering.com/about.php

4. Standards or Semantics for Curriculum Search? 4 The Task: Identifying Useful Curricula Assigned Educational Standards –Thousands of individual standards across multiple states and national groups –Increasing pressure to teach to standards Curriculum is often mapped to standards in one state at most: an opportunity for value-added search in a digital library One seemingly useful approach: identify equivalent standards across states

5. Standards or Semantics for Curriculum Search? 5 Transitive Results Direct Results Query: What lesson plans support my Massachusetts standard? Standards-Enabled Search: Crosswalking at TeachEngineering (TE) TE created a network of transitive mappings (state std.  national std.  state std.) by manually mapping 4 sets of state standards to several national models (3386 state  national alignments) Curriculum is mapped to standards for one state so users can search for standards-relevant items developed in different states Lesson Plan CO. State Std. Lesson Plan National Std. Lesson Plan MA. State Std.

6. Standards or Semantics for Curriculum Search? 6 Curriculum Alignment for Search TE did its manual mapping but always planned to rely on others to map the standards: –JES&Co and the University of Washington –CNLP (Center for Natural Language Processing) at Syracuse University –Standards-based search from collections such as DLESE and TeachersDomain.org –For-profit initiatives, e.g., nettrekker.com, academicbenchmarks.com, statestandards.com. –Ongoing collaborations between some of the above projects and organizations

7. Standards or Semantics for Curriculum Search? 7 Problem 1: Granularity -- an Example The TE system contains some materials developed in Colorado which are related to triangles and the Pythagorean theorem But, no documents are returned for a TE search based on the standard: –MA Math. Geom. 8.G.4: Demonstrate an understanding of the Pythagorean theorem. Apply the theorem to the solution of problems. Why? The Colorado standards do not specifically mention the Pythagorean theorem; there exists no CO-MA standards crosswalk

8. Standards or Semantics for Curriculum Search? 8 Problem 2: Sparseness in the TeachEngineering Std-to-Std Mappings States COMANCOKOverall Standards 1176914157251948 National Alignments 24370237266897 National Alignments/Std..21.54.57.37.46 (transitive) State Alignments 34235240266775 (transitive) State Alignments/Std..29.34.58.37.40 (projected) State Aligns./Std. (U.S.) 4.735.559.476.046.53 We found only.4 transitive alignments on average for each standard. Projecting this for the US (.4/3 * 49) each state standard would have fewer than 7 mapped standards anywhere in the country.

9. Standards or Semantics for Curriculum Search? 9 Looking Ahead: Solutions We are Exploring at TE Term-based search can help teachers find those Pythagorean lessons –tedious for the user? spotty results? Commercial services; e.g., Academic Benchmarks, will align standards to lessons –cost? proprietary/manual methodologies? static mappings to changing standards? millions of unprocessed documents? Learning-based systems (e.g., CNLP’s CAT & SAT tools) –Can comprehensive and reliable training data be developed and maintained? –State of the art provides very useful support tools for semi-automatic alignment

10. Standards or Semantics for Curriculum Search? 10 Our Agenda: Exploit Multiple Linkages to Establish Relevance Educational Standard Lesson Plan Term Lesson Plan Term More pathways may help find more items and improve ranking More Terms Lesson Plan Other Educational Standards AAAS Strand Maps Std. Correlations Other Associations; e.g., Curriculum Structure, Ontologies, or Resource Similarity Lesson Plan

11. Standards or Semantics for Curriculum Search? 11 Preliminary Results: Expanding Queries Using Standards Alignments Adding terms found in aligned standards improved our human-judged accuracy. Using terms from a standard resulted in 11 correct (as judged by a person) lesson plans in the top 15 returned items. When we used terms from the standard and from other aligned standards, accuracy improved to 14/15. Query: What lesson plans support my Massachusetts standard? NC State Std. MA. State Std. crust earth Lesson Plan #1 #2 #3 #4 earthquake

12. Standards or Semantics for Curriculum Search? 12 Preliminary Results: AAAS Plate Tectonics Strand Map AAAS 4.C.4 Grades 9-12 Including terms from all of the parts of this strand map added some interesting dimensions to our results set such as The Energy of Music Seeing and Feeling Sound Vibrations Riding Radio Waves These are not the first thing we think of, but then again….

13. Standards or Semantics for Curriculum Search? 13 Wrap Up Teachers need curriculum that supports assigned educational standards. How can we help them get to the documents they really want? Standards mappings are a start but granularity differences and sparseness are real problems. To address this important user task, hybrid solutions leveraging both standards and term-based semantics are promising.

14. Standards or Semantics for Curriculum Search? 14 Thanks! Questions? Standards or Semantics for Curriculum Search? Byron Marshall and René Reitsma Oregon State University Martha N. Cyr Worcester Polytechnic Institute

15. Standards or Semantics for Curriculum Search? 15 Preliminary Results: Noun Phrases Term-relevance is a well-explored phenomenon but which method of choosing index terms is best in this context? Based on preliminary work we observe that using noun phrases can boost the relevance score of good items.