1. This work is supported by a National Science Foundation (NSF) collaboration between the Directorates for Education and Human Resources (EHR) and Geociences (GEO) under grant DUE - 1125331 InTeGrate Materials Design Rubric Professional Development Webinar The webinar begins at: 9 am PDT | 10 am MDT | 11 am CDT | 12 pm EDT

2. InTeGrate Materials Design Rubric Joshua Caulkins The University of Rhode Island InTeGrate Webinar Coordinator Assessment Team Member Becca Walker Mt. San Antonio College InTeGrate Module Author Climate of Change Stuart Birnbaum Univ. of Texas San Antonio InTeGrate Assessment Team Lead

3. Goals of this webinar Describe how the InTeGrate Materials Design Rubric is used to review materials.

4. Goals of this webinar Describe how the InTeGrate Materials Design Rubric is used to review materials. Identify the most challenging parts of the rubric and how to ensure your team can meet the criteria.

5. Goals of this webinar Describe how the InTeGrate Materials Design Rubric is used to review materials. Identify the most challenging parts of the rubric and how to ensure your team can meet the criteria. Consider materials your team has developed using the rubric as a guide to identify areas that may need additional revision.

6. Six main sections of the rubric: Guiding Principles (5 criteria, requires 15/15)

7. Six main sections of the rubric: Guiding Principles (5 criteria, requires 15/15) Learning Objectives and Goals (5 criteria, requires 13/15)

8. Six main sections of the rubric: Guiding Principles (5 criteria, requires 15/15) Learning Objectives and Goals (5 criteria, requires 13/15) Assessment and Measurement (5 criteria, requires 13/15)

9. Six main sections of the rubric: Guiding Principles (5 criteria, requires 15/15) Learning Objectives and Goals (5 criteria, requires 13/15) Assessment and Measurement (5 criteria, requires 13/15) Resources and Materials (6 criteria, requires 15/18)

10. Six main sections of the rubric: Guiding Principles (5 criteria, requires 15/15) Learning Objectives and Goals (5 criteria, requires 13/15) Assessment and Measurement (5 criteria, requires 13/15) Resources and Materials (6 criteria, requires 15/18) Instructional Strategies (5 criteria, requires 13/15)

11. Six main sections of the rubric: Guiding Principles (5 criteria, requires 15/15) Learning Objectives and Goals (5 criteria, requires 13/15) Assessment and Measurement (5 criteria, requires 13/15) Resources and Materials (6 criteria, requires 15/18) Instructional Strategies (5 criteria, requires 13/15) Alignment (2 criteria, requires 5/6)

12. Section 1: Guiding Principles 1.Course/module addresses one or more geoscience-related grand challenges facing society.

13. Section 1: Guiding Principles 1.Course/module addresses one or more geoscience-related grand challenges facing society. – Rarely an issue: most modules/courses deal with one (or more) grand challenges facing society.

14. Section 1: Guiding Principles 1.Course/module addresses one or more geoscience-related grand challenges facing society. 2.Course/module develops student ability to address interdisciplinary problems.

15. Section 1: Guiding Principles 1.Course/module addresses one or more geoscience-related grand challenges facing society. 2.Course/module develops student ability to address interdisciplinary problems. – Geoscience as it relates to a social, political, or economic system. – Ask yourself: Could a non-geoscientist teach your course/module in a non-geoscience class?

16. Section 1: Guiding Principles 1.Course/module addresses one or more geoscience-related grand challenges facing society. 2.Course/module develops student ability to address interdisciplinary problems. 3.Course/module improves student understanding of the nature and methods of geoscience and developing geoscientific habits of mind.

17. Section 1: Guiding Principles 1.Course/module addresses one or more geoscience-related grand challenges facing society. 2.Course/module develops student ability to address interdisciplinary problems. 3.Course/module improves student understanding of the nature and methods of geoscience and developing geoscientific habits of mind. – Examples: Temporal Reasoning, Spatial Thinking, Geographic Facility, Systems Thinking, Uncertainty – http:serc.carleton.edu/integrate/teaching_materials/habits.html

18. Section 1: Guiding Principles 1.Course/module addresses one or more geoscience-related grand challenges facing society. 2.Course/module develops student ability to address interdisciplinary problems. 3.Course/module improves student understanding of the nature and methods of geoscience and developing geoscientific habits of mind. 4.Course/module makes use of authentic and credible geoscience data to learn central concepts in the context of geoscience methods of inquiry.

19. Section 1: Guiding Principles 1.Course/module addresses one or more geoscience-related grand challenges facing society. 2.Course/module develops student ability to address interdisciplinary problems. 3.Course/module improves student understanding of the nature and methods of geoscience and developing geoscientific habits of mind. 4.Course/module makes use of authentic and credible geoscience data to learn central concepts in the context of geoscience methods of inquiry. – Rarely a problem – Just be sure to use data.

20. Section 1: Guiding Principles 5.Course/module incorporates systems thinking.

21. Section 1: Guiding Principles 5.Course/module incorporates systems thinking. – See Systems Thinking webinar page: http://serc.carleton.edu/integrate/workshops/webinars/2015/systems_th ink.html – Example from Climate of Change module: Students examine real data (sea surface temp, wind, precipitation, etc) and discuss how the parts of the system interact

22. Section 1: Guiding Principles 5.Course/module incorporates systems thinking. – See Systems Thinking webinar page: http://serc.carleton.edu/integrate/workshops/webinars/2015/systems_th ink.html – Example from Climate of Change module: Students examine real data (sea surface temp, wind, precipitation, etc) and discuss how the parts of the system interact – Evolution of our approach to Systems Thinking: Course/module should present Systems Thinking and terminology explicitly and then apply it to different case studies, scenarios, problems, etc. Should be a theme throughout the course/module.

23. Section 1: Guiding Principles 5.Course/module incorporates systems thinking. – See Systems Thinking webinar page: http://serc.carleton.edu/integrate/workshops/webinars/2015/systems_th ink.html – Example from Climate of Change module: Students examine real data (sea surface temp, wind, precipitation, etc) and discuss how the parts of the system interact – Evolution of our approach to Systems Thinking: Course/module should present Systems Thinking and terminology explicitly and then apply it to different case studies, scenarios, problems, etc. Should be a theme throughout the course/module. – Assess Systems Thinking: Make sure you are measuring students ability to apply Systems Thinking in your assessments (quizzes, homework, projects, exams, etc).

24. Section 2: Learning Objectives and Goals 6.Learning objectives describe measurable geoscience literacy goals.

25. Section 2: Learning Objectives and Goals 6.Learning objectives describe measurable geoscience literacy goals. – Use Bloom’s Taxonomy, include variety of cognition levels – See Learning Goals interactive page: http://serc.carleton.edu/integrate/info_team_members/currdev/effective _materials/learning_goals.html

26. Section 2: Learning Objectives and Goals 6.Learning objectives describe measurable geoscience literacy goals. 7.Instructions and/or rubrics provide guidance for how students meet learning goals.

27. Section 2: Learning Objectives and Goals 6.Learning objectives describe measurable geoscience literacy goals. 7.Instructions and/or rubrics provide guidance for how students meet learning goals. – Almost all assessment should have rubrics – See Rubrics webinar or Interactive pages: Webinar: http://serc.carleton.edu/integrate/workshops/webinars/rubrics /index.html Interactive Page: http://serc.carleton.edu/integrate/info_team_members/currde v/effective_materials/rubrics.html

28. Section 2: Learning Objectives and Goals 6.Learning objectives describe measurable geoscience literacy goals. 7.Instructions and/or rubrics provide guidance for how students meet learning goals. 8.Learning objectives and goals are appropriate for the intended use of the course/module.

29. Section 2: Learning Objectives and Goals 6.Learning objectives describe measurable geoscience literacy goals. 7.Instructions and/or rubrics provide guidance for how students meet learning goals. 8.Learning objectives and goals are appropriate for the intended use of the course/module. – Rarely an issue.

30. Section 2: Learning Objectives and Goals 6.Learning objectives describe measurable geoscience literacy goals. 7.Instructions and/or rubrics provide guidance for how students meet learning goals. 8.Learning objectives and goals are appropriate for the intended use of the course/module. 9.Learning objectives and goals are clearly stated for each module in language suitable for the level of the students..

31. Section 2: Learning Objectives and Goals 6.Learning objectives describe measurable geoscience literacy goals. 7.Instructions and/or rubrics provide guidance for how students meet learning goals. 8.Learning objectives and goals are appropriate for the intended use of the course/module. 9.Learning objectives and goals are clearly stated for each module in language suitable for the level of the students. – Rarely a problem.

32. Section 2: Learning Objectives and Goals 6.Learning objectives describe measurable geoscience literacy goals. 7.Instructions and/or rubrics provide guidance for how students meet learning goals. 8.Learning objectives and goals are appropriate for the intended use of the course/module. 9.Learning objectives and goals are clearly stated for each module in language suitable for the level of the students. 10.Learning objectives and goals address the process and nature of science and development of scientific habits of mind.

33. Section 2: Learning Objectives and Goals 6.Learning objectives describe measurable geoscience literacy goals. 7.Instructions and/or rubrics provide guidance for how students meet learning goals. 8.Learning objectives and goals are appropriate for the intended use of the course/module. 9.Learning objectives and goals are clearly stated for each module in language suitable for the level of the students. 10.Learning objectives and goals address the process and nature of science and development of scientific habits of mind. – Rarely a problem.

34. Section 3: Assessment and Measurement 11.Assessments measure the learning goals. 12.Assessments are criterion referenced. 13.Assessments are consistent with course activities and resources expected. 14.Assessments are sequenced, varied and appropriate to content. 15.Assessments address goals at successively higher cognitive levels.

35. Section 3: Assessment and Measurement 11.Assessments measure the learning goals. 12.Assessments are criterion referenced. 13.Assessments are consistent with course activities and resources expected. 14.Assessments are sequenced, varied and appropriate to content. 15.Assessments address goals at successively higher cognitive levels. These are rarely problematic. Be sure to base assessments on learning goals, develop a rubric, use Bloom’s Taxonomy. Be sure to assess student’s abilities regarding System Thinking.

36. Section 4: Resources and Materials 16. Instructional materials contribute to the stated learning objectives. 17.Students will recognize the link between the learning objectives, goals and the learning materials. 18.Instructional materials should be sufficiently diverse and at the depth necessary for students to achieve learning objectives and goals. 19.Materials are appropriately cited. 20.Instructional materials are current. 21.Instructional materials and the technology to support these materials are clearly stated.

37. Section 4: Resources and Materials 16. Instructional materials contribute to the stated learning objectives. 17.Students will recognize the link between the learning objectives, goals and the learning materials. 18.Instructional materials should be sufficiently diverse and at the depth necessary for students to achieve learning objectives and goals. 19.Materials are appropriately cited. 20.Instructional materials are current. 21.Instructional materials and the technology to support these materials are clearly stated. Again, mostly straightforward. These are rarely problematic.

38. Section 5: Instructional Strategies 22.Learning strategies and activities support stated learning objectives and goals. – Rarely an issue.

39. Section 5: Instructional Strategies 22.Learning strategies and activities support stated learning objectives and goals. 23.Learning strategies and activities promote student engagement with the materials. – Also, rarely an issue.

40. Section 5: Instructional Strategies 22.Learning strategies and activities support stated learning objectives and goals. 23.Learning strategies and activities promote student engagement with the materials. 24.Learning activities develop student metacognition.

41. Section 5: Instructional Strategies 22.Learning strategies and activities support stated learning objectives and goals. 23.Learning strategies and activities promote student engagement with the materials. 24.Learning activities develop student metacognition. – More challenging than other criteria. – “Thinking about thinking,” encourage student to reflect – Use your assessment team consultant, Metacognition resources

42. Section 5: Instructional Strategies 22.Learning strategies and activities support stated learning objectives and goals. 23.Learning strategies and activities promote student engagement with the materials. 24.Learning activities develop student metacognition. 25.Learning strategies and activities provide opportunities for students to practice communicating geoscience.

43. Section 5: Instructional Strategies 22.Learning strategies and activities support stated learning objectives and goals. 23.Learning strategies and activities promote student engagement with the materials. 24.Learning activities develop student metacognition. 25.Learning strategies and activities provide opportunities for students to practice communicating geoscience. – Recall ENSO jigsaw activity described previously. – Consider Geoscientific habits of mind (#3 in Guiding Principles). – Students should use the terms listed there.

44. Section 5: Instructional Strategies 22.Learning strategies and activities support stated learning objectives and goals. 23.Learning strategies and activities promote student engagement with the materials. 24.Learning activities develop student metacognition. 25.Learning strategies and activities provide opportunities for students to practice communicating geoscience. 26.Learning strategies and activities scaffold learning.

45. Section 5: Instructional Strategies 22.Learning strategies and activities support stated learning objectives and goals. 23.Learning strategies and activities promote student engagement with the materials. 24.Learning activities develop student metacognition. 25.Learning strategies and activities provide opportunities for students to practice communicating geoscience. 26.Learning strategies and activities scaffold learning. – Start with basic knowledge, build on that, assessing students (both formative and summative), should be straightforward.

46. Section 6: Alignment 27.Teaching materials, assessments, resources and learning activities align with one another. 28.All aspects of the module/course are aligned.

47. Section 6: Alignment 27.Teaching materials, assessments, resources and learning activities align with one another. 28.All aspects of the module/course are aligned. These rarely pose a problem during review.

48. Questions?