1. WORKING TOGETHER TO IMPROVE SCIENCE EDUCATION PRESENTED BY GIBSON & ASSOCIATES A CALIFORNIA MATH AND SCIENCE PARTNERSHIP RESEARCH GRANT WISE II Evaluation Presentation 2008-09

2. Presentation Overview General information about the WISE II program Discussion of evaluation methods Challenges and how we addressed them Our preliminary data results.

3. WISE II Partners  San Francisco Unified School District  San Francisco State University  Cal Academy of Sciences  City College of San Francisco  UC Museum of Paleontology

4. WISE II Program Description Based on the WISE I program, WISE II is a three-year research grant for teachers in grades 3-5 to improve their science content knowledge, pedagogical content knowledge, and pedagogy. Teachers attend a summer workshop, 3 release days, and a Saturday session. Teachers receive in-classroom coaching and lesson planning support 1 to 4 times a month and meet collaboratively with their teams to plan curricula, discuss student achievement, and develop lessons.

5. Challenging Courses and Curricula for Teachers and Students CaMSP defines challenging courses and curricula as aligned with state content standards and frameworks, resulting in a greater number of students participating and succeeding in advanced courses. How do you evaluate challenging courses and curricula for elementary students and teachers?

6. Teacher Data Collection Tools Teacher Data Collection  Content Knowledge Assessment- Nationally Derived Tool  Additional Content Assessments – Locally Derived Tools  Pedagogical Questionnaires – Pre, Mid-Year, and Post  Professional Development Feedback Forms - Ongoing  Classroom Observations –twice a year  Student Notebooks Collected- twice a year  Use of a Control Group

7. Evaluation Methods Teacher Data Collection National Content Knowledge Assessments Local Content Assessments Pedagogical Questionnaires Formative Professional Development Feedback Pre-Post Test content knowledge test  Assessing Teacher Learning About Science Teaching ATLAST by Horizons Research. Assessing Teacher Learning About Science Teaching  Strength- piloted and tested and provides useful information about teacher practice and knowledge.  Challenge-not CA standards, not directly aligned to PD, and it’s difficult – so teachers do not enjoy taking it.

8. Evaluation Methods Teacher Data Collection National Content Knowledge Assessments Local Content Assessments Pedagogical Questionnaires Formative Professional Development Feedback Complement ATLAST with other content knowledge instruments:  Standards-based pre-post twice a year.  Faculty developed pre-post instruments given the day of the treatment. Teachers receive immediate feedback on what they have learned that day.

9. Evaluation Methods Teacher Data Collection National Content Knowledge Assessments Local Content Assessments Pedagogical Questionnaires Formative Professional Development Feedback Questionnaires Pre-Post & Formative Evaluations of PD  Pedagogical questionnaires examine frequency of practice & confidence in knowledge and instruction. They look at growth by individual teacher and area. We compare it to our control group.  After every PD session, we provide feedback forms that include questions on relevancy to standards and to instruction.

10. Control Group Not receive any similar treatment Willing to be in program for 3 years Willing to take additional student assessments Their own confidence in science teaching and instruction Stipend for participation Control teachers for WISE II  Not receive any WISE II “treatment”  Complete ATLAST pre-post tests.  Complete pedagogical questionnaires.  Two or more teachers at site participating in the program.  Students complete the pre-post tests.  Teacher commits to 3 years.  Likely to be in district and at grade level for three or more years- and their school willing to participate.

11. Control Group Easy: a. Have not participated in the program Hard: a. Have similar characteristics of those in the program. Challenge Competing programs and reforms Control group more experienced Higher baseline of performance. Resolution View data in terms of growth. Growth in student scores and growth in competencies, confidence, and practice.

12. Student Data Collection Tools Student Data Collection  Pre-Post Standards Based Content Knowledge Assessment  Additional Content Assessments –used locally by teachers  California Standards Test – review both ELA and Science  Classroom Observations – conducted twice a year  Student Notebooks Collected twice a year  Control Group to compare growth

13. Evaluation Methods Student Data Collection Pre-Post Tests CST Data Student Notebooks Classroom Observations STUDENT  Pre-post tests in science.  Look at growth in student outcomes on California Standards Test.  Use rubric to analyze student science notebooks content, structure, and understanding.  Observe engagement and learning process through classroom observations using a rubric.  Biggest challenge- lack of 3 rd, 4 th grade CST in science.

14. A SAMPLING OF PRELIMINARY DATA FOR WISE II RESULTS

15. CHANGES IN CST PERFORMANCE CHANGES IN PRE-POST TEST REVIEWED IN CONTEXT OF GRANT Student Outcomes

16. WISE teachers improved student science outcomes As compared to Control Group and the district classrooms, WISE II teachers experienced greater growth in science than their peers between 2008 and 2009. In particular, more WISE II teachers had fewer students in below basic bands on the CST and more students were proficient than the previous year.

17. Student outcomes in Science on the CST WISE II teachers showed an increase on the 5 th grade Science CST from the baseline 2008 year to 2009 year.

18. Pre-Post Test Results Review of student growth and average growth in Earth, Life, and Physical Science at each grade level and as compared to control group. Other student outcomes Treatment teachers had average double digit gains on pre-post science tests in every strand (earth, life, physical). Student results on Pre-Post tests analyzed by growth and by WISE program offerings and curricular practices.

19. Example of Pre-Post Analysis 3 rd grade areas of greatest change on the pre-post test of WISE II students

20. Standards Learned 80% of 3 rd grade WISE II areas with greatest growth – pre-to-post test were addressed in WISE. These topics were also seen in student observations and drawings in student notebooks. ES.4.b Students know the way in which the Moon's appearance changes during the four-week lunar cycle. ES.4.d Students know that Earth is one of several planets that orbit the Sun and that the Moon orbits Earth. PS.1.a Students know energy comes from the Sun to Earth in the form of light. PS.1.e Students know matter has three forms: solid, liquid, and gas.

21. CONTENT KNOWLEDGE CONFIDENCE PRACTICE Teacher Changes

22. Teacher outcome results WISE II teachers improved content knowledge based on pre- post assessments. Control Group teachers declined in content knowledge.

23. WISE II Teacher Content knowledge WISE II teachers made growth based on the local –standards-based assessment tool.

24. Treatment teachers increased quantity of instruction. 93% of treatment teachers reported teaching a minimum of 46 minutes per week, as opposed to 78% of the same teachers the previous year.

25. Treatment teachers increased use of effective science teaching strategies. Note-few changes for control group in these areas.

26. WISE II increased teacher confidence Based on baseline and post-treatment surveys, WISE II teachers were more confident in their use of the following instructional strategies: Teaching Comprehension Vocabulary Sketching and Drawing

27. Increased confidence and application WISE II teachers showed an average growth in frequency and confidence of application of effective instructional strategies in every area analyzed.

28. WISE II improved more than control group teachers. WISE II teachers improved their confidence and competence at a greater rate than the control group teachers in nearly every area.

29. WISE II – Other Data Collected and Analyzed  Observation Data  Student Science Notebooks  Feedback forms from Evaluations  Teacher Comments  Faculty Input-Changes

30. WISE II – Challenging Courses and Curricula For more information, Contact Nada Djordjevich Gibson & Associates nada@gibsonandassociates.com 510-986-0990 ext. 207