1. Lessons Learned from Evaluating Science Education Projects Dave Weaver RMC Research Corporation 111 SW Columbia, Suite 1200 Portland, Oregon 97201 dweaver@rmccorp.com (800) 788–1887

2. 2 Presentation Contents Lessons Learned from LASER Sentinel Site Visits 3 Common Pitfalls to Improvement –How LASER and Washington Science Leadership is Responding Implications for Your Strategic Planning

3. 3 Analysis of Science WASL Data Evaluation Question –To what extent did teacher professional development on inquiry-based science instruction contribute to improved student achievement on the Grade 5 and 8 Washington Assessment of Student Learning of science (science WASL)?

4. 4 Finding 1 The number of professional development hours in which a student’s science teacher participated was a small but significant predictor of student performance on the science WASL above and beyond what could be explained by socioeconomics (FRL) and the student’s skill level (previous math WASL). This finding is consistent with earlier studies.

5. 5 PD Has Modest Impact on Achievement Means adjusted by previous year math scores, and FRL.

6. Lessons Learned from LASER Sentinel Site Visits Results from the 2007-2009 LASER Biennium

7. 7 Sentinel Site Selection Schools with significant LASER participation during the 3 years prior to the site visit Schools visited –34 schools during 2007-08 school year –30 schools during 2008-09 school year Used standard protocol and rubrics Defined 2 groups of schools based on science WASL change for 2 years prior to site visit –Demonstrated Significant Positive Gains –Demonstrated Little, No, or Negative Gains

8. 8 Sentinel Site Visits RMC Research staff or consultants Each site visit: 1 ½ to 2 days –Principal interview –At least 5 teacher survey & interviews –At least 3 classroom observations Data collection instruments –http://www.rmccorp.com/LASERSiteVisits/ Conducted web-based training session Results reported as rubric scores

9. 9 Finding 2 There were a number of traits that site visitors consistently rated high, many of which were a direct outcome of Washington State LASER.

10. 10 What Was Going Well At The School Level Materials Support System –The system for maintaining, storing, and refurbishing the instructional modules was effective and well organized. Condition of Modules –Teachers always received modules that were complete and ready for classroom use. Inquiry-Based Materials –The school was implementing 3 or more modules per grade level as the core science curriculum materials. Ninety-four percent of the teachers used the modules as the core science curriculum. Administrative Support –Strong evidence indicated that the school administrators were very supportive of inquiry- based science instruction. District Support –Strong evidence indicated that the district administrators were very supportive of inquiry- based science instruction. Sequence –All science teachers used the modules according to a sequence clearly prescribed by the district. Critical Mass –Most (80% or more) teachers in the school had attended the initial use training for all of the modules they used.

11. 11 School Level Areas Receiving Low Scores Summative District Assessments –Very few school had districtwide or schoolwide summative assessments in science that were administered to students annually Formative Assessments –A few teachers (25% or less) had adopted a standard formative assessment strategy for science. Instructional Time –Science instructional time varied considerably amount teachers at the elementary level and few schools required a minimum amount of instructional time for science. Professional Development Time for Teachers –Teachers rarely had scheduled time during normal work hours to participate in school-based professional development in science. Partnership With Business, Informal Science, or Higher Education –A few teachers had a tentative partnership with a business, an informal science organization, or an IHE around science education.

12. 12 Classroom Observation: High Scoring Alignment of Lesson Activities –Lesson activities addressed the stated learning objectives but there was some question about how the lesson activities would lead to a deeper student understanding of the learning objectives. Motivation –The lesson provided mostly extrinsic and some intrinsic motivation. The intrinsic motivation was truncated by the lesson structure and was relatively short lived. Understanding of Purpose –Throughout the lesson, many students understood why they were doing each activity but the purpose of activities could have been more explicit. Classroom Discourse –For the most part, students and teachers support and encourage respectful and constructive discourse around important science concepts, however, only some students feel comfortable asking questions, backing up their own claims, and/or critiquing claims made by others.

13. 13 Classroom Observation: Low Scoring Lesson Closure –By the end of the lesson, the teacher provided a brief review, but students did not have an opportunity to fully make sense out of how the lesson related to science concepts. Application of Science –A few students applied something they learned in the lesson to a new context. Reflection and Meta-cognition –By the end of the lesson, students had some opportunity to reflect on their thinking but students were not asked to identify ways in which their thinking about the science concepts had changed.

14. 14 Analysis of Gains Divided the schools into 2 groups –Based on: Percent of students who met the science standard Gains calculated over 2 years prior to the site visit –Group definition Schools that demonstrated an increase in student achievement Schools that had no change or decreasing student achievement The 2 groups of schools had significant demographic differences Percent of students who qualified for free or reduced price lunch Percent of Asian students

15. 15 Finding 3 Although schools that demonstrated increasing student science achievement were significantly different demographically from those that did not, there were also significant differences in the instructional practices of the teachers observed by the site visitors.

16. 16 Differences Between Gain Groups Science classes in schools that demonstrated an increase in student achievement were more likely to: –Have clear lesson objectives –Involve activities that clearly address the learning objective –Have students who understand the purpose of the lesson –Have students that are more intellectually engaged in the science content –Have students applying science content to new contexts –Have students engaged in science discourse –Motivate students intrinsically

17. 17 Analysis by Achievement Ranking Divided the schools into 2 groups –Based on percent of students who met the science standard the year of the site visit –Group definition Schools at or above the state average Schools below the state average Methods –Regression analysis to determine which variables were significant predictors of the achievement ranking –Controlling for Student achievement the previous year Percent of students who qualify for free or reduced price lunch

18. 18 Finding 4 Although schools that demonstrated above average student science achievement were significantly different demographically from those that were below average, there were also significant differences in the characteristics of the schools.

19. 19 Characteristics of Above Average Performing Schools Instructional Time Allocated –The elementary school has a designated amount of instructional time allocated for science. Professional Development Time for Teachers –Teachers occasionally had scheduled time during normal work hours to participate in school-based professional development in science. District Support –Evidence indicated that the district administrators were somewhat supportive of inquiry-based science instruction. Integration of Literacy –Many teachers integrated science and literacy through the use of supplementary reading materials and science notebooks. Parent and Community Support –Evidence indicated that the parents/community were somewhat supportive of inquiry-based science instruction.

20. 20 Principal Survey Online survey that closely paralleled data collected during sentinel site visits Selected schools with 15 hours or more of science professional development per teacher over a 5-year period prior to March 31, 2009 319 schools invited Survey administered between May 22 and July 31, 2009 62 principals completed the survey (19.4% return rate)

21. 21 Principal Survey Analysis Regression analysis to determine which survey items were significant predictors of student achievement on the 2009 science WASL Controlling for the percent of students who qualified for free or reduced price lunch (FRL)

22. 22 Finding 5 There were several items on the principal survey that were significant predictors of student performance on the science WASL above and beyond what could be attributed to FRL.

23. 23 Predictors of Student Performance Schools that made an organized effort to identify instructional materials to fill the gaps Schools where the principals observe student using evidence to engage in discourse about science. Schools that provide time during the normal work day for professional development and how often teachers participate in school-based science professional development. Schools that support professional learning communities that focus on improving science teaching and learning.

24. 24 Regression Analysis Results Survey Item Grade Tested Adjuste d R 2 Change in R 2 pBeta Has your school or district made an organized effort to identify instructional materials to fill the gaps? 5.459.114<.001 *.352 Principal observation of classes: Students had opportunities to make claims, and/or use evidence to back up their claims or critique claims made by others. The lesson reinforced the notion that science is a process by which knowledge is constructed. 5.563.122<.001 *.350 Is time scheduled during normal work hours for teachers in this school to participate in organized, school-based professional development specifically for science? 5.338.002.003 *.048 How often do teachers participate in school-based professional development specifically for science? 5.213.017<.001 *.153 Has any of the PLC activities focused on improving science teaching and learning? 5.476.031<.001 *.211 Approximately what percentage of the PLC is devoted to science teaching and learning? 5.434.002.013 *.047

25. 25 Finding 6 There are several consistent themes among the various findings that provide the basis for recommendations and conclusions that lead to a shift in Washington State LASER.

26. 26 Significant School-Level Factors PD time for teachers during the school day Allocated science instructional time Professional learning communities Filling curriculum gaps Integration of literacy District, parent, and community support

27. 27 Significant Instructional Factors Science discourse using evidence Purposeful instruction Intellectual engagement Intrinsic motivation Application of science skills

28. 28 Conclusion The infrastructure to support the use of a core curriculum of inquiry-based science instructional modules is in place and is functioning adequately in the schools visited. Although these conditions are necessary for the implementation of inquiry-based science instruction, they are not sufficient to raise student achievement as measured by the science WASL.

29. 29 Recommendation 1 Ensure that the professional development on research-based instructional practices is consistent and explicit across all of the LASER Alliances –Help teachers understand the elements of effective science instruction and use the modules as a means of carrying out the element with their students.

30. 30 Recommendation 2 Increase support for school-based professional development that helps teachers: –Assume accountability for student learning that results from the use of the modules, and –Collaboratively implement the elements of effective science instruction. –Ample structure and leadership for success

31. 3 Common Pitfalls to Improvement

32. 32 Common Pitfalls Why haven’t past school reform efforts been more successful? Three Common Pitfalls: –Lack of Vision –Lack of Process –Atomized Culture

33. 33 Common Pitfall #1 Lack of Vision –“In most instances, principals, lead teachers, and system-level administrators are trying to improve the performance of their schools without knowing what the actual practice would have to look like to get the results they want at the classroom level.” (City, 2009, p 32) –There is often a “lack of an agreed-upon definition of what high-quality instruction looks like.”

34. 34 Education Has Not Been Scientific Education has been criticized for not being “scientific.” We do know a lot about how students learn various subjects. The problem is that this knowledge is not consistently applied in daily instructional practices.

35. 35 Addressing #1—Lack of Vision “Develops, with colleagues who have to work together on school improvement, a shared understanding of what they mean by effective instruction.” (City, 2009, p 10) Build vision of effective instruction upon knowledge of how students learn the subject –Most instructional practices fail to effectively apply what research tells up about how students learn a given subject.

36. Examining Research

37. 37 Reading Research Converges Reading—50 years of research Effective reading instruction requires a balanced blend of: Phonemic awareness Decoding Vocabulary development Reading fluency, including oral reading skills Reading comprehension strategies Any single approach is inadequate

38. 38 Math Research Converges National Math Panel Report Effective mathematics instruction requires a balanced blend of: Computational fluency Concept development Problem solving Any single approach is inadequate

39. 39 What About Science?

40. 40 Science Is Different From Language Arts and Math Language Arts (& Reading) and Math –Are skills created by people –Involves learning established conventions Science –Understanding how the world works –How to create new knowledge –Living in the world creates a working understanding which may or may not be scientifically valid

41. 41 Importance of Learning Theory in Science Working knowledge is entrenched and difficult to overcome Sometimes called “Private Universe” Key concepts must be internalized –Sometimes called “Big Ideas” or “Enduring Understandings” Requires greater attention to the learning theory embodied in the instructional materials

42. 42 Research On Effective Science Instruction is Also Converging Considerable evidence from research shows that instruction is most effective when it elicits students’ initial ideas, provides them with opportunities to confront those ideas, helps them formulate new ideas based on evidence, and encourages students to reflect upon how their ideas have evolved. (Banilower, 2009)

43. 43 Untrenching Their Private Universe Without these opportunities, students “may fail to grasp the new concepts and information that are taught, or they may learn them for purposes of a test but revert to their preconceptions outside the classroom” (National Research Council, 2003, p. 14)

44. Therefore: Effective science instruction must be guided by what research tells us about how students learn science It is the duty of all science teacher to enact sound learning theory in science instruction

45. 45 LASER Theory of Action If teachers use research-based instructional practices, materials, and assessments so that students: –Draw upon a deep foundation of usable knowledge within the context of a conceptual framework to build scientific understanding –Are intellectually engaged and motivated –Reveal preconceptions and their initial reasoning –Use evidence to generate explanations –Communicate and critique their scientific ideas and the ideas of others –Have full access to activities and sense-making discussions to develop scientific understandings –Reflect on how personal understanding has changed over time and recognize cognitive processes that lead to changes

46. Then: The percent of students who meet or exceed the science standards on the state and other science assessments will increase, Student participation and success in rigorous science courses K–12 will increase, The number of students who seek further studies in STEM content and STEM careers will increase, and The number of students who choose to do science- based activities out of school time will increase. 46

47. 47 In Other Words Student science achievement will increase if we consistently apply what we know about how students learn science!

48. 48 Common Pitfall #2 Lack of a Process –Teachers attend professional development but then it is up to individuals to put the new knowledge and skills into practice –“[Schools] don’t have a process for translating that knowledge systematically into practice.”

49. 49 Addressing #2—Lack of Process Establish a school-based collaborative process that engages teachers in: –Regular structured observations of students for evidence that the vision of how students learn the subject is being enacted –Engage teachers in a process to make meaning from the observation experience to continuously improve instructional practices “Puts educators in a position of having to actively construct their own knowledge of effective instructional practice” (City, 2009, p 10)

50. 50 Common Pitfall #3 Atomized Culture –“Most people in schools work in siloed cultures characterized by independence and autonomy.” (City, 2009, p 62) –Too often schools “are organizations that support the private practice of teachers” and the atomized culture. –Atomized culture is one of the most significant barriers to school improvement.

51. 51 Addressing #3—Atomized Culture Deprivatize Practice –“It is clear that closed classroom doors will not help us educate all students to high levels.” –“Everyone is obligated to be knowledgeable about the common task of instructional improvement and everyone’s practice should be subject to scrutiny, critique, and improvement.” –“We can do more together than we can individually to improve learning and teaching.” (City, 2009, p 3)

52. 52 Addressing #3—Atomized Culture Importance of Observation –“The only way to find out what students are actually doing is to observe what they are doing.” –“What predicts performance is what students are actually doing.” (City, 2009, p 30)

53. Implications for Planning Things to Remember –The infrastructure and initial use training to support the use of a core curriculum of inquiry-based science instructional modules is necessary but not sufficient to raise student achievement in science –Fill curriculum gaps 53

54. Implications for Planning Embrace the LASER vision of effective science learning experiences for students embodied in the LASER theory of action –Ensure all professional development focuses on helping teachers enact that vision –Implementing the effective learning experiences for students is the purpose of professional development 54

55. Implications for Planning Establish an internal process for continuously improving instruction that: –Provides time within the school day for participation –Has sufficient structure –Has creditably leadership –Provides access to content expertise –Deprivitizes practice –Involves observation –Strives for horizontal and vertical articulation 55

56. 56 Questions ??? Dave Weaver RMC Research Corporation 111 SW Columbia, Suite 1200 Portland, Oregon 97201 dweaver@rmccorp.com (800) 788–1887 Presentation is at: http://www.rmccorp.com/LASER/

57. 57 “Must Read” List City, E., Elmore, R., Fiarman, S., & Tietel, L. (2009). Instructional rounds in education. Cambridge, MA: Harvard Education Press. Banilower, E., Cohen, K., Pasley, J., & Weiss, I. (2008). Effective science instruction: What does research tell us? Portsmouth, NH: RMC Research Corporation, Center on Instruction. National Research Council. (2003). How people learn: Brain, mind, experience, and school. J. D. Bransford, A. L. Brown, & R. R. Cocking (Eds.). Washington, DC: National Academy Press.