1. Evaluation of An Urban Natural Science Initiative
Kathy Gullie
Dianna Newman
Justin Chase
University at Albany/SUNY
verbally state, at the beginning of your presentation, that
The instructional practices, materials, and assessments were chosen to meet the needs of
your program. Please point out that other options exist, and the use of the practices,
materials, and assessments chosen by your project is not intended as an endorsement
by the U. S. Department of Education.
AEA Annual Meeting, Washington D.C., October 2013

2. A sign of the times More is better: Exploring the additive effect of professional development on student science exam outcomes

3. Purpose of the Mathematics Science Partnership (MSP) Science Initiative Program
Urban settings : To improve teacher depth of knowledge of and inquiry base instructional practices in elementary natural science that would yield greater student knowledge of natural science
District accountability: To improve student outcome on State and local tests

4. Project Description 20,000+ Students
53% Black, 28% White, 12% Hispanic
Large Up-State NY Urban District
52 Elementary Teachers
Grades K-4
25 Elementary School Buildings
Participant District
SUNY College at Cortland
IHE faculty and staff
SUNY College of Environmental Science and Forestry
Cornell University Lending Library (NSF supported)
NY State Environmental Agency Experts; Parks and Recreation Department
Community Agencies

5. Professional Development
Project Design
Professional Development
Goal
Participants
Outcome
Hours
30 hours per year per teacher
Improve content knowledge and pedagogy
52 Teachers
Participation Logs
Content
Stipend based direct training
Increase knowledge of natural science
IHE partners
Community experts
Increase in teacher content knowledge
Pedagogy
Site-based, push-in
Mentoring
Coaching
Modeling
Lesson Study
Increase in teacher understanding and use of problem solving and inquiry based instruction
District Science IST
Change in teacher practice
Change in student knowledge 5

6. Evaluation Design: Drill Down Variables*
Teacher Knowledge of Natural Science:
Observations of PD process
Results of teacher outcomes on the SCKALE
Pedagogy and Transition to Practice:
Documentation of mentoring, modeling and lesson study
Observations of classroom instruction
Observations of inquiry based interactions
Quasi-experimental designs for student outcomes:
Pre-post test results on locally developed curriculum tests and the NY state grade 4 science test.
post test comparison results with control groups **
Methodology
Alignment with NYS Science Standards
(Key Ideas and Performance Indicators)
Professional Development Units
Teacher Content Test
Teacher-made Student Unit Tests
NYS th grade science exams
Classical Item Analyses
T-Tests to examine differences between:
MSP and Non-MSP students (Independent T-Tests)
Teacher growth and sustainability (Dependent T-Tests)
New York State Education Department Office of Assessment Policy, Development and Administration
*reliability and validity data available on request
** based on NYS tests 6

7. Change in Teacher Knowledge**
Pre- post-test results on SCKALE
Living Environments
/Life Sciences n
Pre
SCKALE
Post
Dependent
t-test
Mean
St. Dev
Year One 36
8.69
1.26
9.47
0.91
4.47*
Year Two 26
6.69
1.62
7.73
.067
2.78*
Year Three 33
18.18
7.70
18.88
1.58
2.09*
*p< .05 (two-tailed)
** Validated items from New York State Science Tests (Living Environment sections)

8. Documentation of Change in Practice: Science Curriculum
Development and implementation of New Science modules (n=6)
Validated through piloting lessons and peer (lesson study) and evaluators observations (n=51) as well as post implementation interviews (n=27)
Documentation of formative and summative assessments within classroom observations (n=44)
Findings from end-of-year surveys documented participants’ follow-up perception of professional development and were triangulated with observations of classrooms observations and documentation through Lesson study activities that focused on successful use of newly acquired strategies
Evaluators noted changes in instructional style and subsequent gains in science achievement for students whose teachers were involved in MSP. There is evidence that professional development focusing on science content and science pedagogy is transforming teachers’ instructional style and students’ science achievement

9. Change in Practice: Instruction
Observed and reported changes in practice
Percent Agreement
Observed increases in
willingness to share, model, explain and coordinate planning (98%)
preparation of students for hands-on activities (63%)
preparing students for problem solving, discussion and application (84%)
Engaging students in higher level thinking skills (73%)
in-depth student content discussions (38%)
Self reported increases in:
Willingness to share model and explain (98%)
use of hands-on/laboratory science activities in the classroom and use of locally available resources (92%)
developing students' conceptual understanding of science (77%)
Asking students to explain concepts to one another (67%)
Development of curriculum objectives that allow student learning through communicate ideas in science effectively (90%).

10. NYS 4th Grade State Science Test
Change in Student Knowledge
NYS 4th Grade State Science Test
More MSP students (+3%) were recording scores at Level 3 and 4, than at baseline, i.e. Year One of the grant.
More MSP students (+4%: 71%) met proficiency levels than students (67%) of non-MSP teachers at the 4th grade in year three
An increase can also be seen in the non-MSP population (+2%), indicating that scores may have been impacted by students having had an MSP trained teacher in one and/or two previous years.
At baseline 51% of students correctly answered Living Environment questions correctly compared to 68% in Year Three (+17%).
% At performance level
Test date
Number
Tested
Level 4
Level 3
Level 2
Level 1
Year 1
MSP 4th grades
259 22 46 11
Non-MSP 4th grades
1381 26 39 20 14
Year 2
267 32 40 17
1252 31 36 13
Year 3**
213 30 41 18 10
1224 33 34 21 12
Data also outlined a significant increase from baseline on Living Environment questions. Overall at baseline in Year One 51% of students correctly answered Living Environment questions correctly compared to 68% in Year Three (+17%).

11. Change in Student Knowledge
Comparisons of pre/post-test results on local curriculum
in MSP classrooms
Percent of students scoring at level
Curriculum tests were designed based on the New York State Science Standards and reflected standards based on material developed within grant related curriculum.
Findings indicate that students taught by a MSP participant had higher post-test scores than pre-test, for all units for each grade level, over all three years.
Students taught by MSP teachers increased significantly in environmental science knowledge presented through the grant developed science curriculum units at grade 2-4.

12. Change in Student Knowledge
Comparison of Post-test Results with Control Groups
When MSP students were compared to Non-MSP students on unit posttests, those MSP students in younger grade levels (2-3), tended to perform better. Students in grade 4 performed better, but not significantly
Unit Post-Assessment n
MSP
Non-MSP df
Independent
t-test
Mean
St. Dev.
Year One
Grade 2: Unit 1
182
7.00
2.79 69
5.00
1.88
3.9*
Grade 3: Unit 1
194
10.18
3.07 75
8.15
2.74
149
5.0*
Grade 4: Unit 1
153
11.94 80
10.35
2.80
170
4.2*
Year Two
247
2.27 93
6.00
2.08
338
3.8*
200
10.53
3.28 28
7.50
2.45
226
4.7*
132
12.13
3.00 57
11.44
2.89
187
1.4
Year Three
Grade 2: Unit 2 95
8.82
2.09 60
9.68
2.33
2.3*
Grade 3: Unit 2
14.49
5.6 19
11.32
4.5 74
2.2*
108
11.73
2.7 21
10.90
2.41
127

13. Additive Effect of Exposure to Multiple MSP teachers
Years with
MSP Teacher
4th Grade NY State Science Test n
% met proficiency
One Year
375 70
Two Years 97 84
Data analysis confirmed an additive effect on students who worked with MSP teacher for Two Years when compared to those students who worked with MSP teachers for only One Year.
Data was insufficient (n=18) when looking at whether three years with an MSP teacher supported an additive effect.

14. Successful Evaluation Documentation Design
Development of a method of evaluating increasing teacher knowledge focusing on sustainability
One
Documentation process for evaluating longitudinal impact on pedagogy (interest, curriculum and instructional practices)
Two
Documentation process for evaluating impact on students (increased environmental science knowledge, affect, and interest)
Three

15. For more information:
Kathy Gullie Dianna Newman Evaluation Consortium Dutch Quad B7 University at Albany/SUNY Albany, New York