1. Using the 2011 NECAP Science Results New England Common Assessment Program

2. 2 Dr. Kevon Tucker-Seeley Assessment Specialist – NECAP Manager Office of Instruction, Assessment & Accountability 1-401-222-8494 Peter McLaren Science and Technology Specialist Office of Instruction, Assessment & Accountability 1-401-222-8454 Jennifer Golenia Science and Technology Specialist Office of Instruction, Assessment & Accountability 1-401-222-8406 Lindsay Wepman Assessment Specialist Office of Instruction, Assessment & Accountability 1-401-222-8254 Welcome and RIDE Introductions

3. NECAP Service Center 1-877-632-7774 Elliot Dunn NECAP Science Program Manager 603-749-9102 ext. 2126 dunn.elliot@measuredprogress.org Harold Stephens NECAP Program Director 603-749-9102 ext. 2235 stephens.harold@measuredprogress.org Carole Soule NECAP Program Manager 603-749-9102 ext. 2450 soule.carole@measuredprogress.org 3 Measured Progress Introductions

4. NECAP Service Center: 1-877-632-7774 Kellie Beaulieu: NECAP Program Assistant 603-749-9102 ext. 2156 beaulieu.kellie@measuredprogress.org Mellicent Friddell: NECAP Program Assistant 603-749-9102 ext. 2355 friddell.mellicent@measuredprogress.org Alison Cady: NECAP Program Assistant 603-749-9102 ext. 2149 cady.alison@measuredprogress.org 4 Measured Progress Introductions

5. Review the different types of NECAP Science reports and share State results Demonstrate a simple data analysis protocol using NECAP Science Data Share some observations from the NECAP benchmarking process 5 Goals of the Workshop

6. 6 Workshop Reference Materials

7. 7 Types of NECAP Science Reports Student Report (Confidential) Information for Parents Item Analysis Report (Confidential) School level by student Results Report (Public) School and District level Summary Report (Public) District/State level Student Level Data Files (Confidential) Excel/csv files by grade on district and school confidential site

8. 8 http://iservices.measuredprogress.org Accessing Your Confidential Reports

9. Select “Interactive” to view Interactive Reports Select “Reports” to view Static Reports 9 Accessing Your Confidential Reports

10. These accounts are intended for the school teachers who will see only the students to whom they have been assigned by the principal. This account is intended for the school principal. One school principal account exists for each school. The principal assigns all accounts for teachers within the school This account is for the district level user and allows access to all reports. District Administrator (Superintendent) School Level (Principal) Classroom (Teacher) Classroom (Teacher) Classroom (Teacher) Classroom (Teacher) NECAP Analysis and Reporting System (NARS) Account Creation Hierarchy Password Assistance: Use the following list to determine whom to contact for assistance with your User Name and Password: Superintendents – Contact NECAP Service Center at 877-632-7774 Principals – Contact NECAP Service Center at 877-632-7774 Teachers – Contact School Principal http://www.ride.ri.gov/assessment/Results.aspx

11. Using NECAP Science Data Focus is improving student learning – The goal is to increase student achievement Engage in collaborative discourse about data – Using relevant, timely data to influence educational decisions is considered a best practice Discussion of data requires a structured approach – Protocols, ground rules, and shared talk time Remember, NECAP Science data is for assessing school-level achievement, NOT individual student achievement – Look for trends and patterns

12. NECAP Science Data Today we’ll take a look at: Demo District Results Demo School Results Test Items and Item Analysis Reports State Results

13. Framing the Question Collecting Data Analyzing Data Organizing Data-Driven Dialogue Drawing Conclusions – Taking Action Monitoring Results Phases of Collaborative Inquiry Adapted from N. Love Using NECAP Science Data

14. Using Data―A Simple Approach 1.Observe the data – Look at the data with your table. – Make objective statements about the data. Avoid statements like, “It seems…” or “I think…” – What do you notice?

15. Using Data―A Simple Approach 2. Discuss – Talk with your colleagues at your table. – Why do think the data might look this way?

16. Using Data―A Simple Approach 3. Action steps – What are some other sources that could confirm or refute this data? – What does the data mean for my school regarding curriculum and instruction? – How can I dig deeper?

17. Using Data―A Simple Approach 4. Reflection (time permitting) – What did you learn from this activity? – Are there new things that you’d like to bring back to your school?

18. Tying it Together Limitations – This is one source of data. You can’t make major programmatic changes from this alone. – Look to other sources of data to confirm or refute. – Think about curricular and instructional action steps. Look for professional development around data use in the 2012–13 and 2013–14 school years.

19. Percentage of Rhode Island Students by Achievement Level GradeYearSubjectSBPPPPPwDP+PwDCHG Average SS 410/11SCI1838431440438 409/10SCI2036431444437 408/09SCI2040 <140---437 810/11SCI2946241253833 809/10SCI314722<1225832 808/09SCI384517<117---831 1110/11SCI25492512651134 1109/10SCI32482012111133 1108/09SCI364519120---1132

20. 10/11 Comparison of NECAP States by Achievement Level GradeStateSubjectSBPPPPPwDP+PwDCHG Average SS 4RISci.1838431440438 4NHSci.836541551441 4VTSci.--- 8RISci.2946241253833 8NHSci.2152271281835 8VTSci.--- 11RISci.25492512651134 11NHSci.26472612731134 11VTSci.--- Note: Vermont’s data is embargoed until 9/28. RI will do a public release on 9/27

21. Benchmarking Short Answer and Constructed Response Items Short answer items receive a score from zero to two. Constructed response items receive a score from zero to three or zero to four. Zeros are employed when a student produces some work, but the work is totally wrong or irrelevant or if he or she leaves the item blank. For purposes of aggregating item results, blanks and zeros both count as zero points towards a student’s score.

22. Preparation for Benchmarking The work in preparation for scoring student responses included: Development of scoring guides (rubrics) Content specialists from the NH, RI, and VT Departments of Education Measured Progress’s test developers Selection of “benchmark” responses Examples of student work at different score points for each item Used in training and continuous monitoring of scorer accuracy

23. Scorer Training Scorer training consisted of: Review of each item and its related content and performance standard Review and discussion of the scoring guide and multiple sets of benchmark responses for each score point Qualifying rounds of scoring in which scorers needed to demonstrate a prescribed level of accuracy.

24. NECAP Benchmarking Process Sample Responses Pulled Lead Scorer chooses samples of responses from open response items (CRs or SAs) Sample responses are chosen as representative of all responses. Sample responses are initially scored based upon rubric for the item. Science Specialists Science Specialists from NECAP States review the sample items and come to consensus on score based upon rubric “Anchor” Papers are selected for scorers to use as a guide when assessments are officially scored. Science specialists also review rubrics to help provide clarity for scorers. Scorers All scorers are trained and tested (10 response qualifying round) Scorers have to pass test in order to move to officially scoring items.

25. Lessons Learned from NECAP Benchmarking What does it mean – to identify? – to describe? – to explain? How does this look over grade levels? What does it mean to compare and contrast?

26. Lessons Learned Grade 4 What is a characteristic? What observations of information can be gained from charts and diagrams? What patterns can be distinguished from graphs? How can students better use evidence from data in their explanations?

27. Lessons Learned Grade 8 When should line graphs be used? When should bar graphs be used? How can students better support or refute their predictions/hypotheses using evidence from data? Why is it important to use multiple trials in an investigation?

28. Lessons Learned Grade 11 What are typical sources of error? – How can they affect the outcomes of an investigation? How can skills in the analysis of data be demonstrated more deeply? By 11 th grade, students should know about the use of proper units of measurement.

29. Development of Assessment Targets Participation in annual test item review committee & bias and sensitivity review committee Use of classroom teacher judgment data Participation in standard setting panels Technical Advisory Committee work 29 Involvement of Local Educators in NECAP Science

30. 30 Use the data Share progress with parents Identify gaps in curriculum, instruction, and assessment Identify students in need of accommodations Anticipate types of accommodations NimbleTools will be an option – Spring 2012 Plan for administration now Review what went well and what went not so well during the 2011 administration Begin preparing your students Share the reference sheets Released Items documents Preparing Students for NECAP: Tips for Teachers to Share with Students Preparing for 2012 NECAP Science

31. 31 Questions? Peter McLaren Science and Technology Specialist Office of Instruction, Assessment & Accountability Peter.mclaren@ride.ri.gov 1-401-222-8454 Jennifer Golenia Science and Technology Specialist Office of Instruction, Assessment & Accountability Jennifer.golenia@ride.ri.gov 1-401-222-8406