2. SGO 2.0: from Compliance to Quality Increasing SGO Quality through Better Assessments and Target Setting 1

3. SGOs 2013-14 1.Measured using poor quality assessments, 2.Designed using a falsely precise growth measure derived from a pre-test post-test model, 3.Not differentiated to take into account starting points of students, 4.Developed using only one measure of prior learning, i.e. pre-test score, and 5.Administrator-driven, paperwork heavy, and only partly reflective of typical teacher work 2 In too many cases, SGOs were:

4. SGOs 2.0 2014-15 1.Use high quality assessments to measure identified critical standards that all students should learn/master, 2.Use multiple measures to get a rough sense of student starting points, 3.Group students and set differentiated targets based on preparedness level, and 4.Are teacher-driven and more authentically reflective of the work of effective teachers. 3 The Department encourages educators to develop SGOs that:

5. SGOs 2.0 2014-15 4 UnderstandTake Action 1.SGOs are learning goals for key concepts and skills that students can be expected to master in a course based on an approximate sense of where they start.  Base learning goals on what you want students to know and do by the end of the SGO period.  Get a rough sense of where students begin by using multiple measures of student prior learning (see example). example  Use pre-assessments only when appropriate. 1.SGO quality is critically dependent on summative assessment* quality.  Increase the quality of the SGO summative assessments and develop common assessments where possible. (SGO 2.0 Presentation)SGO 2.0 Presentation 1.SGOs should be a true reflection of the daily practice of effective teachers and of the curriculum and students an educator teaches. (2013-14: Lessons from Educators, section 6)2013-14: Lessons from Educators, section 6  Align critical standards, effective instruction, and high quality assessment in SGOs.  Incorporate a significant number of students and portion of curriculum within the SGO(s) (see SGO Quality Rating Rubric).SGO Quality Rating Rubric  Set differentiated learning goals for students based on their starting points. 1.SGOs should be collaborative – teacher-driven, administrator- supported, and student-centered (as stated in code 6A:10-4.2 (e) 3).6A:10-4.2 (e) 3  Even though administrators are responsible for approving and scoring SGOs, they should encourage teachers to take ownership of the SGO process as a powerful way to improve teacher practice and student achievement.

6. SGOs 2.0 Resources 5 SGO Webpage

7. Clarify what SGOs are and what they are not. Part 1 6

8. For Educators SGOs provide a method by which teachers can improve their practice through high quality goal setting while clearly demonstrating their effectiveness through the learning exhibited by the students for whom they are responsible The Value of SGOs For Evaluators SGOs provide an authentic measure of teacher effectiveness that is aligned to the learning exhibited by students through an educator’s daily practice of teaching For Students When well-designed, SGOs promote reflective and collaborative teaching practices, alignment among standards, instruction and assessment, and improve student learning. 7

9. What SGOs Are, and What They Are Not Misconception SGOs need to be a significant addition to the work of a teacher. Reality SGOs should be a reflection of what effective teachers typically do. #1 8

10. Standards Instruction Assessment SGO SGOs should be a reflection of what effective teachers typically do 9

11. Excerpt from SGO Quality Rating Rubric Excellent Number of students in combined SGOs represents all or a large majority of the teacher’s students. Includes start and stop dates that include a significant proportion of the school year/course length. Includes a significant proportion of standards for which the teacher is responsible during the instructional period. 10 SGO Quality Rating Rubric

12. 11 General and Specific SGOs GeneralSpecific Captures a significant proportion of the students and key standards for a given course or subject area Focuses on a particular subgroup of students, and/or specific content or skill For teachers whose general SGO already includes all of their students, or those who receive an SGP Most teachers will be setting this type of SGO SGO 1.0 2013-14

13. NameSchoolGradeCourse/Subject Number of Students Interval of Instruction 9 Physics 1 55/55 October-April Rationale for Student Growth Objective Name the content standards covered, state the rationale for how these standards are critical for the next level of the subject, other academic disciplines, and/or life/college/career? Name and briefly describe the format of the assessment method. Standards NJCCCS physical science 5.2.12 C, D and E NJCCCS science practices 5.1.12 A-D Impact of Standards  This SGO includes all of the NJCCCS related to physics creating a foundation important for students who will take AP and/or college-level physics and is fundamental to many careers including architecture, mechanics, engineering, medicine.  The SGO also includes all of the science practice standards, standards crucial in helping student become scientific thinkers. This mindset is valuable for making decisions when a large amount of information is available and must be analyzed for value and accuracy. It is critical in most academic disciplines. Assessment Physics department’s common assessment administered at the end of the 3 rd marking period Written: 60 multiple choice (4 choice), 5 short response questions, Practical: Students design a simple apparatus, take measurements and collect data. High quality test normally administered at this time 2014-15 SGO Form Significant proportion of students, standards and course 12 2014-15 SGO Form

14. What SGOs Are, and What They Are Not Misconception SGOs are an administrator-driven compliance exercise Reality SGOs are driven by teachers, supported by administrators, and centered on student learning #2 13

15. SGOs are driven by teachers, supported by administrators, and centered on student achievement 14

16. Develop a foundational understanding of how to develop and choose high quality assessments. Part 2 15

17. Teachers may use but are not limited to: Portfolios Performance Assessments Benchmark Assessments Finals (modified as needed) Program-based Assessments Standardized Tests, e.g. AP 16 Types of Assessments for SGOs Whether locally-developed or commercial, multiple choice or rubric-based, assessments should follow the rules of good assessment design.

18. What Does Good Assessment Look Like? 17

19. Elements of Assessment Design Purpose 18 Note Taking Handout

20. Elements of Assessment Design Begin with the End in Mind Purpose SGO assessments are measures of how well our students have met the learning goals we have set for them 19

21. Elements of Assessment Design Align to Standards 20

22. Elements of Assessment Design Align to Standards 21

23. Given limited resources, especially time, on which standards do we focus our SGOs and assessments? Elements of Assessment Design Align to Standards 22

24. Determine the relative importance of the standard being taught during the SGO period 1.How much time is spent teaching the standard? 2.Does the standard have value beyond the current course in: i.the next level of the subject, ii.other academic disciplines, or iii.life/college/career? Determine the relative importance of the standard being taught during the SGO period 1.How much time is spent teaching the standard? 2.Does the standard have value beyond the current course in: i.the next level of the subject, ii.other academic disciplines, or iii.life/college/career? Elements of Assessment Design Align to Standards 23

25. 2014-15 SGO Form Rationale for Student Growth Objective Name the content standards covered, state the rationale for how these standards are critical for the next level of the subject, other academic disciplines, and/or life/college/career. Name and briefly describe the format of the assessment method. 24 Standards NJCCCS physical science 5.2.12 C, D and E NJCCCS science practices 5.1.12 A-D  This SGO includes all of the NJCCCS related to physics creating a foundation important for students who will take AP and/or college-level physics and is fundamental to many careers including architecture, mechanics, engineering, medicine.  The SGO also includes all of the science practice standards, standards crucial in helping student become scientific thinkers. This mindset is valuable for making decisions when a large amount of information is available and must be analyzed for value and accuracy. It is critical in most academic disciplines.

26. Using Commercial Products for SGOs 25

27. Elements of Assessment Design Valid/Accurate Inferences 26

28. Elements of Assessment Design Valid/ Accurate Inferences 27

29. Elements of Assessment Design Valid/Accurate Inferences Why does it matter? The assessment should measure what it sets out to measure. What does it look like? The assessment is aligned to standards, skills, and rigor of the instruction and content of the course. The assessment is accessible to all students. 28 Valid/ Accurate Inferences

30. Analyze This Item Perhaps the most famous of all the arts of the Ming Era was: A.the elaborate puzzles of the period, which were popular even in Europe. B.blue-and-white porcelain, which Europeans collected in great quantities. C.the construction of large, elaborate palaces, the finest example of which is the Imperial City in Beijing. D.high-quality Berber rugs, which are still popular today. 6.2.12.C.1.b - Trace the movement of essential commodities (e.g., sugar, cotton) from Asia to Europe to America, and determine the impact trade on the New World’s economy and society. How valid is the inference we can make about student learning using this question? How can we make this a better assessment item? How valid is the inference we can make about student learning using this question? How can we make this a better assessment item? 29 Handout

31. 30

32. Item is not aligned to standards 6.2.12.C.1.b - Trace the movement of essential commodities (e.g., sugar, cotton) from Asia to Europe to America, and determine the impact trade on the New World’s economy and society. essential commodities Perhaps the most famous of all the arts of the Ming Era was: A.the elaborate puzzles of the period, which were popular even in Europe. B.blue-and-white porcelain, which Europeans collected in great quantities. C.the construction of large, elaborate palaces, the finest example of which is the Imperial City in Beijing. D.high-quality Berber rugs, which are still popular today. arts of the Ming Era 31

33. Elements of Assessment Design Range of Rigor/Depth of Knowledge Why does it matter? An assessment that accurately reflects the range of rigor of the course and instruction increases the validity of inferences educators can make about student learning. Provides access points to students of varying ability. What does it look like? The assessment requires a range of thinking skills as proposed by Bloom’s taxonomy and Webb’s Depth of Knowledge (DOK) that reflects the rigor of the course. Range of Rigor/DOK 32

34. Elements of Assessment Design Depth of Knowledge Wheel 4 minute video explaining DOK using the Gettysburg Addressvideo Range of Rigor/DOK 33 Handout

35. Determine the Rigor of this Item Examine the following political cartoon and answer the following questions. 1.What does the snake in this cartoon represent? 2.Whom is the snake attacking? What DOK level does this item represent? What modifications could you make to the question to make it more rigorous? 34 Handout

36. Determine the Rigor of this Item Examine the following political cartoon. Use details from the cartoon to: 1. Explain the symbolism of the snake in the political cartoon. 2.Explain why the artist used children to represent free press, free speech, and honest opinion. What DOK level does this modified item represent? 35

37. A high quality assessment has a range of rigor that: Is representative of the rigor of instructional level and content delivered in the course, and Provides stretch at both ends of ability levels A high quality assessment has a range of rigor that: Is representative of the rigor of instructional level and content delivered in the course, and Provides stretch at both ends of ability levels Elements of Assessment Design NOT Rigor for Rigor’s Sake Range of Rigor/DOK 36

38. Elements of Assessment Design Accessible Assessment Why does it matter? Promotes similar interpretations of the data and informs sound instructional decisions. It’s fair to all students. What does it look like? Provides equal access to all students regardless of personal characteristics/background and pre-existing extra-curricular knowledge. Questions and structure do not disadvantage students from certain groups or those without particular background knowledge. Appropriate modifications for students with learning plans. Format, wording, and instructions are clear. Accessible 37

39. Directions: Choose the one answer that best solves the problem. If one card is taken at random from a deck of playing cards, what is the probability that the card will be an ace? A) 8% B) 50% C) 25% D) 10% Directions: Choose the one answer that best solves the problem. There are 4 aces in a deck of 52 playing cards. If one card is taken at random from the deck, what is the probability that the card will be an ace? A) 8% B) 50% C) 25% D) 10% Examples 38

40. Examples 39

41. Elements of Assessment Design Reliable/Consistent 40

42. Elements of Assessment Design Unreliable Reliable Reliable/ Consistent 41

43. Elements of Assessment Design Reliable/ Consistent Reliable/Consistent Assessment Why does it matter? Provides information about student learning that can be trusted. What does it look like? Assessment administration and scoring is standardized and comparable. Assessment items yield consistent results over time. 42

44. Turn and Talk Provide a physical and emotional environment that encourages students to do their best. Provide clear directions and scoring criteria to students before they start the assessment. Allow enough time to complete the assessment. Make the assessment long enough (longer assessments are generally more reliable). Ensure scoring is done by educators trained using clear criteria; use multiple scorers when possible. Keep the assessment secure before and after test. 43

45. Elements of Assessment Design Bringing the elements together into a coherent whole Blueprint 44

46. Elements of Assessment Design A blueprint document describes the content and structure of an assessment. It defines the: Standards measured Relative importance of the standards on the assessment Item types, number and point value DOK of each item Blueprint 45

47. Elements of Assessment Design Blueprint 2013-14 46

48. Elements of Assessment Design PRIOR TO TEST DESIGNDURING TEST DESIGN Standard and Description of Standard (NJCCCS, CCSS, etc.) Relative Importance of Standard 4= High 3= Medium-high 2= Medium-low 1= Low Type of Question (multiple-choice, constructed- response, essay, etc.) Depth of Knowledge of Question 4= Extended Thinking 3 = Strategic Thinking 2 = Skill/ Concept 1 = Recall Question Number/ Points Total Point Value/ Percentage of Test 4.NBT.B.4 Add and subtract multi-digit whole numbers 4 MC2#1/5 pts 30 pts /10% MC3#3/5 pts CR3#6 /20 pts Blueprint 47 Assessment Blueprint and Completion Guide

49. Elements of Assessment Design AFTER TEST DESIGN CHECKLIST  Is the assessment of a length and format that is appropriate for subject/grade level?  Is the complete assessment and each assessment item accessible to all students?  Can the assessment be administered under comparable conditions across classrooms?  Can the assessment be scored consistently with a readily accessible scoring guide and/or rubric?  Does each item follow the rules of assessment item design? Blueprint 48 Rules of Item Design

50. Investigate appropriate ways to set targets using readily available student data. Part 3 49

51. What SGOs Are, and What They Are Not MisconceptionReality SGOs are learning targets for key concepts and skills that students can be expected to master in a course based on a rough sense of where they start. #3 SGOs are a statistically precise measure of growth based on a pre-test/post- test model of performance. 50

52. Pre-tests- The Siren Song of Simplicity 51

53. Important Considerations if Using the Pre-test Post-test Model Inherent Testing Error Error, present in all tests, is compounded in a pre- post- model, and often greater than the learning gains of the students. Reliability of Results Especially in Pre-test “Don’t worry about it – this doesn’t count.” Stretches Teacher and Student Capacity Two high quality assessments must be developed and administered. Unnecessary tests can interfere with other important work occurring at the start of the school year. Lack of Value for Instructional Purposes “Yep, just as I thought – my kids don’t know any Mandarin yet.” Difficult to Set Reasonable Targets Impossible to extrapolate future learning from one data point. 52

54. What is the Alternative to Pre-/Post-testing Model for SGOs? Create learning targets for key concepts and skills that students can be expected to master in a course based on a rough sense of where they start using a variety of typically-collected information about student learning 53

55. Predict the Final Picture 54

56. Predict the Final Picture 55

57. Predict the Final Picture 56

58. Predicting Student Learning Based on a Rough Sense of Where They Begin ? ? ? 57 Expected learning cannot be determined using one data point. Expected learning is better- determined using multiple measures of starting points.

59. 1. 2. 3. 4. 5. List the information you have used or could potentially use to determine students’ starting points. 58

60. 1. Current grades 2. Recent test performance 3. Previous year’s scores 4. Important markers of future success 5.Well-constructed and administered, high- quality pre-assessments List the information you have used or could potentially use to determine students’ starting points. 59

61. Sample Rubric for Important Markers of Future Success CriterionLevel 4Level 3Level 2Level 1 Active Participant Always prepared Engaged in all of the learning process Mostly prepared Engaged in most of the learning process Sometimes prepared Engaged in some of the learning process Rarely prepared Engaged in little or none of the learning process Academic Independence Consistently demonstrates intellectual curiosity Consistently self- motivated and independent Frequently demonstrates intellectual curiosity Usually self- motivated and independent Sometimes demonstrates intellectual curiosity Sometimes self- motivated and independent Rarely demonstrates intellectual curiosity Rarely or never self- motivated, frequently depends on prompting and/or teacher assistance Class Attendance Never absentRarely absentSometimes absentFrequently absent 60 Rubric for Important Markers of Future Success

62. What data could be used to get a rough sense of students’ starting points? STUDENT PRIOR YEAR TEST SCORES CURRENT YEAR TEST SCORES MARKERS OF FUTURE SUCCESS (see rubric) PREPAREDNESS GROUP Unit 1Unit 2 Average Score Active Participant (1-4) Attendance (1-4) Academic Independence (1-4) Total Points 1. 2521009798.543310 High 2. 201628372.52437 Medium 3. 1435775662136 Low Prior Year Test Score Current Year Test Score Average Markers of Future Success Preparedness Group 250 – 30085 – 1009-12High 200 – 24970 – 845-8Medium <200<700-4Low Key 61

63. Student ID Prior Test Scores Current Year Test ScoresMarkers of Future Success Preparedness Group NJ ASK 8 Math Unit 1Unit 2 Average Score Participates in Class Completes Retakes Completes Homework Total Points 12301009798.5Yes No2High 2202909592.5Yes 3High 321195 Yes 3High 4241858685.5YesNo 1High 5263909291YesNoYes2High 6284908587.5YesNoYes2High 7199918889.5Yes 3High 8201577566NoYesNo1Low 9144505854No 0Low 1018258 No 0Low 11143628372.5Yes No2Medium 12171788380.5NoYesNo1Medium NJ ASK Math Score Current Year Test Score Average Number of Future Success Markers Preparedness Group Target Score on Summative <200<700 – 1Low70 200 – 24970 – 841 – 2Medium80 250 – 300 85 – 1002 – 3High90 Physics 1 SGO Using Multiple Measures of Starting Points

64. Previous Year’s Test Score Current Year Test Score Average Markers of Future Success (points) Preparedness Group Target Score on Summative 79 or below< 700-6Low 70 80-8970-847-9Medium 80 90 or above85-10010-12High90 Determining Appropriate Target Scores 63 What level of performance on the assessment would indicate a sense of competence/mastery of the content and skills I am teaching? How should I modify this performance target based on the preparedness level of my students. Target Score on Summative

65. 64

66. What is the appropriate role of pre-assessments in SGOs? Where improvement in a set of skills is being evaluated When assessments are high quality and vertically aligned When normally used for instructional purposes In combination with other measures to help group students according to preparedness level 65

67. Effectively using high quality pre-assessments in combination with other data to set targets Student Initial DRA Level High Frequency Word Recognition Markers of Future Success Preparedness Group DRA Target 1. 325 7 Medium14 2. 126 4 Low4 3. 335 8 Medium14 4. 662 10 High18 Grade 1 Reading 66

68. 2014-15 SGO Form Starting Points and Preparedness Groupings State the type of information being used to determine starting points and summarize scores for each type by group. Add or subtract columns and rows as needed to match number of preparedness groups and types of information used. Preparedness Group Information #1Information #2Information #3 Prior Year Test Score Current Year Test Score Average Markers of Future Success High 250 – 300 85 – 1009-12 Medium200 – 24970 – 845-8 Low<200<700-4 Preparedness Group Prior Year Test Score Current Year Test Score Average Markers of Future Success High 250 – 300 85 – 1009-12 Medium200 – 24970 – 845-8 Low<200<700-4 67 Preparedness Group Information #1Information #2Information #3

69. Scoring Plan Preparedness Group Target Score on Final Assessment Objective Attainment Level Based on Percent of Students Achieving Target Score Exceptional Attainment (4) Full Attainment (3) Partial Attainment (2) Insufficient Attainment (1) 4≥70 >85% students≥70% students≥55% students<55% students 3≥80 2≥85 1≥90 9 th -grade Physics Using Locally-developed Assessment Large Numbers of Students – Percentage of Students Reaching a Stated Target

70. Scoring Plan Preparedness Group Target Score on Final Assessment Objective Attainment Level Based on Number of Students Achieving Target Score Exceptional Attainment (4) Full Attainment (3) Partial Attainment (2) Insufficient Attainment (1) 3≥4 5/54/53/5<3/5 2≥14-16 ≥12/13≥10/13≥8/13<8/13 1≥18-20 ≥5/64/63/6<3/6 1 st -grade Reading Using DRA2 Small Numbers of Students – Using Number of Students Reaching a Stated Target

71. Scoring Plan Preparedness Group Objective Attainment Level Based on Average Student Performance on Assessment Exceptional Attainment (4) Full Attainment (3) Partial Attainment (2) Insufficient Attainment (1) Viking* ≥ 80% ≥ 70% ≥ 65%<65% Centurion ≥ 90% ≥ 80% ≥ 75%<75% * Using names or some other classification such as numbers avoids any negative associations with the typically-used “low, medium, high” 10 th -grade Social Studies Using Locally-developed Assessment Average Target Score Within Preparedness Groups

72. Reading Level (years above or below grade) Scoring Plan Student Objective Attainment Level Based on Years of Growth on Reading Assessment Exceptional Attainment (4) Full Attainment (3) Partial Attainment (2) Insufficient Attainment (1) 2 below or more John K ≥2.0≥1.5≥1.0<1.0 -2 to 0 Jocelyn U. ≥1.5≥1.25≥1.0<1.0 Courtney P. ≥1.5≥1.25≥1.0<1.0 James L. ≥1.5≥1.25≥1.0<1.0 Emma B. ≥1.5≥1.25≥1.0<1.0 On grade or better Tim N. ≥1.25≥1.0≥0.75<0.75 Peter T. ≥1.25≥1.0≥0.7<0.75 4 th -grade ELA Using Commercial Reading Assessment Greater Rates of Growth Expected for Students Who Start Further Behind Help Close the Achievement Gap

73. Scoring Plan Preparedness Level Objective Attainment Level Based on Score on Math Assessment (%) Exceptional Attainment (4) Full Attainment (3) Partial Attainment (2) Insufficient Attainment (1) Integrity* ≥ 90 Or tests out of program ≥ 85 ≥ 80<70 Consideration ≥ 85 Or tests out of program ≥ 80 ≥ 75<70 Effort ≥ 80 Or tests out of program ≥75≥70 <70 * Using names or some other classification such as numbers avoids any negative associations with the typically-used “low, medium, high” 4 th -grade Math Support SGO Using Commercial Assessment Individual Goals Include Successful Transfer Out of Support Program

74. Scoring Plan Group Target Score Objective Attainment Level Based on Percent of Students Achieving Target Score Exceptional (4) Full (3)Partial (2) Insufficient (1) Grade level or up to one year below ≥10% ≥85%≥75%≥65%<65% More than one year below grade level ≥15% All8/10 *1) Multi-activity assessment measuring fitness, locomotor skills, throwing, catching, and dance 2) Sportsmanship rubric 9 th -grade Physics Using Locally-developed Assessment Large Numbers of Students – Using Percentage of Students Reaching a Stated Target Makes Sense

75. Resources Updated SGO guidebook and forms Expanded SGO library Assessment quality webinars Teacher practice workshops Information www.nj.gov/education/AchieveNJ Questions educatorevaluation@doe.state.nj.us 609-777-3788 74