Presentation on theme: "Teacher Value-Added Reports State Board of Education January 14, 2014"— Presentation transcript:
1 Teacher Value-Added Reports State Board of Education January 14, 2014 Connect for Success Conference 2013Teacher Value-Added Reports State Board of Education January 14, 2014Jamie MeadeManaging Director, Strategic MeasuresBattelle for Kids
2 Session ObjectivesProvide value-added information relevant to the practitioner’s perspective.What should teachers and administrators know and understand about value-added reporting?How may educators use value-added information to improve professional practice and impact student academic achievement and progress?
3 Value-Added in OhioFor over 10 years, Battelle for Kids has provided support for professional learning and fostered collegial dialogue for understanding and using value-added measures.Battelle for Kids continues to advocate for the use of value-added measures, in combination with other educational measures, to improve practice and accelerate student academic progress.
4 Ohio’s Value-Added History 2002: Battelle for Kids’ SOAR CollaborativeSchool and District Value-Added Reports2006: Battelle for Kids’ TCAP ProjectTeacher Value-Added Reports2007: ODE Value-Added on Local Report Card2009: Battelle for Kids’ Ohio VA High Schools2011: RttT: 30% Teachers, VA Reports2012: RttT: 60% Teachers, VA Reports2013: 4-8 Reading & Math Teachers, VA Reports80 RttT Districts, K-3 R & M Teachers, VA Reports
5 Value-Added in Ohio Educational Value Added Assessment System (EVAAS) SAS AnalyticsCustomers in 135 countriesMore than 65,000 business, government and university sitesSAS customers or their affiliates represent90 of the top 100 companies on the 2012FORTUNE Global 500® list.
6 Value-Added Information in Practice: Building Awareness Connect for Success Conference 2013Value-Added Information in Practice: Building AwarenessUnderstanding the DifferenceAchievement MeasuresProgress / Growth Measures
7 Achievement & Growth: Understanding the Difference Connect for Success Conference 2013Achievement & Growth: Understanding the DifferenceGrowthAchievementThe Achievement Bar (blue bar) represents our expectations for what we expect students to know, understand, and be able to do at a specific point in time. Achievement is measured at a single point in time, usually with a single assessment. Perhaps, referenced as “passing” a test.As students enter our classrooms each year, we note that students enter the room with various levels of prior knowledge, skills, and understandings. (Here, click through the animation for the blue dots to represent students at their varying entry points).Then, discuss how the school’s or the teacher’s effect/impact on learning through a single achievement assessment cannot be determined by an achievement measure at a single point in time since students enter at various levels.Measures of student growth take into consideration where a student begins, and his/her historical assessment data (as available). Thus, multiple data points are used. Growth measures a student from point A to point B.Important Note: Value Added and other growth measures are NOT measuring whether or not a student passes a test, this is a common misunderstanding about Value Added and growth measures, in general. Remember, it’s not about “passing” a test.
8 Measuring Growth is Important for ALL Students 3rd4th5th6th7th8thJacobProficientAdamValue-added provides a picture of student growth regardless of students’ achievement levels.Value-added can help us understand whether high-achieving students are making enough progress to sustain or even improve their achievement levels.Value-added can help us understand whether low-achieving students are making enough progress to close the proficiency gap.Talk about this slide from a perspective of a school with Jacob-like students or Adam-like students since value-added is a group effect versus individual student results.In this slide, Student A is currently above the proficiency bar, but is losing ground relative to proficiency.In this slide, Student B is not yet proficient, but is closing the gap on the proficiency bar.Grade
9 ES/MS Data Team Session 1 2010-2011 “Why can’t we simply compare OAA scaled scores from one year to the next to measure growth?”OAA Math Scaled Score Ranges May 2013Level3rd4th5th6th7th8thAdvancedAcceleratedProficientBasicLimitedNotice the maximum scaled score and minimum scaled score are different at each grade level. Additionally, the maximum and minimum scaled score on each grade level test will vary from year to year. This means, measuring progress through the scaled score is not possible because the scales across grade level tests are not vertically linked.Although a common practice among Ohio educators (and some parents), a student’s increase in scaled score from one school year to the next does not indicate student progress.Likewise, a decrease in the scaled score does not indicate a lack of progress. For example, a score of 596 in grade 6 and then a score of 578 in grade 7 does not mean the student did not progress.Simply looking at scaled scores to detect student growth is quite deceiving and simply inaccurate.at scaled scores to detect student growth is quite deceiving and simply inaccurate.There are ___ points up for grabs. What is the maximum PI score….
11 Value-Added Information in Practice: SAS EVAAS MRM Model Connect for Success Conference 2013Value-Added Information in Practice: SAS EVAAS MRM ModelMutlivariate Response Model (MRM)Grades 4–8 Reading and Math VA ReportsUses Ohio Achievement AssessmentsReminder: This session is not about OTES/eTPES; this session is about exploring the “anatomy” of the Teacher Value Added Report so that you will be prepared to interpret the Report.
12 SAS EVAAS Value-Added MRM Model* Connect for Success Conference 2013SAS EVAAS Value-Added MRM Model*Uses Grades 3–8 Reading and Math OAAs.Compares the average growth of students in the most recent year to the average growth of students in 2010 (state’s baseline year)*Growth expectation is defined as maintaining placement in the distribution of NCE scores from one year to the next**conceptual definition
13 Raw Score Scaled Score NCE Connect for Success Conference 2013Raw Score Scaled Score NCESampleRaw ScoreRangeScaled ScoreNormal Curve EquivalentNCE52551247991Here, animations can be used to keep the participants focused on the message as you explain the sequential process for deriving NCEs from a student’s raw score.First, a raw score represents the points received from questions answered correctly. (MC = 1 pt. SA = 2 pts. ER = 4 pts.) The raw score is then converted to a 3-digit scaled score.The scaled scores on this sample, range from the minimum scaled score of 247 to the maximum SS of These scaled scores are then converted to an NCE.On Value Added reports, NCE are illustrated on a range from 1-99; it may be important to note that NCEs can range higher than 99, and is often the case. For simplicity, the EVAAS reports will convey a maximum NCE at 99.NCE = Normal Curve Equivalent; let’s discuss that more on the next slide.
14 Scaled Scores Converted to NCEs in State’s 2010 Baseline Year DCS Value Added 11-12Scaled Scores Converted to NCEs in State’s 2010 Baseline YearConversion Values Are Fixed / FrozenExample2010 Scaled ScoresRank Ordered551247Normal Curve Equivalent (NCE)991Data Team Session 2
15 Value-Added Terminology DCS Value Added 11-12Value-Added TerminologyNormal Curve Equivalent (NCE)The NCE is similar to a percentile rank in that scores are derived from scaled scores and ranked based upon performance.A significant difference between percentile rank and NCE is that an NCE scale is an equal interval scale.Data Team Session 2
16 Normal Curve Equivalent (NCE) Distributionof ScoresPercentileEquivalentsA normal curve equivalent (NCE), indicates a student's rank compared to other students on the same test (similar to Percentile):NCEs run from 1 to 99 with 50 at the center of the base year distribution.BUT:Normal curve equivalents convert scaled scores to an equal-interval scaleSince NCEs are represented on an equal-interval scale, scores can be averaged and compared longitudinallyThe NCE scale enables longitudinal data connections and the definition of a growth standard that does not change from year to yearNCEs represent where a student’s score would place that student relative to student performance in the state’s base year 2010 for OHNormal CurveEquivalents
17 Value-Added Terminology Connect for Success Conference 2013Value-Added TerminologyBaseline ScoreGroup of students’ prior year mean NCEExample: Spring 2012 OAA mean NCEObserved ScoreGroup of students’ new/most recent mean NCEExample: Spring 2013 OAA mean NCEBefore we use the terms on the next slide, a brief explanation of these terms may be helpful.
18 SAS EVAAS MRM Model Basic, Conceptual Example Connect for Success Conference 2013SAS EVAAS MRM Model Basic, Conceptual ExampleBaselineOAA 201299th NCEObservedOAA 201399th NCEExpectedGrowthMaintain Placement in Distribution of ScoresFor example, a student at the 20th NCE must “at least” stay at the 20th NCEStudent 392 NCEStudent 389 NCE (-3)Student 267 NCE (+12)Student 255 NCEStudent 120 NCEStudent 120 NCE (0)
19 Basic, Conceptual Example Scaled scores are converted to NCEsGrade 6BaselineGrade 7Observed394 = 46 Student = 59402 = 50 Student = 54384 = 42 Student = 49394 = 46 Student = 44410 = 52 Student = 57This is an example for teaching purposes only. EVAAS calculations are more statistically sophisticated to ensure that all students are included in the analysis and that confidence intervals reflect the entire history of student testing.The EVAAS methodology also allows future data to refine past data estimates for more accuracy.NCEs are Normal Curve Equivalents. The NCE scale enables longitudinal data connections and the definition of a growth standard that does not change from year-to-year.This model links student data from one year to the next.Mean Baseline = 47.2Mean Observed = 52.6Growth = Mean Observed – Mean BaselineGrowth = 52.6 – 47.2 = 5.4 (Mean NCE Gain)A basic measure of the growth for this group is 5.4 NCEs
21 Levels of Value-Added Effects Connect for Success Conference 2013Levels of Value-Added EffectsStudents are making substantially more progress than the state growth standard.Students are making more progress than the state growth standard.Students are making about the same amount of progress as the state growth standard.Students are making less progress than the state growth standard.Students are making substantially less progress than the state growth standard.
22 DCS Value Added 11-12This slide illustrates the basic calculations of the mean gain model. For example, note the estimated district mean NCE for grade 3 in 2010 was 60.0 (Baseline Mean Score). In 2011, the same cohort of students had an estimated mean NCE of 61.2 (Observed Mean Score). The difference between the two (give / take the standard error doubled of .8) is the Value Added Mean Gain in the top half of the report.Data Team Session 2
23 Connect for Success Conference 2013 Standard ErrorA measure of the uncertaintyAll measures of student learning contain error.In the EVAAS teacher value-added report, the size of the standard error is influenced byN size (size of the student group).Missing scores.
24 Connect for Success Conference 2013 Year 1 Estimate = 1.5True value lies somewhere within the range ofthe standard error4.04321-1-2Estimate isMost likely value-1.0Multi-yr. Ave.Y1Y2Y3
25 Teacher Value-Added Report Connect for Success Conference 2013Teacher Value-Added ReportNote: Battelle for Kids is utilizing visual representations of copyrighted EVAAS® Web reporting software from SAS in this presentation for instructional purposes.Reminder: This session is not about OTES/eTPES; this session is about exploring the “anatomy” of the Teacher Value Added Report so that you will be prepared to interpret the Report.
26 Value-Added Terminology ODE Connecting the Dots ConferenceValue-Added TerminologyGrowth IndexSince the size of the standard error (degree of certainty) will vary across teachers, their estimated gain must be standardized to include both the estimate and the degree of certainty (standard error).Divides a teacher’s estimated gain by the associated standard error.Growth Index appears on the Teacher Report, but is not on the School or District Report. For this reason, this may be a new term for LEAs who have not yet received a Teacher VA report. Take the time to discuss this term before using it in the upcoming slides.
27 The Teacher Value-Added Report GarileeSnapshot of the whole report – breakdown to follow.
29 Levels of Teacher Value-Added Effects Connect for Success Conference 2013Levels of Teacher Value-Added EffectsMost EffectiveTeacher's index: 2 or greaterStudents are making substantially more progress than the state growth standard.Above AverageTeacher's index: equal to or greater than 1, but less than 2 Students are making more progress than the state growth standard.AverageTeacher's index: equal to or greater than -1, but less than 1Students are making about the same amount of progress as the state growth standard.Approaching AverageTeacher's index: equal to or greater than -2, but less than -1 Students are making less progress than the state growth standard.Least EffectiveTeacher's index: less than -2Students are making substantially less progress than the state growth standard.
30 Foundational Statistics GROhio’s Academic Content StandardsMathematics: Statistical Methods10Interpret the relationship between two variables using multiple graphical displays and statistical measures (e.g., box-and-whisker plots and measures of center and spread).8Describe how the relative size of a sample compared to the target population affects the validity of predictions.Explain the mean’s sensitivity to extremes…7…describe how the inclusion and exclusion of outliers affect those measures.6Understand the different information provided by measures of center (mean, mode, median) and measures of spread (range).1Describe the likelihood of simple events as possible/impossible and more likely/less likely.
31 Using Value-Added to Inform Practice Key Considerations:Systemic Programs, Delivery Models, Structures, Services, etc.Professional LearningCurriculum AlignmentWhat I teachAssessmentHow I measure/monitor learning along the wayInstructionHow I teach
32 Combining Measures to Inform Practice Connect for Success Conference 2013Combining Measures to Inform PracticeAll measures of student learning contain error.No single measure can capture the complexity of learning and teaching.There’s an important distinction between a flawed measure and a flawed assumption based upon a single measure.