Presentation on theme: "Putting PVAAS Teacher Reporting Skepticisms to Rest"— Presentation transcript:
1 Putting PVAAS Teacher Reporting Skepticisms to Rest Dr. John White, Manager, SAS EVAAS for K-12PVAAS Statewide Team:Kristen Lewald,Joan Perekupka and Jennifer Ross
2 Today’s Session Talk – Write – Share – Read - Ask Questions Check for UnderstandingProbeFeedback and ClarificationWhat is the Real Issue of the “Misunderstanding”?Binder of Resources – 4 SectionsLunch and Breaks
3 Key Areas Measuring Growth Demographics Transition in PA Assessment SystemGeneral: Teacher Value-AddedPA Specifics: teacher Value-AddedPA Teacher Evaluation SystemQuestions for PDE EE
4 When and How Did You LEARN PVAAS? When and How Do You USE PVAAS? Intro by IU Team
10 Measuring GrowthHow is there enough “stretch” on the PSSA to measure the growth of both low- and high-achieving students when we know that the PSSA test questions at any given grade level are limited to the “eligible content” for that grade? In other words, how can growth beyond (or below) the current grade level be shown when the PSSA is limited to questions from the eligible content for that grade level only.
12 Measuring Growth1. Many educators believe that high performing students have a harder time making growth, so is this model setting us up for failure? How is it possible to show growth with high-achieving students?½ table question #1PartnerUnderlying misunderstanding?How would you respond to this issue?
13 Measuring Growth2. How do you explain what happens when one grade level shows no growth in a given year and then the next year shows tons of growth? There are educators who say it is easy to show growth in some years/grades because the year/grade before the students “lost ground.” Can this be addressed?½ table question #2PartnerUnderlying misunderstanding?How would you respond to this issue?
15 Learning about High Achievement and Growth PVAAS Misconceptions DocumentFirst developed in 2011 and revised in 2012Addressed in misconceptions #8 and #9New Concept of Growth graphics/slidesNew slides address concept plus application for both low-achieving and high-achieving groups of studentsAppear in F2F Overview materials, webinar materials, and fall session materialsOne-pager on High Achievement & High GrowthRevised January 2013Webinar – High Growth, High Achieving Schools: Is it Possible?Live webinars held twice in December 2012Archived versions still available via Elluminate
16 Learning about High Achievement and Growth Virtual Learning Module – High Achievement & GrowthReleased October 2012Available on public and restricted sitesA shorter, more user-friendly version will be released by end of JuneStatewide Scatterplots and District/School Search FeatureAvailable on public siteShows evidence of schools with many high-achieving students making high growthThe Building Capacity for IUs webinar in March (#4 in the series) specifically addressed how to use these features to answer the question, “Can high-achieving schools, or groups of students, show growth?”
17 Measuring GrowthIt is said that value-added modeling is inaccurate; that no one can truly explain how growth is determined; and that psychometricians do not believe value-added measures are valid. Can this be addressed?Dr. John WhiteReference Section 3 in BinderLiterature Review for Teacher Value-Added
18 Measuring GrowthHow can you calculate value-added estimates for grade 4 science? If grade 3 is the first year we test reading and math and we don’t have value-added estimates in grade 3, then why can we get value-added estimates for grade 4 science when that is the first year that subject is assessed?PSSA Science as a GRADE level testFeedback from Dr. John White
19 Measuring GrowthHow do we tease out the factors that contribute to growth? When districts are analyzing their value-added reports what should they look at to determine what they did in order to meet or not meet the standard for PA Academic Growth? How do we help districts that do not have or use a written curriculum? How do we help them interpret or make informed decisions based on the value-added data?1 minute thinkExamples from 3 of You
21 Article “Adjusting for Student Characteristics in Value-Added Models” Section 1 in BinderReadWrite Down 3 Key Points
22 DemographicsWhat evidence exists about growth and demographics at the teacher level?Dr. John WhiteEvidence over time
23 DemographicsIf achievement is influenced by demographics, and growth is based on achievement scores, then why isn’t growth influenced by demographics?How is the demographic makeup of classes (groups of students) taken into account, or not?
24 DemographicsHow are groups of students who learn differently taken into account, especially if there are high numbers of economically disadvantaged students or students with IEPs in the group?Dr. John WhiteUsing All Prior Achievement Data
25 DemographicsWhat research has been done on the growth of students with emotional disabilities? The research lumps all special education students together.PA’s Plan
27 Transitions in PA Assessments Section 4 of Binder: PVAAS and the Transition of PA’s Assessment System3 Minute SkimOn PVAAS Login page
28 Transitions in PA Assessments Will teacher-specific value-added estimates be reliable for Keystone exams being that we are just now administering these for the first time?
29 Transitions in PA Assessments How will teacher-specific value-added estimates be reliable if the base year is changing each year? Can you explain the implications of resetting the base year each year for the next few years? Will this lead to larger standard errors? Are there ramifications for this in the 3-year rolling average for teacher reporting?Will the baseline for Keystone exams be changing each year?
30 Transitions in PA Assessments The grade 7 and grade 8 Math PSSAs and the Keystone Algebra I exam require two very different types of thinking (i.e., PSSAs seem to target more facts and processes, while the Keystone exams target higher-order thinking). Will the grade 7 and grade 8 PSSAs accurately predict Keystone exam performance?
31 Transitions in PA Assessments What will be the overall impact to PVAAS reporting now that the PSSA-M is not administered?
32 Transitions in PA Assessments If the projections to Keystone exams are for when students take the exam, how can we use the data to determine early readiness for the course (i.e., a high projection does not necessarily equate to early readiness)?
34 General: Teacher Value-Added How do we reflect the fact that time after time we were told PVAAS would not drill down to the teacher and student level, and now it is going to that level?Act 82PVAAS Prior to Act 82No student-teacher linkagesCould NOT get teacher specific reporting
35 General: Teacher Value-Added What happens when a teacher has a year of “that class”? Teachers always have a year when that happens, and they don’t want to be judged by a single year.PVAAS 3 Year Rolling Average in EE SystemIs there a hidden need for professional development?Dr. John WhiteFindings from research on teacher effectiveness measures? Consistency of measures? Effect as teachers change types of class/school/district?
36 General: Teacher Value-Added There are so many people involved in an individual student’s education. How is the teacher value-added measure for a single teacher credible for evaluation purposes?
37 General: Teacher Value-Added There are concerns that administrators could unfairly influence teacher-specific reporting by assignment of students to specific classes. Can this be addressed?Turn to a partnerWhat is the underlying misunderstanding?
38 General: Teacher Value-Added How do administrators and districts combat the issue of teachers resisting teaching assignments for groups of students that are low performing as they won’t want to claim those students when it comes to value-added reporting?What is the underlying misunderstanding?
39 General: Teacher Value-Added Teachers are now skeptical about taking on student teachers. We recognize there is research indicating there is no impact of that on value-added measures. Can this be addressed?Feedback from Dr. White
40 General: Teacher Value-Added Our district and schools have not met the growth standard for over 3 years in grade 5. Does this mean that our grade 5 teachers will not meet the growth standard either?District-School-Teacher MeasuresFeedback from Dr. White
41 General: Teacher Value-Added How is class size taken into account with value-added measures? How are teachers who have small numbers of students affected?Feedback from Dr. John White
42 General: Teacher Value-Added In a district where students stay and then go to the DR and come back a month later (and they have attendance issues), then they do poorly on the test, how is it fair to solely contribute that to the teacher?
43 General: Teacher Value-Added How fair is it to measure growth with three different groups of students when using it for teacher-specific reporting?Stability of EVAAS MeasuresDr. John White
44 General: Teacher Value-Added How do the teacher value-added measures take into account a teacher who is co-teaching, and the co-teacher is harmful to the academic achievement of the students? The teacher may only have 50% instructional responsibility for those students, but the teacher’s value-added measure is still being affected by the co-teacher really “pulling these students down.”
45 General: Teacher Value-Added How do we tease out the teacher factors that contribute to growth? For example, in a grade 8 Algebra I course, it may be more rigorous and have higher achieving students than the grade 9 Algebra I course in the same district. However, if the same teacher is teaching both courses with two different sets of curriculum and the value-added results are all lumped together, how do we help teachers sort that information out?PVAAS Custom Diagnostic Report
47 PA Specifics: Teacher Value-Added Data Quality ConcernsFrom Collecting to UsingAddressing Data QualityLEA SISPIMSPVAAS Roster VerificationPAIMS Tech ReadinessPAIU Data Quality Event Last WeekPlanning any follow-up meetings with LEAs?Involve Program and Data Staff
48 Impact in PIMS and Local SIS Staff Data such as:NEW: Staff Work sNEW: PVAAS Role for PVAAS Account CreationPIMS Course/HQT (Student-Staff-Courses)NEW: All teachers linked to a student for a grade/subject/courseTeachers beyond/different than HQT (HQT is a different PIMS need)WHICH teachers REALLY had responsibility for planning the instruction, delivering the instruction, and assessing the effectiveness of the instruction (Domains 1 AND 3)Could include LTSNEW: Keystone Related Course Codes/Course(s) that “trigger” a Keystone examNEW: % Instructional Responsibility
49 PA Specifics: Teacher Value-Added Data Quality Districts are concerned about the integrity of the data being used in calculation of teacher value-added estimates. Can this be addressed?There are concerns about the accuracy of student roster assignments to teachers. Can this be addressed?How is the attribution for the percentage of time that a student is enrolled in a class accurately reported and reflected?
50 PA Specifics: Teacher Value-Added Data Quality There are concerns with how to implement the roster verification system at a large scale once this becomes operational next year (i.e., the amount of paperwork and cross-checking that is required to attribute all students to a class is a lot). Can you address how districts are to accomplish this?There are concerns that enrollment data will not be accurately reflected for teachers. Can this be addressed?
51 PA Specifics: Teacher Value-Added Data Quality What data are we getting back and does that have to sync with PIMS after the teachers and administrators change the data?Under discussion by PDE PIMS team
52 PA Specifics: Teacher Value-Added Data Quality There are concerns of the logistics of enrolling students and having teachers initially determine their percent of instructional responsibility. Can this be addressed?LEA SISPIMSPre-population of PVAAS
53 PA Specifics: Teacher Value-Added Data Quality There are concerns regarding the how the information can be systemized so that the teachers and principals have as much data as possible pre-loaded into the roster verification system instead of having all rosters show 100%. Can the shared responsibilities somehow be represented in the pre-load of the verification tool?LEA SISPIMSPre-population of PVAAS
54 PA Specifics: Teacher Value-Added Data Quality Given the numerous time, behavioral, management, and instructional demands placed on principals, teachers, and central office administrators, what is the anticipated time demand needed to affirm or correct the data input into the system?PILOT SURVEY
55 PA Specifics: Teacher Value-Added Role/Responsibility of Admin What happens to teacher value-added scores if teachers are reassigned after two years? Will this cause pressure to administrators to reassign teachers to a different assignment after two years so that teachers will never get the 15% for use in the evaluation system?
56 PA Specifics: Teacher Value-Added Role/Responsibility of Admin There are concerns about potential union issues regarding teachers who have improvement plans. Could there be potential union concerns regarding how attribution of instructional responsibility of teachers is decided?There are concerns that principals might not have the time or know the teachers and their relationships to the students well enough to provide accurate validation of the data. Can this be addressed?
57 PA Specifics: Teacher Value-Added Role/Responsibility of Admin There are concerns that teachers might skew the percentages they enter for instructional responsibility in order to show the best results. Can this be addressed?What would you think this might look like??Skewed how?Underlying misunderstanding?
58 PA Specifics: Teacher Value-Added Role/Responsibility of Admin There are concerns that administrators do not have a clear understanding of how to interpret PVAAS data, and therefore will not be able to correctly explain the methodology to teachers. Can this be addressed?
59 PA Specifics: Teacher Value-Added % Instructional Responsibility How do you attribute instructional responsibility for teachers involved in flexible grouping of RtII students?There are concerns about teacher calculation and attribution for long-term substitutes. Can this be addressed?
60 PA Specifics: Teacher Value-Added % Instructional Responsibility How do districts handle special education teachers if the students are itinerant?There are concerns about the percentage of instruction for students with IEPs who are on a teacher’s roster and are to receive support services from another teacher, but yet they do not. Can this be addressed?
61 PA Specifics: Teacher Value-Added LEA Implementation There is concern that the PVAAS roster verification process will drive service delivery models rather than having the PVAAS roster verification process reflect service delivery models. Can this be addressed?
62 PA Specifics: Teacher Value-Added LEA Implementation There are concerns about attendance issues, and districts not having tight enough attendance and enrollment policies and procedures in place. Can this be addressed?
63 PA Specifics: Teacher Value-Added Are there policies or guidelines regarding the breach of confidentiality and sharing of data for both student and teacher?FERPAProfessional Conduct
64 PA Specifics: Teacher Value-Added PVAAS Pilot FAQs Section 4 in BinderPA Specifics: Teacher Value-Added PVAAS Pilot FAQs
65 PILOT in Process Now: 273 LEAs All PILOT materials are posted on PVAAS home pagehttps://pvaas.sas.comIU PVAAS Contact for List of Pilot LEAs in Your region
66 PA Specifics: Teacher Value-Added PVAAS Pilot FAQs It is stated in the PVAAS Roster Verification web-based system that EVAAS will make adjustments if a student is over attributed in the final submission. This is a high priority concern. Can this be addressed?How will time, and therefore the percentage of student growth, be attributed to teachers who take a leave of absence during the school year?
67 PA Specifics: Teacher Value-Added PVAAS Pilot FAQs How do you attribute instructional responsibility to homebound students?Why is enrollment data being used versus attendance data?Do teacher value-added scores follow teachers if they leave the district? For example, if a teacher was with a district for two years, do those PVAAS growth scores follow them if they enter another district?
69 PA Teacher Evaluation System 1. There are so many teachers who are not eligible for teacher value-added measures (i.e., they are in non-tested subjects and grades), so how is the evaluation system fair? Not all teachers will be evaluated in the same way.2. These systems discriminate against teachers who teach tested subjects and grades where value-added measures can be calculated. Other teachers are then “off the hook.” Can this be addressed?
70 PA Teacher Evaluation System 3. There are concerns about inequity or challenges between teachers. Can this be addressed? 4. There is concern that the assessments don’t measure total student learning (i.e., transfer, generalization, creativity, emotional intelligence), but yet teachers will be evaluated on these tests. Can this be addressed?
71 PA Teacher Evaluation System 5. How can we articulate the weighting and importance of the formula used and its impact on student achievement?6. How will the teacher value-added reporting affect teachers as part of the teacher effectiveness rating system?
72 PA Teacher Evaluation System 7. There are concerns that the public will be able to identify specific teachers who are effective and ineffective. Can this be addressed?8. How will the teacher-specific value-added data component be implemented into the entire teacher effectiveness model for PA? In other words, it is 15% of the model, but 15% of what?
73 PA Teacher Evaluation System 9. Can you provide an explanation of teacher-specific reporting for the School Performance Profile (3-year average versus year-to-year reporting of the AGI)?10. Can you explain the School Performance Profile and its use, especially given the weight of PVAAS and its use in building-level data used in the teacher evaluation system?
75 Questions for PDE EEWill there be videos, webinars, and FAQs available for teachers specifically on the new teacher evaluation system?What will be the turn-around time for training on the reporting tool?Ask your regional IU EE Contact
76 When and How Will You Continue to LEARN PVAAS When and How Will You Continue to LEARN PVAAS? When and How Will You Continue to USE PVAAS?
77 Other Questions? DRAFT PVAAS Team Plans PVAAS Pilot Training and SupportPVAAS Teacher Specific Reporting (Late Fall/Winter)SY13-14 ImplementationPreparing for Roster Verification (Early Fall)PVAAS Teacher Specific Reporting- What to ExpectRoster Verification Training (Winter/Spring)
78 Other Questions? Next Steps? Needs? Ideas for Support/Follow-up with Your Group?Building Capacity Sessions