Presentation on theme: "Putting PVAAS Teacher Reporting Skepticisms to Rest Dr. John White, Manager, SAS EVAAS for K-12 PVAAS Statewide Team: Kristen Lewald,Joan Perekupka and."— Presentation transcript:
Putting PVAAS Teacher Reporting Skepticisms to Rest Dr. John White, Manager, SAS EVAAS for K-12 PVAAS Statewide Team: Kristen Lewald,Joan Perekupka and Jennifer Ross
Todays Session Talk – Write – Share – Read - Ask Questions Check for Understanding Probe Feedback and Clarification What is the Real Issue of the Misunderstanding? Binder of Resources – 4 Sections Lunch and Breaks
Key Areas Measuring Growth Demographics Transition in PA Assessment System General: Teacher Value-Added PA Specifics: teacher Value-Added PA Teacher Evaluation System Questions for PDE EE
When and How Did You LEARN PVAAS? When and How Do You USE PVAAS? Intro by IU Team
Skepticisms -Misunderstandings Need to Respond
Measuring Growth How do you describe the concept of growth as measured by PVAAS?
How does PVAAS Measure Growth? Paper Pencil 3 minutes
How does PVAAS Measure Growth? Share with a Partner 2 minutes EACH Large Group Report Out
Learning about Measuring Growth
Measuring Growth How is there enough stretch on the PSSA to measure the growth of both low- and high- achieving students when we know that the PSSA test questions at any given grade level are limited to the eligible content for that grade? In other words, how can growth beyond (or below) the current grade level be shown when the PSSA is limited to questions from the eligible content for that grade level only.
1. Many educators believe that high performing students have a harder time making growth, so is this model setting us up for failure? How is it possible to show growth with high-achieving students? – ½ table question #1 – Partner Underlying misunderstanding? How would you respond to this issue?
Measuring Growth 2. How do you explain what happens when one grade level shows no growth in a given year and then the next year shows tons of growth? There are educators who say it is easy to show growth in some years/grades because the year/grade before the students lost ground. Can this be addressed? – ½ table question #2 – Partner Underlying misunderstanding? How would you respond to this issue?
Learning about High Achievement and Growth
PVAAS Misconceptions Document First developed in 2011 and revised in 2012 Addressed in misconceptions #8 and #9 New Concept of Growth graphics/slides New slides address concept plus application for both low- achieving and high-achieving groups of students Appear in F2F Overview materials, webinar materials, and fall session materials One-pager on High Achievement & High Growth Revised January 2013 Webinar – High Growth, High Achieving Schools: Is it Possible? Live webinars held twice in December 2012 Archived versions still available via Elluminate
Learning about High Achievement and Growth Virtual Learning Module – High Achievement & Growth Released October 2012 Available on public and restricted sites A shorter, more user-friendly version will be released by end of June Statewide Scatterplots and District/School Search Feature Available on public site Shows evidence of schools with many high-achieving students making high growth The Building Capacity for IUs webinar in March (#4 in the series) specifically addressed how to use these features to answer the question, Can high-achieving schools, or groups of students, show growth?
Measuring Growth It is said that value-added modeling is inaccurate; that no one can truly explain how growth is determined; and that psychometricians do not believe value-added measures are valid. Can this be addressed? – Dr. John White – Reference Section 3 in Binder Literature Review for Teacher Value-Added
Measuring Growth How can you calculate value-added estimates for grade 4 science? If grade 3 is the first year we test reading and math and we dont have value-added estimates in grade 3, then why can we get value-added estimates for grade 4 science when that is the first year that subject is assessed? – PSSA Science as a GRADE level test – Feedback from Dr. John White
Measuring Growth How do we tease out the factors that contribute to growth? When districts are analyzing their value-added reports what should they look at to determine what they did in order to meet or not meet the standard for PA Academic Growth? How do we help districts that do not have or use a written curriculum? How do we help them interpret or make informed decisions based on the value-added data? – 1 minute think – Examples from 3 of You
Article Adjusting for Student Characteristics in Value- Added Models Section 1 in Binder Read Write Down 3 Key Points
Demographics What evidence exists about growth and demographics at the teacher level? – Dr. John White – Evidence over time
Demographics If achievement is influenced by demographics, and growth is based on achievement scores, then why isnt growth influenced by demographics? How is the demographic makeup of classes (groups of students) taken into account, or not?
Demographics How are groups of students who learn differently taken into account, especially if there are high numbers of economically disadvantaged students or students with IEPs in the group? – Dr. John White – Using All Prior Achievement Data
Demographics What research has been done on the growth of students with emotional disabilities? The research lumps all special education students together. – PAs Plan
TRANSITIONS IN PA ASSESSMENT SYSTEM
Transitions in PA Assessments Section 4 of Binder: PVAAS and the Transition of PAs Assessment System 3 Minute Skim On PVAAS Login page
Transitions in PA Assessments Will teacher-specific value-added estimates be reliable for Keystone exams being that we are just now administering these for the first time?
Transitions in PA Assessments How will teacher-specific value-added estimates be reliable if the base year is changing each year? Can you explain the implications of resetting the base year each year for the next few years? Will this lead to larger standard errors? Are there ramifications for this in the 3-year rolling average for teacher reporting? Will the baseline for Keystone exams be changing each year?
Transitions in PA Assessments The grade 7 and grade 8 Math PSSAs and the Keystone Algebra I exam require two very different types of thinking (i.e., PSSAs seem to target more facts and processes, while the Keystone exams target higher-order thinking). Will the grade 7 and grade 8 PSSAs accurately predict Keystone exam performance?
Transitions in PA Assessments What will be the overall impact to PVAAS reporting now that the PSSA-M is not administered?
Transitions in PA Assessments If the projections to Keystone exams are for when students take the exam, how can we use the data to determine early readiness for the course (i.e., a high projection does not necessarily equate to early readiness)?
GENERAL: TEACHER VALUE-ADDED
General: Teacher Value-Added How do we reflect the fact that time after time we were told PVAAS would not drill down to the teacher and student level, and now it is going to that level? Act 82 PVAAS Prior to Act 82 – No student-teacher linkages – Could NOT get teacher specific reporting
General: Teacher Value-Added What happens when a teacher has a year of that class? Teachers always have a year when that happens, and they dont want to be judged by a single year. – PVAAS 3 Year Rolling Average in EE System – Is there a hidden need for professional development? – Dr. John White Findings from research on teacher effectiveness measures? Consistency of measures? Effect as teachers change types of class/school/district?
General: Teacher Value-Added There are so many people involved in an individual students education. How is the teacher value-added measure for a single teacher credible for evaluation purposes?
General: Teacher Value-Added There are concerns that administrators could unfairly influence teacher-specific reporting by assignment of students to specific classes. Can this be addressed? – Turn to a partner – What is the underlying misunderstanding?
General: Teacher Value-Added How do administrators and districts combat the issue of teachers resisting teaching assignments for groups of students that are low performing as they wont want to claim those students when it comes to value-added reporting? What is the underlying misunderstanding?
General: Teacher Value-Added Teachers are now skeptical about taking on student teachers. We recognize there is research indicating there is no impact of that on value-added measures. Can this be addressed? Feedback from Dr. White
General: Teacher Value-Added Our district and schools have not met the growth standard for over 3 years in grade 5. Does this mean that our grade 5 teachers will not meet the growth standard either? District-School-Teacher Measures Feedback from Dr. White
General: Teacher Value-Added How is class size taken into account with value-added measures? How are teachers who have small numbers of students affected? – Feedback from Dr. John White
General: Teacher Value-Added In a district where students stay and then go to the DR and come back a month later (and they have attendance issues), then they do poorly on the test, how is it fair to solely contribute that to the teacher?
General: Teacher Value-Added How fair is it to measure growth with three different groups of students when using it for teacher-specific reporting? – Stability of EVAAS Measures – Dr. John White
General: Teacher Value-Added How do the teacher value-added measures take into account a teacher who is co- teaching, and the co-teacher is harmful to the academic achievement of the students? The teacher may only have 50% instructional responsibility for those students, but the teachers value-added measure is still being affected by the co-teacher really pulling these students down.
General: Teacher Value-Added How do we tease out the teacher factors that contribute to growth? For example, in a grade 8 Algebra I course, it may be more rigorous and have higher achieving students than the grade 9 Algebra I course in the same district. However, if the same teacher is teaching both courses with two different sets of curriculum and the value- added results are all lumped together, how do we help teachers sort that information out? – PVAAS Custom Diagnostic Report
PA SPECIFICS: TEACHER VALUE- ADDED
PA Specifics: Teacher Value-Added Data Quality Concerns – From Collecting to Using – Addressing Data Quality LEA SIS PIMS PVAAS Roster Verification – PAIMS Tech Readiness – PAIU Data Quality Event Last Week Planning any follow-up meetings with LEAs? Involve Program and Data Staff
Impact in PIMS and Local SIS Staff Data such as: – NEW: Staff Work s – NEW: PVAAS Role for PVAAS Account Creation PIMS Course/HQT (Student-Staff-Courses) – NEW: All teachers linked to a student for a grade/subject/course Teachers beyond/different than HQT (HQT is a different PIMS need) WHICH teachers REALLY had responsibility for planning the instruction, delivering the instruction, and assessing the effectiveness of the instruction (Domains 1 AND 3) Could include LTS – NEW: Keystone Related Course Codes/Course(s) that trigger a Keystone exam – NEW: % Instructional Responsibility 48
PA Specifics: Teacher Value-Added Data Quality Districts are concerned about the integrity of the data being used in calculation of teacher value- added estimates. Can this be addressed? There are concerns about the accuracy of student roster assignments to teachers. Can this be addressed? How is the attribution for the percentage of time that a student is enrolled in a class accurately reported and reflected?
PA Specifics: Teacher Value-Added Data Quality There are concerns with how to implement the roster verification system at a large scale once this becomes operational next year (i.e., the amount of paperwork and cross-checking that is required to attribute all students to a class is a lot). Can you address how districts are to accomplish this? There are concerns that enrollment data will not be accurately reflected for teachers. Can this be addressed?
PA Specifics: Teacher Value-Added Data Quality What data are we getting back and does that have to sync with PIMS after the teachers and administrators change the data? – Under discussion by PDE PIMS team
PA Specifics: Teacher Value-Added Data Quality There are concerns of the logistics of enrolling students and having teachers initially determine their percent of instructional responsibility. Can this be addressed? – LEA SIS – PIMS – Pre-population of PVAAS
PA Specifics: Teacher Value-Added Data Quality There are concerns regarding the how the information can be systemized so that the teachers and principals have as much data as possible pre-loaded into the roster verification system instead of having all rosters show 100%. Can the shared responsibilities somehow be represented in the pre-load of the verification tool? – LEA SIS – PIMS – Pre-population of PVAAS
PA Specifics: Teacher Value-Added Data Quality Given the numerous time, behavioral, management, and instructional demands placed on principals, teachers, and central office administrators, what is the anticipated time demand needed to affirm or correct the data input into the system? PILOT SURVEY
PA Specifics: Teacher Value-Added Role/Responsibility of Admin What happens to teacher value-added scores if teachers are reassigned after two years? Will this cause pressure to administrators to reassign teachers to a different assignment after two years so that teachers will never get the 15% for use in the evaluation system?
PA Specifics: Teacher Value-Added Role/Responsibility of Admin There are concerns about potential union issues regarding teachers who have improvement plans. Could there be potential union concerns regarding how attribution of instructional responsibility of teachers is decided? There are concerns that principals might not have the time or know the teachers and their relationships to the students well enough to provide accurate validation of the data. Can this be addressed?
PA Specifics: Teacher Value-Added Role/Responsibility of Admin There are concerns that teachers might skew the percentages they enter for instructional responsibility in order to show the best results. Can this be addressed? – What would you think this might look like?? – Skewed how? – Underlying misunderstanding?
PA Specifics: Teacher Value-Added Role/Responsibility of Admin There are concerns that administrators do not have a clear understanding of how to interpret PVAAS data, and therefore will not be able to correctly explain the methodology to teachers. Can this be addressed?
PA Specifics: Teacher Value-Added % Instructional Responsibility How do you attribute instructional responsibility for teachers involved in flexible grouping of RtII students? There are concerns about teacher calculation and attribution for long-term substitutes. Can this be addressed?
PA Specifics: Teacher Value-Added % Instructional Responsibility How do districts handle special education teachers if the students are itinerant? There are concerns about the percentage of instruction for students with IEPs who are on a teachers roster and are to receive support services from another teacher, but yet they do not. Can this be addressed?
PA Specifics: Teacher Value-Added LEA Implementation There is concern that the PVAAS roster verification process will drive service delivery models rather than having the PVAAS roster verification process reflect service delivery models. Can this be addressed?
PA Specifics: Teacher Value-Added LEA Implementation There are concerns about attendance issues, and districts not having tight enough attendance and enrollment policies and procedures in place. Can this be addressed?
PA Specifics: Teacher Value-Added Are there policies or guidelines regarding the breach of confidentiality and sharing of data for both student and teacher? – FERPA – Professional Conduct
PA SPECIFICS: TEACHER VALUE-ADDED PVAAS PILOT FAQS Section 4 in Binder
PILOT in Process Now: 273 LEAs All PILOT materials are posted on PVAAS home page https://pvaas.sas.com IU PVAAS Contact for List of Pilot LEAs in Your region 65
PA Specifics: Teacher Value-Added PVAAS Pilot FAQs It is stated in the PVAAS Roster Verification web-based system that EVAAS will make adjustments if a student is over attributed in the final submission. This is a high priority concern. Can this be addressed? How will time, and therefore the percentage of student growth, be attributed to teachers who take a leave of absence during the school year?
PA Specifics: Teacher Value-Added PVAAS Pilot FAQs How do you attribute instructional responsibility to homebound students? Why is enrollment data being used versus attendance data? Do teacher value-added scores follow teachers if they leave the district? For example, if a teacher was with a district for two years, do those PVAAS growth scores follow them if they enter another district?
PA SPECIFICS: TEACHER EVALUATION SYSTEM
PA Teacher Evaluation System 1. There are so many teachers who are not eligible for teacher value-added measures (i.e., they are in non-tested subjects and grades), so how is the evaluation system fair? Not all teachers will be evaluated in the same way. 2. These systems discriminate against teachers who teach tested subjects and grades where value- added measures can be calculated. Other teachers are then off the hook. Can this be addressed?
PA Teacher Evaluation System 3. There are concerns about inequity or challenges between teachers. Can this be addressed? 4. There is concern that the assessments dont measure total student learning (i.e., transfer, generalization, creativity, emotional intelligence), but yet teachers will be evaluated on these tests. Can this be addressed?
PA Teacher Evaluation System 5. How can we articulate the weighting and importance of the formula used and its impact on student achievement? 6. How will the teacher value-added reporting affect teachers as part of the teacher effectiveness rating system?
PA Teacher Evaluation System 7. There are concerns that the public will be able to identify specific teachers who are effective and ineffective. Can this be addressed? 8. How will the teacher-specific value-added data component be implemented into the entire teacher effectiveness model for PA? In other words, it is 15% of the model, but 15% of what?
PA Teacher Evaluation System 9. Can you provide an explanation of teacher- specific reporting for the School Performance Profile (3-year average versus year-to-year reporting of the AGI)? 10. Can you explain the School Performance Profile and its use, especially given the weight of PVAAS and its use in building-level data used in the teacher evaluation system?
QUESTIONS FOR PDE EE
Questions for PDE EE Will there be videos, webinars, and FAQs available for teachers specifically on the new teacher evaluation system? What will be the turn-around time for training on the reporting tool? Ask your regional IU EE Contact
When and How Will You Continue to LEARN PVAAS? When and How Will You Continue to USE PVAAS?
Other Questions? DRAFT PVAAS Team Plans – PVAAS Pilot Training and Support PVAAS Teacher Specific Reporting (Late Fall/Winter) – SY13-14 Implementation Preparing for Roster Verification (Early Fall) PVAAS Teacher Specific Reporting- What to Expect Roster Verification Training (Winter/Spring)
Other Questions? Next Steps? Needs? Ideas for Support/Follow-up with Your Group? Building Capacity Sessions