1. Go for the Touchdown with ESC-2
PBMAS 101
What’s the Score???

2. Performance Based Monitoring Analysis System PBMAS=
Show audience an example of a district report. Use the ESC-2 report that shows Bilingual, CTE, NCLB, and Sped.

3. PBMAS Report Bilingual Career and Tech No Child Left Behind
Automated data system that reports on the performance of school districts and charter schools in the 4 Program Areas below:
Bilingual
Career and Tech
No Child Left Behind
Special Education

4. TEA’s Monitoring Systems: Historical Context
78th TX Legislative 2003 Session
Resulted in a realignment of TEA’s functions
Emphasis on a coordinated approach to TEA’s monitoring to districts

5. PBM Overall Coordination
Performance-Based Monitoring
Program Monitoring and Interventions
This is a visual of how the three areas fit together.
Program Areas

6. Historical Context Different Monitoring Components
PBMAS
Data Validation System
Federal Compliance
State Accountability System (AA, AU, etc.)
Federal Accountability System (AYP)

7. Historical Context Different Monitoring Components
Financial Integrity Rating System (FIRST)
Financial Audits
Complaints
Due Process Hearings
Governance
Other Monitoring such as Office for Civil Rights

8. A Look into One of the TEA Monitoring Components: PBMAS
This session will focus only on ONE of those monitoring systems (PBMAS).

9. PBMAS Indicators Student performance indicators
Types of indicators generally fall into these categories:
Student performance indicators
(ie TAKS Passing Rate; annual dropout rate)
Program effectiveness indicators
(ie TAKS Participation Rate)

10. 10 Guiding Principles of PBM
School District Effectiveness
Statutory Requirements
Valid Indicators of Performance
Maximum Inclusion
Individual Program Accountability
High Standards
Annual Statewide Focus
Public Input and Accessibility
Systems Evolution
Coordination
Program Effectiveness: PBM efforts are designed to assist school districts and charters in their efforts to improve student performance and program effectiveness.
Statutory Requirements: PBM efforts are designed to meet statutory requirements.
Valid Indicators of Performance: PBM indicators are designed to reflect critical areas of student performance, program effectiveness, and data integrity. (Indicators in each program area--- bilingual / ESL; CATE; Sped; NCLB)
Maximum Inclusion: PBM is designed to evaluate a maximum number of school districts and charters by using appropriate alternatives to analyze the performance of districts with small numbers of students.
Individual Program Accountability: PBM evaluations are structured to ensure that low performance in one program area cannot be masked by high performance in other program areas or lead to interventions in program areas where performance is high.
High Standards: PBM is designed to encourage high standards for all students in all districts and charters.  Standards will be adjusted over time to ensure high expectations continue to be met.
Annual Statewide Evaluation: PBM allows for the annual evaluation of a maximum number of school districts and charters in the state, and all evaluated school districts can access their PBM performance data on a yearly basis.
Public Input and Accessibility: The design, development, and implementation of PBM are all informed by ongoing public input. Performance information that PBM generates is accessible to the public.
System Evolution: PBM is a dynamic system that includes a multi-year phase-in process to allow for indicators to be added, revised, or deleted in response to changes and developments that occur outside of the system, including new legislation and the development of new assessments.
Coordination: PBM is part of an overall agency coordination strategy for the data-driven evaluation of school district and charter school effectiveness.

11. PBMAS Components Data Sources Filters Rounding Masking
No Data Available for an Indicator
Rounding
Masking
Standards & Performance Levels
Earlier, we talked about components, but they were actually components for all the monitoring systems at TEA.

12. PBMAS Components Report Only Indicators
Minimum Size Requirements and Special Analysis
Required Improvement
System Safeguards
Earlier, we talked about components, but they were actually components for all the monitoring systems at TEA.

13. Component #1 Data Sources Student Assessment Dropouts
Annual Graduation Data
Longitudinal Graduation Data
AEIS
PEIMS
These are some of the places TEA gathers information from for PBMAS. They all fit together within each program area as well.

14. Component #2 Filters May be necessary before
Performance Indicators are calculated
i.e. Residential Facilities increase
% of special education identified students in a program

15. No Data Available for an Indicator
Component #3
No Data Available for an Indicator
District can’t be evaluated for absence of data.
Data validation analysis and reviews are conducted by TEA to ensure proper coding…
i.e. coding for migrant students: Is the district identifying and coding students appropriately?

16. Calculations rounded to one decimal point
Component #4
Rounding
Calculations rounded to one decimal point

17. To protect student identity
Component #5
Masking
To protect student identity

18. Standards and Performance Levels (PL)
Component #6
Standards and Performance Levels (PL)
PL = Level System in which a standard is applied to a performance on an indicator:
0 = Met Standard 1 2
3= farthest from meeting standard
SA= Special Analysis
PJSA=Professional Judgment Sp Analysis
A district can get a 0SA or 0 PJSA as well as all the others in 1, 2, 3.
See Minimum Size Requirements for SA and PJSA.

19. Performance Levels (PL) Example: TAKS Reading Standard = 70%
Component #6
Performance Levels (PL) Example:
TAKS Reading Standard = 70%
If a district meets or exceeds the 70% in example, it would receive a “0” PL

20. Standards and Performance Levels (PL)
Component #6
Standards and Performance Levels (PL)
2 Types of Standards:
Absolute and Relative

21. Standards and Performance Levels (PL)
Component #6
Standards and Performance Levels (PL)
Absolute Standards:
0 PL = Met Standard
“1” PL = % below standard
“2” PL = % below standard
“3” PL = 20.1% or below
Tied to an absolute requirement or goal that all districts have a possibility of achieving each year. See pages 6 and 7 in manual

22. Component #6 Relative Standards:
Standards and Performance Levels (PL)
Relative Standards:
based on the distribution of scores to the population being evaluated
Not tied to an absolute requirement or goal.

23. ESC-2 Example of Relative Standard
Description
Indicator 12
2007
State
ID Rate
ESC-2 PL
Assigned
Calculation
Sped ID
8.5%
12.1% 2
12,844
105,828
You never know what this number will be at the state level, so the bar moves up or down depending on the state. A cutoff is assigned for PLs.
See page 101 for this example of Special Ed Identification #s. Also, see handout of ESC-2 data.
In this case the state rate is 8.5% this year but it could be 7.5% next year! TEA ranks all the scores and assigns cut points.

24. Example of Relative Standard
Not Assigned
PL = 0
PL = 1
PL = 2
PL = 3
8.5% or lower!
8.6% - 12%
12.1%-16%
16.1% or up
See page 102.

25. See chart on page 8 of manual
Changes to Standards
Changes are made after careful consideration. Following list includes a few:
Actual improvement over time on an indicator
Overall impact on PBMAS of increased standards
External input
Whether a state or federal goal has been identified
See chart on page 8 of manual

26. Component #7: Report Only Indicators
Gives a district a “preview” for planning purposes.
No PL assigned for 2 years

27. Component #8: Minimum Size Requirements and Special Analysis
At least 30 students are needed for calculations.
Can be met in current year or over 2 recent years.
Always in the favor of the district

28. Component #8: Minimum Size Requirements and Special Analysis
If a district meets standard regardless of amount of students, then district receives a “0” PL.
See page 10.

29. Component #8: Minimum Size Requirements and Special Analysis
Special Analysis: for districts and charters that do NOT meet the minimum size requirements.
See page 10.
Flowcharts on pages 11-13

30. Component #8 (continued)
There are two types of special analysis in PBMAS:
Automated special analysis
Professional judgment special analysis (PJSA)
See Page 10
PJSA is by hand.

31. Component #8 (continued)
GROUP SIZE DETERMINES TYPE OF ANALYSIS:
30 or more in current year or over two years: STANDARD ANALYSIS is applied.
SPECIAL ANALYSIS is applied in different ways depending on the following group sizes:
15-29
5-14
Less than 5

32. Component #8 (continued)
Special analysis does not apply to indicators for which the district has NO students in a particular student group; these districts receive a designation of No Data (ND).
Example: Neutral ISD has no CTE students. The district receives a designation of ND – No Data for the CTE TAKS Passing Rate indicator.

33. Component #8 (continued)
Special analysis (continued)
Does not apply to indicators for which the district performance level is 0 – Met Standard, regardless of student group size.
Example: Terrific ISD has four CTE students, and all four of these students met the TAKS passing standards. The district performance level on the CTE TAKS Passing Rate indicator is 0 – Met Standard.
If a district has 4/4 passing = 0.

34. Key Monitoring Concepts (continued)
Example #1 for Group Size 15-29: District’s Performance on a TAKS Reading Indicator
Current Year = 4/10 (PL2)
Prior Year = 4/10 (PL2)
PL = 2SA because of group size and PL is the same for both years.
4/10 = 40%
Reading Standard should be 50%; 10% below = PL2 (Aggregated data because of size)

35. Key Monitoring Concepts (continued)
Example #2 for Group Size 15-29: District’s Performance on a TAKS Reading Indicator
Current Year = 4/10 (PL2)
Prior Year = 7/15 (PL1)
PL = 1SA because neither is a PL3 so the better of two PLs is assigned.

36. Key Monitoring Concepts (continued)
Example #3 for Group Size 15-29: District’s Performance on a TAKS Reading Indicator (50% = 0)
Current Year = 1/10 = 10% (PL3)
Prior Year = 4/10 = 40% (PL2)
PL = determined by professional judgment special analysis.
PJSA is because the district received a “PL3”
See Flowchart on page 12. (Take the participants through the Flowchart)
May be determined by PJSA – see * on page 12
See page 31 to show “PJSA available for this indicator” under Minimum size requirements for bilingual ESL indicator #6 i- iv.

37. Key Monitoring Concepts (continued)
Example #4 for Group Size 15-29: If the same performance results were for Sped Year After Exit rather than Reading, the PL=NA (Not Assigned) because professional judgment special analysis is not currently available.
Not available in all subject area indicators. See page 84 for sped example.

38. Key Monitoring Concepts (continued)
Example #5 for Group Size 15-29: District’s Performance on a TAKS Reading Indicator
Current Year = 1/10 (PL3)
Prior Year = 8/10 (PL0)
PL = NA because PLs are more than one level apart over two years.
According to the flowchart---- look at questions to arrive at the decision.

39. Key Monitoring Concepts (continued)
Example #1 for Group Size 5-14:
District’s Performance on a TAKS Reading Indicator
Current Year = 5/11 (PL1)
Prior Year = 2/2 (PL0)
PL = NA because size of student group.
See page 15.

40. Key Monitoring Concepts (continued)
Example # 2 for Group Size 5-14: District’s Performance on a TAKS Reading Indicator
Current Year = 5/11 (PL1)
Prior Year = 0/2 (PL3)
PL = NE because PLs are more than one level apart.

41. Key Monitoring Concepts (continued)
Example #3 for Group Size 5-14: Current District’s Performance on a TAKS Reading Indicator
Current Year = 0/7 (PL3)
Prior Year = 2/7 (PL3)
PL = determined by professional judgment special analysis.

42. Key Monitoring Concepts (continued)
Example #4 for Group Size 5-14: If the same performance results were for Social Studies rather than Reading, the PL=NE because professional judgment special analysis is not currently available for TAKS Social Studies.

43. Key Monitoring Concepts (continued)
Example #1 for Group Size Less Than 5: District’s Performance on a TAKS Reading Indicator
Current Year = 0/2 (PL3)
Prior Year = 0/2 (PL3)
PL = NE because of size of student group.
Page 13 if student group size is less that 5 over two years PL= NE

44. Component #9: Required Improvement
Built-in system to help districts that demonstrate improvement from one year to the next.
List of indicators on pages 15-16

45. Component #9: Required Improvement
Example: Indicator 12 Sped Identification
Calculate actual difference from :
15% - 25% = -10%
Calculate Required Improvement for the sped identification rate: % (state) – 25%
2 ( )
Compare the two numbers to see if there is an actual change < or = to RI < = 0 PL
See page 18.
15% Is from 2008
25% is from 2007
Already, the district can see there is a 10% difference or actual change from one year to the next.
The RI is over 2 year period.
= -8.25

46. Component #10: System Safeguards
To ensure “Integrity”
Uses Data Validation Analyses:
Leaver Data
Student Assessment Data
Discipline Data
Other Means
Other means:
Accuracy of data reporting!
Implementation of intervention activities!!
Implementation of plans for improvement or correction
Accuracy of findings made through PBM process.

47. System Safeguard “Other Means”:
Accuracy of data reporting!
Implementation of intervention activities!!
Implementation of plans for improvement or correction
Accuracy of findings made through PBM process.

48. Let’s Jump to INTERVENTION LEVELS These are different than the PL #’s for each indicator within each program.

49. Intervention Levels
Levels are assigned per program area as an overall “stage” level:
Bilingual
CTE
NCLB
Special Education
Districts must work through intervention activities and upload documents to TEA on different deadlines.
Take participants to the website to view Intervention Activities.

50. Basic Intervention Activities
Focused Data Analysis
Program Effectiveness Review
LEA Public Meeting
Compliance Review
Continuous Improvement Planning

51. Accessing PBM Intervention Documents
After reviewing the district PBMAS reports / Performance Levels, click on the following link for Interventions:

52. Who Ya Gonna Call???
Bilingual: Velma Escamilla
CTE: Bob Chapman
NCLB: Andi Kuyatt
Special Ed: Barbie Tumlinson