Combining Multiple Measures What are the indicators/ components? What are the priority outcomes? What are the performance expectations? How can we evaluate effectiveness? 1

Components Indicators are specific measures in the system such as performance on assessments or graduation rate. Components refer to broader (non-mutually exclusive) categories and can include: – Achievement (Status) – Growth – Equity – Readiness – Inclusion 2

What to Report Classifications – How many levels? How will you define these levels? ‘Monikers’ rarely have intuitive meaning. Consider starting with policy descriptors for each classification – Scores Report outcomes at indicator, component, or overall level? 3

How to combine? There are many approaches to combining within and across components. The ‘how’ is much easier than the ‘why’ – What should outcomes look like and how does this fit into the overall theory of action? Strike balance between simple – flexible Methods for combining multiple parts into a whole that yields a single outcome include: – Conjunctive – Disjunctive – Compensatory – Profile 4

Conjunctive A conjunctive rule means that ALL parts have to meet a standard in order to achieve a target outcome. Conjunctive rules are used when each part is distinct and necessary for the outcome. Example: – Currently, NCLB is a conjunctive model in that each subgroup must meet target performance in each area for a school to make AYP. 5

Disjunctive A disjunctive rule means that ANY part can meet an established standard in order to achieve an overall target outcome. A disjunctive rule should be used when each component represents a similar or equally valued part of the whole. In other words, there are multiple ways to show quality and any one way is as good as another. Example: – If school meets the target in status OR growth the overall standard is achieved. 6

Combine Conjunctive/ Disjunctive Blend elements of each into overall decision For example: – To be classified as meeting standards… 1. schools should meet EITHER growth or proficiency standards (disjunctive) 2. these standards have to be met for the whole school AND all subgroups (conjunctive) 7

Compensatory A compensatory approach means that each component contributes to the outcome, but being higher on one component can offset or ‘compensate’ for being lower on another. Example: – Take the average of each component and base the outcome on the resulting value. – Produce an index, which can be weighted to reflect priorities (e.g. 70% growth, 30% status) – Can set a ‘floor’ or conditions to prevent unintended outcomes (e.g. average must equal X unless Y is less than…”) 8

Profile Yet another way to combine indicators/ components is to create profile descriptions. A profile refers to a pattern of performance determined to meet the target or not. This approach is particularly useful when combinations of points are ‘conditional’. For example, a combination of 1,1,2 = 1; while a combination of 1,2,1 = 2. In an index, these combinations might yield the same points, which may be undesirable, because HOW the points are earned matters. 9

Equity and Excellence Who are the students for whom equity concerns are greatest? – Demographic factors Multiple subgroups ‘Super-subgroup’ – Performance factors Normative (e.g. all students in lowest 25%) Criterion (e.g. all students below proficient) – What outcomes should be prioritized Attainment of status Growth rate Principles: – Schools that are already successful – those that have no or very small achievement gaps – should not be disadvantaged. – Schools that have the largest achievement gaps should have the most incentive to improve. Differentiated expectations. – Schools should not be rewarded for closing the achievement gap when it is accomplished by lowering the score of the higher performing group. 10

11 Growth and School Type Low Growth High Growth Low Status High Status Low/LowLow/High High/High Status/Growth Combinations High/Low A thoughtful approach should consider how standards interact with status.

Build-In Incentives Consider ways to incentivize actions to support lowest performing students For example, what will be the motivation to work with students who don’t graduate in four years? Consider ‘incentive’ points 12 Possible OutcomesPoints - A Additional OutcomesPoints - B Time to OutcomePoints - C 1 Student earns CCR diploma with honors125 Student earns (or is likely to earn) college credit via AP/IB/dual enrollment 10 per course 4th yearNA Student earns CCR diploma100 Student earns non-CCR (e.g. ‘general’) diploma75 5th year (sum of A and B) multiplied by.75 Student earns GED50 Student earns industry certification 10 per certificate Student earns other certificate50 Student persists, does not meet graduation requirements years (sum of A and B) multiplied by.50 Student drops out0 1 Late graduates are not averaged into the index score for the current cohort. However, bonus points are added to the current cohort score after averaging.

Performance Expectations Is it important to establish criteria (whether reported or not) within each component, or should criteria be based only on overall results? Are there external indicators that should factor into decisions? Are there expectations for the distribution of results? How should expectations differ for schools/ subgroups? 13

Normative Normative growth provides an indication of what is attainable Examine patterns of performance for schools and subgroups to set initial expectations for what is possible and what is reasonable – For example: What is rate of growth for high achieving students compared to low achieving students? 14

Criterion For students that grow at specified rates, what is the probability of attaining or maintaining target status? For example: – What percent of non-proficient students meeting a specified growth target reach proficiency in 1 year, 2 years etc. 15

Both are important and complimentary Ideally, the are used iteratively – Examine patterns of performance for schools and subgroups to set initial expectations for what is possible and what is reasonable – For students that grow at specified rates, what is the probability of attaining or maintaining target status? 16 Growth Expectations Example depiction of norm and criterion referenced growth based on SGP. Betebenner, D. W.

To what degree are outcomes stable for schools/ groups of various types and size? (reliability) To what extent are the results associated with variables not related to effectiveness? (e.g. percent ED) What evidence bolsters the claim that classifications are credible? (e.g. related to other valued quantitative and qualitative indicators not modeled) Are the results useful for improvement? Are negative consequences mitigated? Research is ongoing and should shape both initial design and ongoing refinements 17 Evidence the System is Working