Presentation on theme: "An Overview of Methodologies of Assessing Educational Adequacy"— Presentation transcript:
1 An Overview of Methodologies of Assessing Educational Adequacy Bureau of Legislative ResearchPolicy Analysis & Research SectionAn Overview of Methodologies of Assessing Educational AdequacyAugust 20, 20132014 Adequacy
2 BackgroundA primary “driver” of educational finance has been litigation in courtsthroughout this country over the past four decades.Historically, two primary legal frameworks or arguments, adequacyand equity, have served as the guiding principles for making decisionsabout resource distribution to school districts that vary in terms ofstudent characteristics and concentrations of poverty.Equity arguments are concerned with comparative measures of anyinequities in resources needed to provide equal educational opportunitiesto all students, irrespective of individual disadvantages or community wealth.In contrast, adequacy studies focus on sufficiency of resources inevery school district to provide an adequate education to all studentsbased on some educational standard.
3 Equity & AdequacyThe written report contains a brief overview of evolution of legal argumentsfrom the concept of “equity” to one of “adequacy” in this country.The pivotal distinctions between equity and adequacy are revealed in theirprimary goal and measures. For example, equity (or equality) of resourcescan be achieved by “leveling down” the distribution of resources to everyschool district. Since equity is measured comparatively, it is possible tohave equality in resource allocation without providing resources needed foran adequate education.Similarly, it is possible to provide sufficient resources to all districts in order tooffer an adequate education without attempting to equalize the educationalopportunities for all students in a state.In practice, while providing sufficient resources to offer an adequateeducation to all children has become the primary goal in resource allocation,equity remains in the background as states develop categorical funding fordisadvantaged students, or calculate mathematical weights for additionalfunding for these students, to create equal educational opportunities.
4 Purpose of the ReportThus, Rebeca Whorton, an analyst with the Bureau of Legislative Research,will present equity analyses of Arkansas school districts in February, 2014.The purpose of the current report is to present a discussion and briefcritique of the four primary methodologies used to assess educationaladequacy: 1) evidence-based model, 2) professional judgment, 3) successfulschools approach, and 4) cost function (or statistical) analysis. It is not theaim of this presentation to advocate for one method over another, or to offerany recommendations for change in current study procedures.Arkansas and Kentucky have relied on the evidence-based model todetermine amount of resources needed to provide an adequate education.In the evidence-based model, consultants are hired to design a package, orrecipe, of resources (e.g., our Matrix handout) needed to provide anadequate education to all students. Based on their knowledge of researchmethodology and existing evidence, consultants make decisions about typesand amounts of resources needed to achieve defined education standards.
5 Evidence-based ModelRecommendations regarding the distribution of resources are based onconsultants’ interpretations of the effectiveness of various resources infacilitating student achievement.Adequacy is assessed by comparing district resources to the resourceallocation package (Matrix) recommended by the expert consultants. Costs(or expenditures) for each resource are based on estimates found in theschool finance literature, and these costs are summed to arrive at a totalcost (or per pupil cost) for funding adequacy.A salient strength of the evidence-based model is the use of experts whohave extensive experience in conducting and evaluating research onresource allocation and student performance. They typically are highly trainedmethodologists who have a thorough knowledge of the research on schoolfinance and student achievement.Furthermore, the evidence-based adequacy study process is transparent andeasily understood, and it examines current data from the educational systembeing evaluated.
6 Critique of Evidence-based Model Critics point out that the evidence-based model focuses exclusively on inputs(resources) to the exclusion of outcomes (e.g., student achievement).Many contemporary researchers and policy-makers argue that the ultimategoal of education is student achievement, and therefore, an adequateeducation should be assessed in terms of student performance.The only linkage between resources and student achievement in theevidence-based approach to educational adequacy are the extrapolationsconsultants make from prior research in designing a resource allocation planor Matrix. However, critics argue that studies cannot be generalized acrossstates that differ in demographics, such as poverty and parent education.They also criticize the piecemeal practice of providing evidence for eachresource separately instead of investigating the impact of the total resourceallocation package consultants recommend to states.
7 Critique of Evidence-based Model Critics point out expert judgments are influenced in immeasurable ways bypersonal preferences and biases, and they cite examples of how consultantshave “cherry picked” studies that support their resource allocation, whileignoring contrary evidence.They also note that there is scarce research on the effectiveness of resourceson student achievement, and in some cases there is no evidence to supportrecommended resources.Finally, reviewers observe that reports written by expert consultants typicallydo not articulate the methodological criteria used to select studies thatprovide support for resources recommended, which raises questions aboutthe validity of studies used.There appear to be no published examinations of the validity (accuracy) orreliability (consistency between evaluators) of the evidence-based method.
8 Professional Judgment Model The professional judgment method of examining educational adequacy relieson the opinions of a panel of local educators (e.g., superintendents, educationprofessors) instead of hired expert consultants. Panels have been constitutedby expert consultants, legislators, and governors.This panel of educators meets over a period of time to construct a packageof resources. Once the resource allocation plan has been determined, thesame panel, or a different group of experts, estimates the costs of eachresource component.To address the issues of validity and reliability, some states have used morethan one panel that work separately to derive independent resourcedistribution models and costs.The panels eventually meet together to integrate the independently-derived resource allocation plans. At times, panels have been informed bysurveys of teachers, principals, and superintendents concerning neededresources and costs.
9 Critique of Professional Judgment Model An important advantage of the professional judgment approach to assessingeducational adequacy is the deliberation by local professionals who arecurrently involved in the system being assessed, and intimately familiar withdifferences in districts, their resource needs, and regional costs.This intimate knowledge is a “dual-edged” sword because, in practice,panels of educators have tended to simply present a “wish list” of resources,rather than design an efficient distribution of resources that supportseducational adequacy.Professional judgment methods of assessing adequacy typically have resultedin costly resource allocation models.Educator panels rely on personal judgments based on knowledge andexperience rather than on a systematic, clearly articulated set ofmathematical procedures, such as statistical analyses. The personalpreferences and biases inherent in professional judgments are especiallyproblematic in making adjustments in resource allocation for highconcentrations of poverty and other disadvantages.
10 Successful Schools Model The fundamental premise of the successful schools approach is that it ispossible to determine an adequate base cost level by examining the basicspending of successful school districts. The successful schools methodbegins by identifying a subset of the schools in a state that are effective atmeeting educational goals concerning student performance.After agreement is reached on what constitutes a successful school, schoolsmeeting the criteria are identified, and current expenditures for resources ofthese schools are calculated.The average expenditure of these schools, or some percentage thereof, isestablished as sufficient for providing an adequate education. Expendituredata identify how dollars are spent, and spending patterns are used toestablish resource allocation plans to provide an adequate education.To assess differences in resource allocation between successful andcomparison schools, researchers use regression statistics to examine howwell resource expenditures and student and district characteristics predictachievement.
11 Critique of Successful Schools Model The regression analyses provide estimates of the impact of resources onachievement, after considering the effects (or influences) of student anddistrict characteristics. The results of the regression analyses are used todevise resource allocation plans.The most salient strength of the successful schools model is the intuitivelyappealing process of using the resource allocation of “successful schools” asa model for other schools.This apparent strength has also proved to be a limitation in practice becauseevidence indicates that successful schools have different resource needs anddemographic characteristics than schools with high concentrations of poverty.Early studies attempted to address this disparity by identifying successfuland unsuccessful schools within different demographic categories (or groups),such as high, middle, and low income schools.
12 Critique of Successful Schools Model However, studies have found that there are too few successful schools withhigh concentrations of poverty. More recent studies have addresseddemographic disparities by using sophisticated statistics that examine therelationship between resources and achievement, after adjusting for studentand district demographic characteristics.Despite sophisticated adjustments for demographics, resource allocationmodels in successful schools often are not useful for schools with highconcentrations of poverty because of differences in resource needs.For example, schools with high concentrations of poverty often are locatedin communities that experience problems recruiting and retaining teachers,especially in critical subject areas.Furthermore, analyses of successful schools typically do not have data onfactors that have been shown to make the difference between success andfailure, such as quality teaching and leadership.
13 Cost Function (Statistical) Model The cost function approach provides cost estimates of every resource neededby each district to reach a particular performance standard.These estimates reply on clearly defined standards, accurate data, andsystematic statistical adjustments for student and district characteristics thatmake the cost of achieving any given standard higher in some districts than inothers.A cost function statistical analysis systematically examines the relationshipsbetween resources (costs and staffing) and student achievement for eachschool district based on its particular student and district characteristics.The statistical coefficients derived from regression statistics provide costestimates for every resource for each district, and these estimates indicatethe relative impact of each resource on student performance.Finally, the statistical analysis indicates the amount of influence the resourcepackage (or Matrix) has on student achievement.
14 Factors in Cost Function Analysis PovertyInstructional FacilitatorsStudent Services CostsInstructional CostsAchievementCommunity WealthEfficiencyNSLTutoring
15 Critique of Cost Function Model A particular strength of the cost function approach is the inclusion of anefficiency measure in estimating costs for each district.The cost function approach to assessing educational adequacy has twomajor vulnerabilities that can limit its usefulness.Sophisticated mathematical procedures used in the cost function analysesrequire valid (accurate) and reliable (consistent) data. These statisticalprocedures compound any errors in data, which results in distorted findings.The cost function approach typically relies exclusively on availableadministrative data, including salaries, types of staffing, and expenditures forvarious educational functions.However, national research (meta-analyses) indicates that interaction factors,such as quality of teaching and leadership, play a more significant role instudent achievement gains.
16 Discussion & Conclusions Each of the methodologies used for assessing educational adequacy offersadvantages and weaknesses in comparison with the other methods.An alternative to selecting one of these approaches for adequacy study is touse more than one and compare results. Several states have either usedtwo approaches simultaneously, or alternated methods from one year toanother.Also, states have taken advantage of the differential strengths of the methodsby using them for different purposes.For example, expert consultants might be commissioned to design a nationalstate-of-the-art resource allocation system (Matrix) that accounts for thediversity in student and district characteristics.Their system of resources then could be used as a template to initiate adiscussion of resource needs in a local educator panel (or panels). Thispanel’s proposed resource allocation strategy could serve to inform a moresystematic statistical analysis of resources, student and districtcharacteristics, and student performance outcomes.
17 Various Adequacy Methodologies The documents below can be found at the following link:A 50 State Strategy to Achieve School Finance Adequacy, Odden and Picus, 2008Achieving Educational Adequacy Through School Finance Reform, 2000Equity vs. Adequacy in the State's Provision of Education, 2006Performance Standards and Educational Cost Indexes, 1998Responding to the Charge of Alchemy, 2006School Outcomes and School Costs, Texas A&M UniversityStudy of Educational Adequacy, How Much Money is Enough, 2010The Alchemy of Costing Out an Adequate Education, Hanushek, 2005Pseudo-Science and a Sound Basic Education-Voodoo Statistics in New York, Hanushek, 2005
18 Bureau of Legislative Research ContactDr. Brent BendaBureau of Legislative Research(501)