Presentation on theme: "Measuring Child and Family Outcomes"— Presentation transcript:
1 Measuring Child and Family Outcomes Session OneLinking Assessment to Functional Outcomes and AccountabilityAnne Brager, MS, RNProgram SupervisorFrederick County Infants and Toddlers ProgramAdapted from ECO Center
2 Good Outcomes Begin with Good Assessment! What are Outcomes?Outcomes are the benefits that children and families experience as a result of early interventionThe three child outcomes, as measured for accountability purposes are different than IFSP outcomes in that:OSEP’s three child outcomes for accountability apply to all children and reflect global functioning in broad areas of development (social-emotional, knowledge and skills, getting needs met)IFSP outcomes are specific to an individual child, based on his or her individual needs
3 What is Assessment?“Early childhood assessment is a flexible, collaborative decision-making process in which teams of parents and professionals repeatedly revise their judgments and reach consensus about the changing developmental, educational, medical and mental health services needs of young children and their families.”(Bagnato and Neisworth, 1991)Quoted in DEC Recommended Practices, 2005
4 Assessment is…“Assessment is a generic term that refers to the process of gathering information for decision-making.”Mary McLean, 2004
5 Purposes of Assessment In the early intervention systemassessment serves several purposes:Identification of children with special needs (including screening, diagnosis, and determination of eligibility for services)Planning instruction or interventionMonitoring of the child’s level of knowledge or progressProgram evaluation and accountability
6 Accountability???Accountability involves embedding ongoing data collection into program practices because the need to examine the effectiveness of the services being provided is ongoingGiven the number of factors that influence the provision of services (e.g., staff changes, reimbursement formulas), it is not reasonable to assume that a specific program that was effective at one point in time will always be effective
7 Assessment for Accountability Policymakers and funders are raising legitimate questions about which programs are effective as they struggle with how to allocate resourcesThe Government Performance and Results Act (GPRA) means Part C Programs need to provide evidence that program participants demonstrate positive outcomesAn important issue for assessment is that the Office for Special Education Programs (OSEP) decided not to require states to use the same assessment tool. Each state is free to chose their own
8 Assessment as a Process It is important to distinguish betweenassessment as a process andassessment as a single instrument or test!Assessment as a process involves multiple ways of collecting information about a child’s performance, including, for example, observation, documentation of the child’s work, checklists, rating scales, portfolios and norm-referenced or curriculum based tests
9 Variation in What Tools are Used So far these are the tools we know thatstates are using for accountability:Of 28 states who listed specific assessment instruments:HELP-15 statesBDI/BDI-2-13 statesAEPS-11 statesCreative Curriculum-6 statesELAP-6 statesNot yet determined-23 states
10 Variation in How Assessments Have Been Used In the past, there has been much variationin assessment procedures in earlyintervention programs across the country:In some programs, the only formal assessment has been for eligibilityOther states completed assessment annuallySome states have always completed assessment at exit
11 Views on Assessment There have been major changes in last 15 years in how assessment of youngchildren is viewed by the early childhoodcommunity:Old position: Do not test little kidsNew position: Ongoing assessment is part of a high quality early childhood program
12 New Thoughts on Assessment “The best way to understand the development of children is to observe their behavior in natural settings while they are interacting with familiar adults over prolonged periods of time”--Uri Bronfenbrenner
13 What Changed our Thinking The purpose of assessment was redefinedin the last decade:It’s no longer about sorting, labeling, using to deny access to servicesIt’s now about getting a rich picture of what children can do and can’t do and using that information to help them acquire new skillsIt’s now about monitoring progressNew and different tools became available for generalearly childhood: Curriculum-based assessments were developed, e.g., Creative Curriculum, Work Sampling,etc.
14 What Changed Our Thinking Assessment had always been seen as a process with multiple purposesDistinctions have been made between good and bad uses of assessment with young childrenGood assessment practices are now widely recognized as a key component of high-quality early childhood programsGood uses are now promoted!For more information: National Association for Education of Young Children (NAEYC) web site (Position statement on Curriculum, Assessment and Evaluation)
15 NAEYC Position Statement Policymakers, early childhood professionals and otherstakeholders in young children’s lives have a sharedresponsibility to:Make ethical, appropriate, valid and reliable assessment a central part of all early childhood programs.To assess young children’s strengths, progress,and needs, use assessment methods that aredevelopmentally appropriateculturally and linguistically responsivetied to children’s daily activitiessupported by professional developmentinclusive of families
16 NAEYC Position Statement continued… Connected to specific, beneficial purposes:Making sound decisions about teaching and learningIdentifying significant concerns that may require focused intervention for individual childrenHelping programs improve their educational and developmental interventions
17 DEC Recommended Practices for Assessment The Recommended Practices emerged fromfocus groups and are supported by literaturefrom the Division of Early Childhood (DEC):Parents and professionals who participated in these focus groups identified repeatedly two concernsPrinciple stakeholders, parents and family members must play a vital and indispensible role in assessment from beginning to endAssessment methods and materials must accommodated children’s developmental and disability specific characteristics
18 DEC Recommended Practices for Assessment The Recommended Practices are organizedaround five statements:Professionals and families collaborate in planning and implementing assessmentAssessment is individualized and appropriate for the child and familyAssessment provides useful information for interventionProfessionals share information in respectful and useful waysProfessionals meet legal and procedural requirements and Recommended Practice guidelines
19 Assessment for Accountability The assessment process we use to determine eligibilityand program planning can be used for accountability!The three child outcomes:Children have positive social-emotional skills (including social relationships)Children acquire and use knowledge and skills (including early language/communication [and early literacy])Children use appropriate behaviors to meet their needsAssessment tools can inform us about children’s functioning in each of the three outcome areasChallenge: There is no assessment tool that assesses the three outcomes directly
20 Additional Challenges Young children are unreliable test takersBecause of their limited ability to understand and follow the directions of the testThey may not be able to control their behaviorThey have short attention spansAnd are easily distractedWhat young children demonstrate in a test situation may not be an accurate representation of what they are capable of doing on a daily basis
21 Types of Tests Standardized vs curriculum based By definition, standardized tests must be administered the same way to all test takers.The use of a consistent approach during test administration enhances reliability between administrators and confidence that children are responding to the same tasks.Standardized tests usually provide normative data to compare the child’s functioning with that of other children of the same age.
22 Standardized Tools Few standardized assessment tools provide detailed guidelines for appropriateaccommodations during testing toeffectively measure capabilities amongthe broad range of children who havespecial needs.” McLean, 2005
23 Curriculum-based Assessments Bagnato and Neisworth (1991) define curriculum-based assessment as a “form of criterion-referenced assessment wherein curricular objectives act as the criteria for the identification of instructional targets and for the assessment of status and progress”Some of the newer assessment tools emphasize critical skills to facilitate the development of fuctional behavior
24 Currently Available Assessment Tools Most assessment tools are organized around developmental domains and are based on sequences of skills that build on each otherThis organization may pose difficulties for accurate assessments of functioning in young children with disabilities because delayed skills in one domain can influence assessment of or functioning in other domainsEach assessment tool sees children through its own lensEach lens is slightly differentLens are not right or wrong
25 Variations Among States States are continuing to make key decisions related to their accountability systems, so any information about state choices must be seen as time-limited and interpreted cautiouslyIn many state, some assessment-related decisions are being made by local programs, so there is variation within the statesThe information available thus far suggests that much of the data on child outcomes will be collected with curriculum-based tools
26 Variations Among States Most states are building their accountability systems around assessments that were already being administered for other purposes (such as eligibility determination or program planning)Few have adopted a new statewide measure solely for collecting data for accountability purposes
27 Maryland’s Top FiveFollowing a survey distributed to all local Program Directors in the state, Maryland state Department of Education (MSDE) identified the 5 most commonly used assessment instruments in Maryland.Results from at least one of these assessment instruments (or other instrument if more appropriate), in conjunction with information obtained from other possible sources, including record review, interview, observation, and screening, will be used in providing the information required to complete the Child Outcomes Summary Form (COSF).Ideally, the same tool would be used at entry and exit. However, use the tool that is most appropriate for the child.
28 Maryland’s Top Five The five most common assessment measures are: Early Learning Accomplishment Profile (E-LAP)Early Intervention Developmental Profile (EIDP)Hawaii Early Learning Profile (HELP)Battelle Developmental Inventory IIAges and Stages
29 Key Questions for Accountability: How much and what information will the tool provide about the child’s current level of functioning and the attainment of the 3 child outcomes?Would providers intentionally or unintentionally give children higher scores than warranted, knowing that conclusions about program effectiveness would be based on the information?
30 Assessment Tool LensEach assessment tool carries its own organizing frameworkMost are organized around domains but children integrate skills when performing functional activitiesNo right or wrong lens:How can what the assessment tool provides be used to develop functional outcomes for young children?How can what the assessment tool provides beused as information about the achievement ofthe three outcomes?
31 Critical Assumptions Related to the 3 Child Outcomes Achievement of the outcomes is age based, children of different ages will demonstrate achievement in different waysThere are many pathways to competence for children with atypical development (using sign language, wheelchair). This seems obvious but can get lost when an assessment tool uses a different lens.
32 Remember ThisFlexibility is required in using assessment tool results to make decisions about a child’s functioning on the outcomesTeam needs to decide what information from an assessment tool is relevant to understanding child’s functioning
33 More DEC Recommended Practices for Assessment Assessment provides useful informationfor intervention!Assessment results are immediately useful for planning program goals and objectivesGood assessment practices assess not only immediate mastery of skill, but also whether child can demonstrate skill consistently across settings and with other peopleGood assessment practices also assess the presence and extent of atypical child behavior that may be a barrier to intervention and progress
34 Using the Assessment Results for Outcomes What is an Outcome:Outcomes are the benefits that children and families experience as a result of early interventionThe three child outcomes, as measured for accountability purposes are different than IFSP outcomes in that:The three child outcomes reflect global functioning in broad areas of development (social-emotional, knowledge and skills, getting needs met)IFSP outcomes are specific to an individual child, based on his or her individual needs
35 What Makes an Outcome Functional? Functional Outcomes: Let’s Look at Using Assessment Information to Develop Functional IFSP OutcomesWhat Makes an Outcome Functional?Functional Outcomes:Refer to things that aremeaningful to the child in thecontext of everyday livingRefer to an integrated seriesof behaviors of skills thatallows for the child to achievethe important everyday goals
36 Meaningful OutcomesMeaningful outcomes promote a child’s functioning in three key foundations of early development - social interaction, mastery over environment, and engagement in learning - in ways specific to each family and child(McWilliam, 2002; National Research Council, 2000).
37 Writing IFSP Family Centered, Measureable, Functional Outcomes Family-centered outcomes are written to address:The desires and goals a family has for them and their childCreated with the family after assessing what activities are meaningful to individual family members.Includes not only activities of daily living but also the family's ability to participate in cultural and social experiences they hold valuable
38 Functional Outcomes“Function” refers to those activities identified by the family that support the development of the child’s physical, social and psychological well being, i.e. the ability to feed oneself, to hold a toy, to communicate, or to playEarly intervention teams do not “treat” the primary diagnosis, i.e. cerebral palsy, autism, but instead treat the functionaldelay or disability
39 Determining Appropriate IFSP Functional Outcomes A few questions to consider:What are the activities the family would like to do that are difficult?Has the family given up on any family outings, experiences or chores?What kinds of things would the family like for the child to do that would make life easier for them or more fun for the child?Read more at MD’s IFSP Tutorial: Assisting Families to Select Functional Outcomes
40 Measureable IFSP Outcomes Outcomes should be written so that all members of the team will understand when the outcome has been met.Outcomes should address one of the following areas:They should enhance the family’s ability to care for or to engage in activity with their childThey should enhance the child’s ability to participate in functional activities (feeding, dressing, moving in his environment, communicating, playing, etc.)They should expand on activity settings in which the child already participatesRead more at MD’s IFSP Tutorial:Measurable Criteria: How we Know we have Achieved Outcomes
41 Effective Strategies on an IFSP Identify How Functional Outcomes Will be Achieved Each strategy should address:Individualization- Do strategies build on child and family interests?Context- Do strategies build on familiar places, people, and routines?Mastery- Do strategies ensure generalization of a child’s actions and interactions in multiple settings and tasks?Collaboration- Do strategies specify who will do what?Read more at MD’s IFSP Tutorial:Effective Strategies Identify How Functional Outcomes will be Achieved
42 Making a Difference! The results of improved assessment and individual IFSP outcomes developmentand services is two-fold:First and foremost, it ultimately leads to enhanced services for children and familiesWhich, as a result, can’t help but positively impact Maryland’s outcome measurement results!
43 More to Come… Next week: Explore Maryland’s approach to gathering child outcome data tomeasure OSEP’s Child OutcomesThe role of the ChildOutcomes Summary FormExplore other state’s approaches