1. OPENING THE BLACK BOX OF INDIVIDUALIZATION IN PROGRAMS FOR YOUNG CHILDREN Division of Early Childhood Conference San Francisco, CA October 18, 2013

2. The Black Box

3. HOW DO WE USE PROGRESS MONITORING TO HELP US INDIVIDUALIZE?

4. Panelists Judy Carta, Senior Scientist, University of Kansas Sally Atkins-Burnett, Senior Researcher, Mathematica Policy Research Charlie Greenwood, Senior Scientist, University of Kansas Alisha Wackerle-Hollman, Research Associate, University of Minnesota Jane Squires, Professor, University of Oregon

5. Questions we want to discuss How do we do progress monitoring and how do we use it to individualize for young children in our programs? How do we know when progress monitoring and individualizing are being done well? What are our quality indicators? Do we have evidence that progress monitoring and individualizing leads to better outcomes? What do programs need to know to determine when progress monitoring is done well?

6. Format for today’s session Sally will describe a recently completed review of literature describing what we know about progress monitoring and how it’s used for individualization Charlie will describe a general outcomes approach to progress monitoring with infants and toddlers, describe how it’s used for individualization and ways to determine the quality of implementation of progress monitoring Alicia will describe a general outcomes approach to progress monitoring with preschool-aged children Jane will describe a curriculum-based approach to progress monitoring and individualization We will have time to hear from YOU.

7. We want to hear from you!

8. Assessing Early Childhood Teachers’ Use of Child Progress Monitoring to Individualize Teaching Practices DEC Conference October 18, 2013 Sally Atkins-Burnett, Lauren Akers, Patricia Del Grosso, Shannon Monahan, Judith Carta, Barbara A. Wasik, Kimberly Boller

9.  Evaluate the existing evidence –Are there existing measures of teachers’ use of assessment for individualization? –What is important to measure?  Develop evidence-informed conceptual model of ongoing child assessment for individualizing instruction  Develop a plan to efficiently assess implementation of the model –What methods and modes of data collection are needed? –What is feasible?  Develop and pretest a measure that could inform early childhood research and practice Project Aims 9

10. Literature Review Approach and Preliminary Findings 10

11.  Identify the key areas to include when evaluating the activities involved in the process of monitoring child progress and using that information for instruction and individualization  Find examples of how others have measured teachers’ implementation of ongoing assessment and progress monitoring  Identify gaps in the literature  Inform the project’s conceptual framework Purpose of the Literature Review 11

12.  To identify studies we: –Conducted a structured library search of last 10 years of research –Solicited recommendations from the project team and expert consultant group  We identified 1,322 studies; of those 198 met relevance criteria and were screened into the review –Studies screened out for being off-topic, not an eligible target population, not a relevant document type, duplicate studies, or not published in English Methods and Results 12

13.  To identify studies we: –Conducted a structured library search of last 10 years of research –Solicited recommendations from the project team and expert consultant group  We identified 1,322 studies; of those 198 met relevance criteria and were screened into the review –Studies screened out for being off-topic, not an eligible target population, not a relevant document type, duplicate studies, or not published in English Methods and Results 13

14.  10 studies reported on teachers’ perceptions or experiences with progress monitoring and using data to inform instruction –Commonly cited barriers to implementation: skill needed to use data for individualization and knowledge of the subject matter –Teachers reported the need for additional training Teachers’ Perceptions of Progress Monitoring 14

15.  No studies focused on how teachers actually select observation and assessment targets and methods  Over one-third of studies described methods for documenting children’s progress and systems for organizing information –Most only briefly mentioned the types of methods used –Studies frequently discussed web-based or technology-enhanced systems –8 studies explored implementation experiences Activities Involved in Using Progress Monitoring 15

16.  Few studies described how teachers interpret data and/or apply it to instruction and individualization  17 studies discussed ways to engage families in progress monitoring –Studies described using it to provide regular feedback to parents on children’s progress or engaging families in collecting data and using data for goal-setting –One study focused on teaching primary caregivers to conduct formative assessments Activities Involved in Using Progress Monitoring (cont’d) 16

17.  Studies most often measured what teachers do –14 of 18 studies measured whether teachers implemented a specific progress monitoring tool with fidelity or reliability –5 studies examined the instructional modifications teachers made  4 studies measured what teachers think –1 study used teacher written reports; another study used a series of teacher interviews, including a think- aloud data analysis scenario  2 studies assessed what teachers know about child development, assessment, and/or instruction Measures Used to Assess Implementation 17

18.  Web- or computer-based systems can assist teachers in documenting, organizing, interpreting, and planning how to use data –Implementation issues can hinder the utility of these systems  Ongoing professional development and support for teachers may assist teachers with using data to individualize instruction  Families are essential partners Features of Successful Implementation 18

19.  Very limited research exists about the use of progress monitoring in domains other than language/literacy, social-emotional, and math  Minimal research has focused on –Using progress monitoring in home visiting programs –Supporting families to in collecting assessment information and observing their children  Research points to the importance of ongoing support for teachers –Although much of this research has been conducted with teachers in K-3 Gaps in the Literature 19

20.  Few studies have assessed implementation of progress monitoring and the individualization process –No studies have assessed implementation across a range of progress monitoring tools –Studies typically only looked at one or two of the activities involved in the process of using child progress monitoring for instruction and individualization Gaps in the Literature (cont’d) 20

21.  Few studies have assessed implementation of progress monitoring and the individualization process –No studies have assessed implementation across a range of progress monitoring tools –Studies typically only looked at one or two of the activities involved in the process of using child progress monitoring for instruction and individualization Gaps in the Literature (cont’d) 21

22.  Continuous, frequent, and standard assessment of child progress toward a long-term goal or outcome  Repeated measurement of a set of indicators predictive of a later outcome –Increasing proficiency indicated by rate of growth –Trend line compares expected versus actual rates of learning  Brief and quick to administer General Outcomes Approaches 22

23. Conceptual Model for the GOM Approach 23

24.  Information often collected within the context of the delivery of the curriculum  Assessments closely aligned to the curriculum  Assessments intended to be authentic in context Curriculum-Embedded Approaches 24

25. Conceptual Model for the Curriculum-Embedded Approach 25

26.  Please contact: –Sally Atkins-Burnett SAtkins-Burnett@mathematica-mpr.com –Kim Boller kboller@mathematica-mpr.com For More Information 26

27. A GENERAL OUTCOMES APPROACH TO PROGRESS MONITORING AND INDIVIDUALIZING FOR INFANTS AND TODDLERS Charles Greenwood, Division of Early Childhood Conference San Francisco, CA October 18, 2013

28. My Topics for Today How is individualization defined? What is my approach to progress monitoring? What framework guides data-based individualization? How is progress monitoring data used to track a child’s response to intervention? How is implementation quality addressed? Is it working to improve children’s outcomes?

29. How is Individualization Defined? A long held tenet of educational and behavioral psychology and early intervention is that instruction (intervention) should be “adjusted” based on its observed effects on the learner If effective, continue and improve the intervention If not effective, change the intervention and try something else Individualization is occurring when this kind of dynamic “adjusting” is happening for all children in a program when needed

30. What is My Approach to Progress Monitoring? General outcome measurement in the form of infant/toddler Indicators of Individual Growth and Development (IGDIs) Seasonal universal screening (quarterly) Monthly progress monitoring for children receiving intervention IGDI benchmarks and trends over time (decision points) are used for making intervention decisions

31. Individual Child’s Growth Chart with Benchmark Indicators Child’s Scores Normative Trajectorie s

32. Yes What Framework Guides Data-Based Individualization? No Quarterly ECI Assessments Is the intervention being implemented? What intervention strategies should be used? Yes What is causing the problem? Is there a problem? Is the intervention working? Tilly, W. D. (2002). Best practices in school psychology as a problem solving enterprise. In A. Thomas & J. Grimes (Eds.), Best practices in school psychology IV (Vol. 1, pp. 21-36). Washington, DC: National Association of School Psychologists. No

33. How is Implementation Quality Addressed? The infant/toddler IGDI website provides supports: information training Implementation data handling (entry, charting) reporting (child and program levels) making intervention decisions (MOD)

35. How is Implementation Quality Addressed (Continued)? Programs can monitoring their own implementation quality in two ways: Monitoring the quality of IGDI data collection within a program and managing issues Tracking the fidelity of intervention provided by home visitors and parents in the home

36. Program Level Reports Data Collection Indicators

37. Two Fidelity Indicators: 1. Certified Assessors 2. Percentage Outliers

38. What Should Be Done?

39. Is it Being Done?

40. How is Progress Monitoring Data Used to Track a Child’s Response to Intervention? A child’s individual growth trajectory interrupted at a point in time is examined for evidence of change in level and slope A positive effect of intervention is signaled by a change in level, change in slope, or both A none effect of intervention is signaled by no change in level or slope

41. Is the Intervention Working – Child Level? Change in Slope Slopes Before and After

42. Is It Working To Improve Children’s Outcomes? Program’s can monitor: Child-level progress over time Program level progress over time

43. Is it Working to Improve Children’s Outcomes? – Program Level

44. References Buzhardt, J., Greenwood, C. R., Walker, D., Anderson, R., Howard, W. J., & Carta, J. J. (2011). Effects of web-based support on Early Head Start home visitors’ use of evidence-based intervention decision making and growth in children’s expressive communication. NHSA Dialog: A Research-to- Practice Journal for the Early Childhood Field, 14(3), 121-146. Buzhardt, J., Walker, D., Greenwood, C. R., & Carta, J. J. (2011). A study of an online tool to support evidence-based practices with infants and toddlers. NHSA Dialog: A Research-to-Practice Journal for the Early Childhood Field, 14(3), 151-156. Greenwood, C. R., Buzhardt, J., Walker, D., Howard, W. J., & Anderson, R. (2011). Program-level influences on the measurement of early communication for infants and toddlers in Early Head Start. Journal of Early Intervention, 33(2 ), 110-134. Greenwood, C. R., Walker, D., & Buzhardt, J. (2010). The Early Communication Indicator (ECI) for Infants and Toddlers: Early Head Start Growth Norms from Two States. Journal of Early Intervention, 32(5), 310-334. Greenwood, C. R., Walker, D., Buzhardt, J., Howard, W. J., McCune, L., & Anderson, R. (2013). Evidence of a continuum in foundational expressive communication skills. Early Childhood Research Quarterly, 28, 540-554. Greenwood, C. R., Walker, D., Buzhardt, J., McCune, L., & Howard, W. J. 2013). Advancing the construct validity of the Early Communication Indicator (ECI) for infants and toddlers: Equivalence of growth trajectories across two Early Head Start samples. Early Childhood Research Quarterly, 28, 743-758.

45. Intentional Individualization: Using IGDI 2.0 progress monitoring data to inform instruction Alisha Wackerle-Hollman Ph.D. Division of Early Childhood Conference October 18th, 2013

46. “OUR BEST TEACHERS TODAY ARE USING REAL TIME DATA IN WAYS THAT WOULD HAVE BEEN UNIMAGINABLE JUST FIVE YEARS AGO. THEY NEED TO KNOW HOW WELL THEIR STUDENTS ARE PERFORMING. THEY WANT TO KNOW EXACTLY WHAT THEY NEED TO DO TO TEACH AND HOW TO TEACH IT. (DUNCAN, 2009)

47. Individual Growth and Development Indicators (IGDIs 2.0)- Design Principles General Outcome Measure Approach –Treatment Independent ( Slavin & Madden, 2011) Construct-Aligned and Validated for Specific Purposes –Validity as an argument (Kane, 2013) Psychometrically Robust –Rasch Modeling Potential to match test material to student ability “window” Increased opportunities for sensitivity to growth

48. Individual Growth and Development Indicators (IGDIs 2.0)- Content & Tasks Four domains of early language and literacy –Oral Language –Phonological Awareness –Alphabet Knowledge –Comprehension Five IGDI 2.0 tasks: Picture Naming, Rhyming, First Sounds, Sound Identification, and Which One Does Not Belong?

49. Individual Growth and Development Indicators (IGDIs 2.0)- Content & Tasks

50. Progress Monitoring: Making Decision 30 item sets selected from a performance range characteristic of Tier 2/Tier 3 candidacy. Designed to be used every three weeks –Efficiency in data-based decision making and practical utility (Jenkins & Terjeson, 2011) –However, K-8 data suggests frequency may need to be MORE often ( Christ, Zopluoglu, Monaghen & Van Norman, 2012) Addressing Responsiveness The balancing act- the year before kindergarten Waiting for a skill to emerge Waiting to make instructional changes Waiting for robust trends in performance

51. Individual Growth and Development Indicators (IGDIs 2.0)- Content & Tasks

52. Progress-Monitoring: Making Decision Evaluation of Progress –Slope-based standards based on relative goals (Jenkins & Terjeson, 2011; Jenkins, Graff, Miglioretti, 2009) –Goals established in reference to seasonal cut- points Intervention Individualization – Connecting the construct of interest to meaningful and evidence-based intervention –Improving individualization through additional data sources Combining GOM indicators with Mastery Monitoring information to best serve the student. Error Analysis

53. Instructional Changes How do we examine student responses to use data to make modifications?

54. Instructional Changes

55. Intervention Considerations What duration of intervention is sufficient? –Constraints of Classroom dosage –4 weeks as a bottom-line minimum to see effectiveness? (Christ et al., 2012) –8 weeks minimum for appropriate instructional changes? (Ardoin, Christ, Morena, Cormier & Klingbeil, 2012; Shapiro, 2013) Intervention Effectiveness Factors –Quality of intervention –Quality of Fidelity of intervention administration –Dosage

56. Implementation Quality Administrator Training – Bi-annual fidelity checks – Emphasis on Standardization Scheduling and Real-Time Data-Based Decision Making –Behavioral Momentum –Zone of Proximal Development Chicken or the Egg? – Quality of intervention –Quality of assessment tool in sensitivity to change

57. Improving Outcomes: Certain Success? Reaching students with comprehensive efforts for intervention –Across academic and social settings –Increased opportunities to respond Monitoring progress frequently and with fidelity Utilizing opportunities for data-based decision making – Movement between Tier-level intervention –Maintaining Intervention long-enough to substantiate potential changes. Reflecting on screening data –Are we changing the RTI triangle proportions?

58. Black Box of Individualization

60. Individualization  To consider or treat individually; particularize  Modified to suit wishes or needs of particular individual

61. Individualized Education Program/Individualized Family Service Plan  Child’s present levels of functioning/achievement  How disability affects involvement in general ed curriculum  How disability affects child’s participation in activities  Goals that designed to meet child’s needs resulting from disability

62. Systems Approach  Individualization in context of linked system, using CBA  Individualization and progress monitoring embedded in assessment and in daily intervention activities

63. Linked System  System in which program goals, assessment procedures, curricula/intervention, and program evaluation are related in content and in process

64. Program Goals & Philosophy Linked System Approach Collecting Information Summarizing Information Monitoring Curricular Approach Weekly Weekly Quarterly Quarterly Annual Annual Observations Observations Direct tests Direct tests Report Report Child goals Child goals Family outcome Family outcome Activity- based Intervention Activity- based Intervention Assessment IFSP Evaluation Intervention

66. Individualization of Instruction 4 Primary Steps 1. Get to know each child’s interests, needs, and abilities 2. Create opportunities for learning that build on children’s interests  Pretti-Frontczak & Bricker, 2004

67. Individualized Instruction 3. Scaffold children’s learning through supportive interactions 4. Monitor children’s progress toward achieving important goals Pretti-Frontczak & Bricker, 2004

68. Individualizing Instruction  Be knowledgeable about learners  Create learning opportunities embedded in daily routines, activities, experiences and draw into instructional interaction

69. Individualizing Instruction  Implement a planned and structured approach for curriculum content  Make thoughtful decisions about kind and amount of support for children  Monitor success of instruction to make sound decisions to support learning and development  Boat, Dinnebeil, & Baie.( 2010) Individualizing Instruction in Preschool Classrooms, Dimensions of Early Chilldhood (38)1, 3-10.

70. Activity-Based Intervention  Foundation of effective individualization  Child-initiated transactions  Children’s goals/objectives embedded in routine, planned or child-initiated activities  Logically occurring antecedents and consequences  Functional skills learned that can be generalized across settings and people

71. Advantages of CBA o Measures functional skills o Comprehensive—across domains o Assessment/intervention in a natural environment o Can be adapted and modified

72. Advantages of CBA  Items can be generalized across settings  Curriculum with steps to follow  Items translate into IFSP goals & objectives  Family involvement  Research base

73. AEPS Item

74. AEPS Goal CRITERION Child performs the following play skills: Catches ball or similar object (3.1) Kicks ball or similar object (3.2) Throws ball or similar object at target (3.3) Rolls ball at target (3.4) Child must score a 2 in Objective 3.1 through Objective 3.4 in order to score a 2 on Goal 3 Goal 3 Catches, kicks, throws, and rolls ball or similar object

75.  Collecting on-going data  Group and individual activities  Assessment activity plans  Making sound decisions based on data

76. Progress Monitoring At the heart of accurate individualization and effective intervention

77. Scoring Option: 2  Child Consistently performs skill as specified in the criterion  Performs the skill independently  Behavior is a functional part of the child ’ s daily routine  Child uses the skill across time, materials, settings, and people

78. Progress Monitoring One child, one area One child, multiple areas Several children, one area Several children, multiple areas

79. Three to Six Data Recording Form

80. Group Data Collection asks 4-wdP playwalkhandrequest Jose Lydia Amie Nina Mateo Arnie

81. Fine Motor Area Fine Motor skills are those that involve the movement and use of the hands. These skills include grasping, releasing, and using the index finger and thumb. 1. Does your child bring both hands to the middle of the body at the same time? (A1) 2. When playing with toys or objects, does your child bang the toys or objects together at midline when one toy or object is in EACH hand? (A2) 3. Does your child hold a hand-size object, such as a block or a small bowl, with either hand using the end of the thumb, the index and the second finger? The object is held by the fingers and is not resting in the palm. (A3) 4. Does your child pick up pea-size objects such as Cheerios or raisins with with either hand using the thumb and the index finger without resting the arm or hand on the table? (A4) date Lin k

82. Things were done very differently on the farm when I was your age, Kenny.

83. Individualizing Instruction  Providing meaningful learning opportunities to all children  On-going monitoring progress towards goals

84. Individualization  Embedded in systems approach  Precise individualized goals  Identified during CBA  Embedded in on-going activities, routines  Progress monitored weekly, monthly…  Data based formative and summative evaluation

87. Thank you! Jane Squires, Ph.D. Center on Human Development University of Oregon jsquires@uoregon.edu

88. Discussion  How do we know when individualization is being done well?  What type of information would be helpful to programs in answering that question?  What types of indicators would be helpful?  Do different definitions of individualization lead to different types of indicators?  What types of research are necessary to move us ahead?

89. Hope you’ve enjoyed the peek inside!