Presentation is loading. Please wait.

Presentation is loading. Please wait.

REL Midwest Using Evaluation Methods to Monitor Implementation and Impact of Competency Based Pathways Jim Lindsay, Ph.D. Senior Researcher December, 2011.

Similar presentations


Presentation on theme: "REL Midwest Using Evaluation Methods to Monitor Implementation and Impact of Competency Based Pathways Jim Lindsay, Ph.D. Senior Researcher December, 2011."— Presentation transcript:

1 REL Midwest Using Evaluation Methods to Monitor Implementation and Impact of Competency Based Pathways Jim Lindsay, Ph.D. Senior Researcher December, 2011

2 2 One More Thing to Plan: Evaluation  How well is your competency-based pathway model running? Fidelity of implementation  What parts of model need additional attention? Formative information  Are competency-based pathways having an impact on student learning? Outcomes data + research design/staged roll-out help show impact.

3 3 Presentation on Evaluation Methods  Using data to assess fidelity and identify parts of pathway model that need improvement.  Thinking about impacts: Developing a Theory of Action.  Examining outcomes and impacts: Data and design.

4 4 Fidelity of Implementation  Determining the essential components of competency-based pathway. Examples: Community and faculty buy-in Teacher attendance at PD sessions Specification of learning objectives Assessments for competencies Data system: immediate feedback to students Teacher assignment Students having difficulty receive more attention

5 5 Fidelity of Implementation  Examples at school/district level: Administrator/team reviews students’ progress through objectives Administrator/team meets with teachers to discuss student progression Students getting awarded credits.

6 6 Assessing Fidelity  For each essential component, identify a data element that reflects whether component is working For that data element, create a threshold that signifies “adequate” implementation.  Data element may be as simple as “presence” /”absence” of something ComponentData ElementThreshold ComponentData ElementThreshold Buy-in--community ComponentData ElementThreshold Buy-in--communitySchool board vote66% ComponentData ElementThreshold Buy-in--communitySchool board vote66% Buy-in--facultyPercent of teachers favoring adoption80%

7 7 Assessing Fidelity  For each essential component, identify a data element that reflects whether component is working For that data element, create a threshold that signifies “adequate” implementation.  Data element may just be as simple as “presence” /”absence” of something ComponentData ElementThresholdData Buy-in--communityApproval by board66% Buy-in--facultyPercent of teachers favoring adoption 80% ComponentData ElementThresholdData Fidelity Scores Buy-in--communityApproval by board66% Buy-in--facultyPercent of teachers favoring adoption 80% ComponentData ElementThresholdData Fidelity Scores Buy-in--communityApproval by board66%80% Buy-in--facultyPercent of teachers favoring adoption 80% ComponentData ElementThresholdData Fidelity Scores Buy-in--communityApproval by board66%80%1 Buy-in--facultyPercent of teachers favoring adoption 80% ComponentData ElementThresholdData Fidelity Scores Buy-in--communityApproval by board66%80%1 Buy-in--facultyPercent of teachers favoring adoption 80%83%1

8 8 Assessing Fidelity ComponentData ElementThresholdData Fidelity Scores Buy-in--communityApproval by board 66%80%1 Buy-in--faculty% of teachers voting “yes” 80%83%1 Teacher PDPercent of teachers attending all 3 PD sessions 90%85%0 ComponentData ElementThresholdData Fidelity Scores Buy-in--communityApproval by board 66%80%1 Buy-in--faculty% of teachers voting “yes” 80%83%1 Teacher PDPercent of teachers attending all 3 sessions 90%85%0 Data systemsSystem recording students’ scores on competencies No or YesYes1 Data systemsProduces automated reports for teachers No or YesYes1 ComponentData ElementThresholdData Fidelity Scores Buy-in--communityApproval by board 66%80%1 Buy-in--faculty% of teachers voting “yes” 80%83%1 Teacher PDPercent of teachers attending all 3 sessions 90%85%0 Data systemsSystem recording students’ scores on competencies No or YesYes1 Data systemsProduces automated reports for teachers No or YesYes1 Specification of learning objectives for course Standards-based learning objectives identified and sequenced No or YesYes1

9 9 Assessing Fidelity ComponentData ElementThresholdData Fidelity Scores Buy-in--communityApproval by board66%80%1 Buy-in--faculty% of teachers voting “yes”80%83%1 Teacher PDPercent of teachers attending all 3 sessions90%85%0 Data systems System recording students’ scores on competencies No or YesYes1 Data systemsProduces automated reports for teachersNo or YesYes1 Specification of learning objectives for course Standards-based learning objectives identified and sequenced No or YesYes1 Assessments for learning objectives Each learning objective has valid/reliable assessment No or YesYes1 Teacher assignment Certified/endorsed teachers score assessments 100% certified with endorsement 85%0 Credit only with completion of competencies Record of students’ competencies completed and course credit 100%97%0 Students having difficulty get more attention Observations: students having difficulty get more attention from teachers No or YesYes1 Fidelity = sum of scores/number7/10 (70%)

10 10 Assessing Fidelity ComponentData ElementThresholdData Fidelity Scores Buy-in--communityApproval by board66%80%1 Buy-in--faculty% of teachers voting “yes”80%83%1 Teacher PDPercent of teachers attending all 3 sessions90%85%0 Data systems System recording students’ scores on competencies No or YesYes1 Data systemsProduces automated reports for teachersNo or YesYes1 Specification of learning objectives for course Standards-based learning objectives identified and sequenced No or YesYes1 Assessments for learning objectives Each learning objective has valid/reliable assessment No or YesYes1 Teacher assignment Certified/endorsed teachers score assessments 100% certified with endorsement 85%0 Credit only with completion of competencies Record of students’ competencies completed and course credit 100%97%0 Students having difficulty get more attention Observations: students having difficulty get more attention from teachers No or YesYes1 Components needing attention (70%)

11 11 Thinking Impact: Theory of Action  Figure that shows anticipated benefits of implementing pathways. Immediate outcomes  Effects that occur while students participate in course/pathway or shortly thereafter Longer-term outcomes  “ultimate” outcomes--tend to be ones of most interest to stakeholders  Figure helps planning team think about data that will indicate impact of pathways.

12 12 Thinking Impact: Theory of Action Competency-Based Pathways - Specification of sequence of learning objectives for courses - Students direct pace of completion of learning objectives - Differential support to students based on their need Fidelity Information Utilization ProgramImmediate Outcomes Student motivation Student Attitudes Toward Subject Time to complete course Number of students re-enrolling in CB pathways Student Engagement Learning Course Content Scores on EOC Exams Graduation rate School Completion Credit accumulation Long-Term Outcomes SAT/ACT- scores Other standardized achievement scores Broader Learning % taking SAT/ACT Enrollment in AP courses % of students applying to college Post-Secondary Enrollment

13 13 Examining Outcomes and Impact  Impact: Is initiative effective? Ability to make strong conclusions about whether initiative “works”  Impact: Is initiative effective?  Clear Comparison Group!  At school, classroom level, or student level  Preference: Random assignment to conditions,  Pre-post not good enough  other things may contribute to pre-post differences. Easy comparison group: Schools/classrooms that implement 1-2 years later.  Schools that implement first “work out the kinks”

14 14 Example Theory of Action Competency-Based Pathways - Specification of sequence of learning objectives for courses - Students direct pace of completion of learning objectives - Differential support to students based on their need Fidelity Information Utilization ProgramImmediate Outcomes Learning Course Content Scores on EOC Exams Graduation rate School Completion Long-Term Outcomes SAT/ACT- scores Broader Learning Enrollment in AP courses % of students applying to college Post-Secondary Enrollment Student motivation Student Attitudes Toward Subject Time to complete course Number of students re-enrolling in CB pathways Student Engagement

15 15 Illustrating Impact **

16 16 Advantages of Considering Evaluation Now District A Cannot answer “how is it working?” question. Unable to find particular components that aren’t working New superintendent- “is CBE effective”?  Unable to answer District B Have data available to show stakeholders how it is working. Can determine which components need refinement. Can provide strong evidence of whether CBE is effective.

17 17 Jim Lindsay, Ph.D. P: > F: REL Midwest 1120 East Diehl Road, Suite 200 Naperville, IL General Information:


Download ppt "REL Midwest Using Evaluation Methods to Monitor Implementation and Impact of Competency Based Pathways Jim Lindsay, Ph.D. Senior Researcher December, 2011."

Similar presentations


Ads by Google