Presentation on theme: "Accelerating Opportunity Evaluation Planning the Evaluation with the Accelerating Opportunity States February 10, 2012 11:30 a.m. – 1:00 p.m."— Presentation transcript:
Accelerating Opportunity Evaluation Planning the Evaluation with the Accelerating Opportunity States February 10, :30 a.m. – 1:00 p.m.
Webinar Agenda Introduction of the Evaluation Team Goals of the Evaluation Evaluation Design Plans for the Evaluation Q&A with the Evaluation Team
AO Evaluation Team Randall Wilson, JFF, Evaluation Director Robert Lerman, Urban Institute, Principal Investigator and Cost Analysis Lead Lauren Eyster, Urban Institute, Project Director and Implementation Analysis Co-Lead Maureen Conway, Aspen Institute, Implementation Analysis Co-Lead Burt Barnow, George Washington University, Impact Analysis Lead
Goals of the AO Evaluation To generate evidence for state and federal policymakers, college administrators, funders, and other stakeholders about: 1.The process of implementing integrated college and career pathway designs and taking these designs to scale 2.Their impact for ABE and ESL students in college and in the labor market 3.Their cost effectiveness and financial sustainability
AO Evaluation Design Impact on AO Students Cost Effectiveness and Financial Sustainability Implementation and Scalability
Implementation Analysis Questions How did states and community colleges establish integrated pathway designs? What shifts in the culture (attitudes and behavior) occurred within states and colleges concerning ABE/ESL students? How did states and community colleges move toward scale and the sustainability of pathways through policy changes, innovative financing, the use of data to support continuous improvement, and other means?
Site Visits to States and Community Colleges Twice during the grant period (Fall 2012 and Spring 2014) Three-day visits with two evaluation team members Two community colleges will be visited during each site visit Selection of the colleges will be determined with the state team
Community College Surveys Used to collect systematic data on implementation, financing, and sustainability across ALL community colleges Fielded once after the end of the first year of the grant (December 2012) and once at the end of the grant period (October 2014) Web-based survey but will be available in paper if requested Will work with the states in fielding the survey – reaching the right contacts at each college and following up with non-responders
Quarterly Check-in Calls To keep updated on changes to AO program design and implementation To provide states with an opportunity to ask questions about the evaluation activities Will occur March, June, September, and December each year during the grant period
Impact Analysis Questions What impacts do integrated career pathway designs have on student progress and outcomes in college and in the labor market? How do outcomes for participating students change relative to comparison groups of students similar to the population in AO, but not participating in the program?
Requirements for the Impact Analysis Ensure that AO education and support services differ significantly from services received by those who do not participate Have a well-articulated selection process, which is consistently applied by colleges Identify an appropriate comparison group whose outcomes will be compared to those of AO participants Have enough participants and comparison group members to be able to detect statistically significant differences in outcomes
Planned Quasi-Experimental Designs Regression discontinuity design Compares the treatment group to non- participating applicants to the AO program who are close to a test cutoff score used to determine program eligibility Propensity score matching Uses a large pool of similarly prepared and motivated ABE/ESL students in the state to compare to the treatment group on outcomes of interest
Data for Impact Analysis Data need to be of high quality and collected for both treatment and comparison group members Data will be submitted twice per year to the Urban Institute through secure file transfer protocols Test and retest scores on ABE/ESL tests are necessary for developing comparison groups Data collected by states will include student characteristics, services received, educational progress, and labor market outcomes
Cost Analysis Questions What are the benefits and costs of implementing and scaling up integrated pathway designs to states, community colleges, and students? What methods were used to finance the pathways, and how financially sustainable is the initiative in the state? The key issue is how much more resources must be devoted to the AO model than do standard ABE/ESL programs and who bears any extra costs
Data for Cost Analysis Template will be provided for documenting state financing plans/budgets and college costs. College cost data will also be collected through the implementation surveys. Student cost and foregone earnings data be collected through semi-annual data reporting. Data on financial sustainability issues will be collected in surveys and during site visits.
One-Day Planning Visits to States (now scheduled for March) To learn more about states’ program design, including selection procedures, test scores, and career pathways To gain an understanding of data availability to help the evaluators develop analysis plans for the evaluation To work closely with the states in identifying comparison groups To continue developing the states’ understanding of the evaluation and its requirements
Who Should Participate in Planning Visits? From states: State team leads State staff responsible for data submission From one or two community colleges: College representatives who are knowledgeable about pathway design, recruitment and selection processes, and student data Burt or Bob will each visit two states and will be accompanied by JFF or other evaluation team staff
Main Topics for Visits Continued discussion on the evaluation design and help the states and community colleges meet evaluation requirements AO program design, specifically: Recruitment and intake Eligibility requirements Educational milestones for students Contrast with services received by other adult education students Availability of data Student characteristics at intake, participation in AO pathways, educational progress, and labor market outcomes Data confidentiality and security and FERPA Costs and benefits to students, community colleges, and states
Tools for Collecting Student Data Intake form (draft provided for review and comment) Data template for semi-annual reporting (draft provided for review and comment) Template for collecting state and college cost data will be provided by spring 2012
Overall Evaluation Activity Timeline Evaluation ActivityDates Initial Evaluation Planning Visits (One-Day Visits) March 2012 First Full Site Visit (Three-Day Visits)September-October 2012 First Community College SurveyDecember 2012 Second Full Site Visit (Three-Day Visits)April-May 2014 Second Community College SurveyOctober 2014 Ongoing Activities - Quarterly Check-in Calls - Semi-Annual Data Submissions March, June, September, and December during the grant period June and January