DEVELOPING & USING INTERMEDIATE MEASURES: ASKING NEW & DIFFERENT QUESTIONS TO SUPPORT STUDENT SUCCESS James Sass, Rio Hondo College Agi Horspool, Fullerton College RP Conference, 4/9/2015

DEVELOPING AND USING INTERMEDIATE MEASURES Overview to Session

Intermediate Measures 101 Introduction to intermediate measures of student success. Techniques for using intermediate measures in research and planning. Group-process approach for integrating intermediate measures at your college. Steps for adapting Scorecard measures for local use.

Focus: Organizational Change at Your College or District Jim Sass Rio Hondo College Research Analyst PhD, Arizona State University, Organizational Communication Working with program evaluation since 1998 Agi Horspool Fullerton College Project Manager, Research PhD, Claremont Graduate University, Organizational Behavior Working with program evaluation since 2005

STUDENT SUCCESS DATA Challenges and Opportunities

Student Outcomes Scenarios Volunteers: – Please read aloud each of the scenarios about the challenges of measuring student outcomes in California community colleges.

Student Outcomes Scenarios Introduce yourself to others at your table For a few minutes, discuss the scenarios you listened to: – In what ways did these scenarios get you thinking about the measures you currently/typically use on your campus? Select an individual from your table group to report out

Gap between Course-Level and Completion Measures Course-Level Measures Retention & Success Little change across time Not addressing student progress Completion Measures Degrees, Certificates & Transfer Not about current students Not addressing students who did not complete

Solution: Intermediate Measures Measures of progress toward completion – Completing course sequences – Achieving specific milestones Progress by current students – One-year and three-semester cohorts – Three-year cohorts Support campus decision making and evaluation of initiatives

INTERMEDIATE MEASURES OF STUDENT SUCCESS A Brief History

Development of Intermediate Measures: Publications Reaching Consensus on Common Indicators (Ewell, 2006) Using Longitudinal Data to Increase Community College Student Success (CRCC, 2008) Taking the Next Step (IHELP, 2010) – Steps to Success (IHELP, 2009) – Advancing by Degrees (IHELP, 2010)

Development of Intermediate Measures: Themes “Information about student progression... is an important policy tool” (Ewell, 2006). There are significant markers of student progression toward completion. Tracking these markers and disaggregating by demographic and enrollment groups can be useful for planning and improvement.

Development of Intermediate Measures: Terms Milestones: “Necessary intermediate educational achievements” – Persisting for two or three semesters – Earning one year of college-level credits – Completing the General Education curriculum Success Indicators: “Academic behaviors that predict success” – Completing a college success course – Attempting/completing courses within a timeline – Earning summer units

Intermediate Measures in the California Community College System Student Success Task Force (2012) Student Success Scorecard (2013) – Completion Outcomes Degree/Transfer (SPAR) Career Technical Education – Momentum Points Basic Skills (“Remedial”) Persistence 30 Units

INTERMEDIATE MEASURES Implications for Research and Planning

Using Intermediate Measures: Some Examples Rio Hondo College – Added Scorecard measures to Institution-Set Standards – Focus on 3-year cohorts Creative Ideas from Anonymous Colleges Responding to Survey – At 30 units, survey on ILOs and non-cognitive factors. – Awarding “badges” to students achieving milestones

Intermediate Measures in Research & Planning Some colleges incorporating Scorecard measures as is. Some confusion on nature of intermediate measures. Some good examples of tracking intermediate measures with recent cohorts. – Persistence on yearly basis to inform strategic planning. – Persistence and 30 units for first-time college students. – Gateway course performance and 15-, 30-, and 45-unit thresholds. – Persistence, basic-skills sequence, and 30 units on an annual basis and strategizing where not improving.

Potential Uses for Intermediate Measures Conversations about goals and objectives. Setting institutional standards. Developing logic models for programs and initiatives. Evaluating programs and initiatives.

INTERMEDIATE MEASURES AT YOUR COLLEGE OR DISTRICT A Group-Process Activity

Local Intermediate Measures For this activity, work in small groups at your tables to discuss the two questions on your “Activity #2: Group Process” worksheet. – Take notes at your table so we can collect and share with participants after the conference. – Record your notes on the flipchart. These will be posted so others can see after the session. – Designate a speaker who will report out.

Debrief: Local Intermediate Measures From your “Activity #2: Group Process” discussion, report out on the following question: – How might you use a similar group process to open conversation at your campus or district about important and/or relevant intermediate measures?

DEVELOPING INTERMEDIATE MEASURES Adapting Scorecard Metrics for Local Use

Scorecard Momentum Points Persistence 30 Units Basic Skills Sequence Completion – English – ESL/ENLA – Math – Reading

Persistence and 30 Units Use first-time students. Window: fall semester or full academic year. Cohort entry criteria: – Attempting 9 units in first term. – Attempting 6 units, including math or English, in first year. – Earning 3 units and attempting math or English in first year. Outcomes: Use Scorecard criteria.

Basic Skills Sequence Completion Follow Scorecard procedures for entry to cohorts. Window: fall semester or full academic year. Outcomes: Use Scorecard criteria For Reading outcome, use transfer-level Reading or English course.

Time to Outcome Achievement Set cohort lengths appropriate to the outcome. – Persistence takes 3 semesters. – 30 units can be reasonable after 2 years (15 units after 1 year) – Basic skills: 1 to 5 years Consider using 3 years’ progress as a standard for planning.

CONCLUSION

Measurement Issues Traditional measures of student success can be of limited usefulness. The Scorecard’s use of intermediate measures is a step ahead, but 6-year cohorts are not practical for decision making. Intermediate measures for recent cohorts have great potential to support planning and student success.

Practical Implications Colleges can and should develop their own intermediate measures. Cohorts can be created to meet the needs of the college. Group process is vital to developing and integrating intermediate measures. Scorecard measures can be adapted for local use.

QUESTIONS? James Sass Agi Horspool

Link to Published Resources on Intermediate Measures