1. Andrew LaManque, Foothill-De Anza CCD Alice VanOmmeren, CCC Chancellor's Office Ryan Fuller, CCC Chancellor's Office Daylene Meuschke, College of the Canyons Barbara McNeice-Stallard, Mt. San Antonio College Terrence Willett, RP Group RP Regional Research Groups – Spring Convening

2. 2 Measuring Our Success, RP Conference, April 2013 A Brief Overview - RP Report on Prerequisite Validation - Student Success Scorecard - Wage Tracker - Professional Development Opportunities - IRB Issues - RP Group Update

3. Update on the Scorecard (formerly know as ARCC) and WageMart Alice van Ommeren Ryan Fuller

4. Developments In 2012, the Scorecard Advisory Group and TRIS developed metrics and reporting format In January 2013, researchers had 45-days to review report format and Scorecard metrics April 9, the public release of the Scorecard Last week, the Scorecard on the DataMart System of the State this summer and the advisory group reconvenes in the fall

5. College Responsibilities ARCC- Alias List for Updates and Revisions – Two changes already, ESL and Math Remedial Rate Board of Trustee Interactions (due March 2014) Use the 45-day review period in January for questions and data resubmission (data on demand) Check other Scorecard resources on webpage – New peer groups and % of missing SSN report Use scorecard@cccco.edu or arcc@cccco.edu for questions and feedbackscorecard@cccco.eduarcc@cccco.edu

6. Most FAQs/Issues of Concern Why do Prepared students have lower rates than Unprepared (persistence)? What are the N’s behind the metrics? – To disclose, or not? When is the cell size too small? Revise Data on Demand files to include flags Why are some of my remedial courses not qualifying for the cohort? – TOP code and CB04

7. Remedial ESL Rate (cohort) Cohort is first attempt of an ESL course in any levels below transfer: CB03 COURSE-TOP-CODE = 4930.84, 4930.85, 4930.86, 4930.87 SB01 STUDENT IDENTIFIER STATUS = S (only students with SSN) CB21 COURSE-PRIOR-TO-COLLEGE-LEVEL = A, B, C, D, E, or F CB04 COURSE-CREDIT-STATUS = C

8. Remedial Math Rate (cohort) Cohort is first attempt of a Math course in two to four levels below transfer: CB03 COURSE-TOP-CODE = 1701* SB01 STUDENT IDENTIFIER STATUS = S (only students with SSN) CB21 COURSE-PRIOR-TO-COLLEGE-LEVEL = B, C, or D CB04 COURSE-CREDIT-STATUS = C (not D)

9. We are working on… Revisions for next years’ Scorecard – Advisory Group in the fall, student-counselor ratio Guidelines for Measuring the Achievement Gap using the Scorecard metrics ISP and OOS Report (modifying methodology) Exploring uses of the Scorecard metrics for other reporting requirements WageTracker or WageMart (release May 29 th )

10. New Wage Tracker Tools on Datamart

11. Methodology Two different methodologies – Systemwide wages for award completers – By campus wages for award completers Award completer data joined with EDDUI wage data. Both wage methodologies break out by award discipline (6 digit TOP code) and award type.

12. Systemwide Award Completer Wages Award completers in a given academic year, exclude: – Enrolled anywhere in higher education after award – 21 and under at time of award Join with EDDUI (Unemployment Insurance) wage file Sum all quarters of wages to student annual wage All wages inflation adjusted using California CPI-U Calculate median wages by discipline and award type Median calculated for students with wages 2 years before award, 2 years after and 5 years after award

13. By College, Award Completer Wages Same methodology as statewide wages, except: – Series of 8 years of award completers (to increase the number of students in the cohorts) – Median calculated for students in cohort with wages 3 years after award Calculate median wages by “college,” discipline and award type

14. Systemwide vs. By College Methodology Wages 03-0404-0505-0606-07 07-08 08-0909-10 By College Awards 00-0101-0202-0303-0404-05 05-06 06-0707-08 10-11 05-06 Systemwide 03-04 04-05 05-0606-07 07-08 08-0909-10 10-11 Awards Wages

15. Data Caveats EDDUI Data contains wages of occupations covered by CA Unemployment Insurance Excludes military, federal government, self- employed, out of state and unemployed No way of determining part time or full time wages. Hours worked not in the EDDUI data. Only students with SSNs Wages are adjusted for inflation to current dollars using the California CPI-U

16. Analysis Caveats Data reflects only the wage outcomes of award earners who remained in California Region/locale significantly impacts wages (i.e. cost of living), caution in comparing earnings across the state. Wages are not necessarily from employment associated with a particular award discipline, no information on job Short term wage outcomes should not be a sole measure of institutional effectiveness or program quality Other factors besides earnings motivate students to earn an award in a specific discipline

19. Accessing the new tools Query modules using both the Systemwide and ‘By College’ methods will be available in datamart (System Wage Tracker & College Wage Tracker) Release on May 29 th Median wages by program code will be available for colleges in Data on Demand.

20. Daylene Meuschke, College of the Canyons Barbara McNeice-Stallard, Mt. San Antonio College Proposed RP Professional Development Plan 2013-14

21. 21 Measuring Our Success, RP Conference, April 2013 A Brief Overview The following slides present the proposed professional development plan for the RP Group in 2013-14. The first slide presents the an overview of the plan. Slides 3-5 provide detail for each segment of the plan. Slides 6-8 identify possible webinar topics for each of the three big training areas.

22. ConferencesWebinars Regional Research Groups WorkshopsCoaches Working Groups RecognitionDissemination RP Group Professional Development

23. SSS (October) RP (April) Conferences 30 minutes Monthly Moderated 3 main areas: planning, assessment/evaluation & resources/tools Webinars Propose meeting 3x / year (fall, spring, summer) Devote one hour to training via webinar with skilled facilitators at each RRG to lead activities & discussion Allot 30 minutes for updates from CCCCO, RP, & National issues Include agenda item on communicating and helping campuses use data to inform decision-making processes* Regional Research Groups

24. Leading from the Middle “Research Connects” (in-person workshops – Fee TBD) Workshops/Other In-Person Training Can be used to disseminate Summer Institute (S.I.) Content Webinars can be used to supplement S.I. content and facilitate connection with S.I. leaders Fee for S.I. course content TBD Pair up researchers with a “coach” either as part of S.I. or outside of participation in S.I. (upon request) Coaching Program Reporting Efficiency Planning SLO Assessment Working Groups

25. RP Awards Power Awards Leadership* Recognition RP Perspectives Regional Research Groups Listservs (RP, SLO Assessment, Regional Research Groups) RP website Dissemination

26. Planning Big Picture (webinar) Program Review & ACCJC Standards Strategic Plans/Educ Facility Master Plans Alternative Approaches to Facilitate Engagement* Lessons Learned

27. Assessment / Evaluation Pre-Req Validation Alternative Placement Segmentation Analysis Disproportionate Impact Student Equity & DI Scorecard (additional analyses)

28. Resources / Tools Statistical Techniques Wage Tracker Analyses Needs Assessments (Labor Market analyses) Hold for GIS – look at AEA Cal-PASS/PLUSVideos*IRB

29. Andrew LaManque, Ph.D. President, RP Group Regional Research Meetings RP Group Prerequisite Guide Overview RP Regional Meeting, May 3, 2013

30. This project was made possible through the efforts of many dedicated people within the California Community College system! Special thanks goes to:  Anne Danenberg, Sacramento City College  Lan Hao, Citrus College  Andrew LaManque, Foothill De Anza Community College District  Katherine McLain, Cosumnes River College  Daylene Meuschke, College of the Canyons  Ken Meehan, formerly Fullerton College  Steven Spurling, San Francisco City College  Keith Wurtz, Crafton Hills  Lijuan Zhai, Fresno City College Thanks! 30

31. Pre-implementation  Are students highly unlikely to succeed in the target course without the knowledge, skills, and abilities learned in the prerequisite course?  Do student success rates in the target course vary for those students with or without the pre-requisite?  Is the projected impact on course availability, enrollment and success different depending on student demographics? Post-implementation  Did the student success rates of a target course increase after the implementation of a pre-requisite?  Is the actual impact on course enrollment and success different depending on student demographics? Using Research Questions to Frame the Discussion 31

32. The Prerequisite Research Framework (pre) 32

33. The Prerequisite Research Framework (post) 33

34. Will be seeking input in the coming weeks:  Institutional Researchers  Faculty  Chancellor’s Office  Instructional and Student Service Officers Will be reviewing the Disproportionate Impact section 34 Next Steps

35. Questions? Please submit any additional questions to your webinar moderator and they will get back to you with answers. Find out more: Andrew LaManque, Foothill-De Anza CCD lamanqueandrew@fhda.edu 35

36. Terrence Willett, The RP Group For the latest on our projects visit: http://www.rpgroup.org/projects/current RP Group Update