1. A Mixed-Method Evaluation of a Series of Strategies Designed to Improve Student Success in Mathematics and English Developmental Education Courses at Essex County College Dr. Leigh Bello de Castro Essex County College Dr. Charyl Yarbrough The Heldrich Center for Workforce Development at Rutgers

2. The Center for Academic Foundations ECC Incoming Freshman Accuplacer and Companion Tests Results 85% - Math 086 10% - Math 092 5% - Math 100 or higher

3. ECC Center for Academic Foundations Objectives Established in 2007 Target Group: students scoring between 20 -29 Accuplacer 25% pass rate in Math 086 (Target Group) Develop Learning Communities for students in need of Math and English developmental/transitional courses. Provide an array of support services. Counseling Early Intervention Career Counseling Intensive Tutoring Implement new teaching strategies into the curriculum.

4. ECC Developmental Sequence Mth 100Mth 092AFM 083Mth 086 Eng 101Eng 092AFE 083Eng 085 *** AFM and AFE Divisional Midterms and Finals

5. ECC Center for Academic Foundations Staff Full time Coordinator Full time Teacher Advisor Math Full time Teacher Advisor English Adjunct Math and English Faculty Supplemental Instruction Leaders Resource Specialists Math and English Faculty Advisors

6. ECC Center for Academic Foundations Implement new teaching strategies into the curriculum. Customized Book ALEKS My Skills Lab Early Intervention Supplemental Instructional Leaders Block Scheduling Math, English and Computer Course (CSS 101) Tutoring with SI Leader after class 2 class periods of lecture, 1 class period in Lab with ALEKS. Stand Alone Math Sections

7. ESSEX COUNTY COLLEGE INSTITUTIONAL RESEARCH BASIC SKILL TRACKING - CAF and Regular Spring 2009, Spring 2010 and Fall 2010 Cohorts Criteria Cohort Original CohortReturn Following Sem.Success* Rate inPercent WReturn Following Sem.Success* Rate inPercent W Size/ N Whotook MTH 092MTH 092 took ENG 096ENG 096 Passed Course FALL 2010N%N%N%N% original/ passed AFM 083609/ 33527044.3%/ 80.6%10438.5%25.9% MTH 0862202/ 114689640.6%/ 78.2%41746.5%19.0% AFE 083444/ 27823352.5%/ 83.8%15667.0%11.6% ENG 085957/ 70857860.4%/ 81.6%39568.3%11.1% SPRING 2010 AFM 083470/ 2209921.1%/ 45.0%1212.1%41.4% MTH 0861528/ 67034822.8%/ 51.9%12034.5%24.1% AFE 083305/ 1597323.9%/ 45.9%3446.6%23.3% ENG 085426/ 2379225.6%/ 38.8%5458.7%9.8% SPRING 2009 AFM 083302/ 914213.9%/ 42.6%921.4%19.0% MTH 0861400/ 57727819.9%/ 48.1%10437.4%15.5% AFE 083183/ 1124223%/ 37.5%1945.2%23.8% ENG 085321/ 2007322.7%/ 36.5%4561.6%11.0% *Success rate = A,B+,B,C+,C/All Grades The numbers in RED refer to the number or percent of students who successfully completed their first remedial course and would be allowed to take the next level course. For Example: For Fall 2010 AFM 083 - originally there were 609 students registered for this course. Of these, 335 passed the course and would qualify to take The next level mathematics course. In fact, 270, or 44.3 % of the original cohort took MTH 096. This increases to 80.6% of those qualified to take MTH 092. Of the 270 who took MTH 092, 104 or 38.5% successfully completed MTH 092 and 25.9% withdrew.

8. Center for Academic Foundations Math 092 Students were not succeeding in next course Spring 2011 PBI Funds for CAF 09 level Block scheduling Intrusive Support Resource Specialist Mandatory Tutoring Supplemental Instructional Leaders MTH 092 and ENG 096 Curriculum Books Instructional Software Implemented the original CAF model Additional Staffing Evaluation

9. Purpose of Evaluation Describe ECC’s PBI Formula Grant Activities Generate actionable information on the implementation of the PBI Formula Grant initiatives Assess the overall effectiveness of ECC’s PBI Formula Grant initiatives in achieving positive outcomes

10. Key Research Questions 1.Do the ECC project services have the intended positive effects on student outcomes (e.g., higher percentage of students passing mathematics and English courses)? 2.Which of the project services have worked well and which have not been as effective in helping students perform better in subject courses? 3.In what ways can the services be modified to maximize their effectiveness?

11. Evaluation Design Consisted of formative and summative components Utilized qualitative and quantitative methodologies Evaluators from multiple academic disciplines participated in the evaluation. Submit interim reports Focused on understanding the culture and politics of the school as part of the evaluation process Considered the views of all stakeholders

12. Qualitative Data Collection (Spring 2011) MethodTime FrameNumber of Participants/Sessions Course Observations2-3 times per course per semester sixteen observations Instructor InterviewsPre-Post SemesterFive interviews Student Focus Groups Post SemesterFour groups consisting of 5-8 students Self-Efficacy and Subject Matter Confidence Surveys Pre-Post SemesterN = 42-67 Competing Priorities and Barriers Scales Pre Semester

13. Quantitative Data Collection Administrative Data The Heldrich Center collects class level data from the program administrator.

14. Program Description Profile of Students 121 students in five developmental math sections 109 students in five developmental English sections Most students were comfortable with use of computers at baseline 67% of students worked 20 hours per week or less Moderate levels of confidence at the beginning of the semester

15. RQ 1. Do the ECC project services have the intended positive effects on student outcomes? Success rate is the number of students who passed the course divided by the number of students enrolled. Spring 2011

16. RQ 1. Do the ECC project services have the intended positive effects on student outcomes?

17. Spring 2011 The charts below represent the grade distribution for CAF and the Division of Mathematics and Physics. Data reflects a 68% success rate for those students taking the Math course in comparison to their counterparts taking the traditional MTH 092 whose success rate is 40%. In the ENG 096, the CAF courses had 69% success rate as compared to 60% in the traditional ENG 096 courses. MTH 092 Sect.TotalA'sB+BC+CDFIWA - CA - FAllPass*Success AF 13047251101002930 97% AF 22641342080414222664%54% AF 42224406020416182289%73% AF 52965515030422252988%76% AF 6150010108052101520%13% TOTAL12216171510250220178310512279%68% ENG 096 AF 12532547020221232591%84% AF 22602244050912172671%46% AF 423133010040217212381%74% AF 5212441604001721 81% AF 61607201040210141671%63% TOTAL11161816928019015779611180%69% ALL MTH 092194417012914013623280634164078071521194453%42% CAF12216171510250220178310512279%68% less CAF182215411212512620780612163907241416182251%40% ALL ENG 096150416815221214124415363181919171295150471%61% CAF11161816928019015779611180%69% less CAF139316213419613221615344181768401199139370%60% * Pass rate = A,B+,B,C+,C/A,B+, B, C+, C, D, and F * Success Rate = A, B+, B, C+, C/ all grades including W and I.

18. RQ 2. Which of the project services have worked well and which have not been as effective in helping students perform better in subject courses? Scheduling The CAF program successfully grouped the components of the courses to support block scheduling. Each CAF course required the same group of students to meet in three different session types (i.e., classroom, tutorial, and lab). Students in CAF Math were not necessarily in CAF English with the same group of students

19. RQ2. Which of the project services have worked well and which have not been as effective in helping students perform better in subject courses? Supplemental Instructors Typically advanced ECC students or recent graduates attending four year college Varied in experience and skill Established positive rapport with students Administrators matched inexperienced supplemental instructors with lead teachers

20. RQ2. Which of the project services have worked well and which have not been as effective in helping students perform better in subject courses? Inclusion of technology: Students and instructors reported that the math lectures, computer programs, and tutorials aligned rather seamlessly in the MTH-100 course. Students interpreted the grammatical issues presented by the web- based tutorial program and the supplemental instructors (in tutorial sessions) as distinct and discrete topics, unrelated to writing and reading comprehension topics presented by the course instructor. Students were more motivated to complete computer-aided assignments in math than in English

21. RQ 2. Which of the project services have worked well and which have not been as effective in helping students perform better in subject courses? Program Culture The CAF program built a trusting environment in which students felt comfortable seeking help from instructors and supplemental instructors.

24. RQ 2. Which of the project services have worked well and which have not been as effective in helping students perform better in subject courses? Program Culture The students did not develop the sense of community that learning communities are designed to foster and that have been shown to have positive effects on student achievement. CAF instructors did not provide substantive guidance to students related to collaborative learning.

26. RQ 2. Which of the project services have worked well and which have not been as effective in helping students perform better in subject courses? General Teaching Approach Across math and English courses, the instructors implemented procedural teaching techniques. The teaching methods did not encourage students to engage proactively in the “process” of learning.

27. RQ3. In what ways can the services be modified to maximize their effectiveness? Administrators should provide professional development training in “scaffolding” Instructors should emphasize analysis and critical thinking Instructors should ensure that all components of a course are clearly connected Instructors should lead activities that foster interaction and collaboration between students Ensure that instructors are punctual and present at each class session

28. Fall 2011 – Success Rate Update Success rate is the number of students who passed the course divided by the number of students enrolled.

29. Fall 2011 Comparison Data courseNO_ANO_B+NO_BNO_C+NO_CNO_DNO_FNO_I NO_ WA-CA-FA-WPreviousEnrolledSuccess Rate AFE08345438546965109274315429505 431 50562.4% ENG0851141032021081321230258659890950 95769.4% AFM083105548239951314833118375536687 603 69454.6% MTH086250136213122217577011934293816962057 216445.6% ENG096AF202717 19021017100121138 72.5% ENG096-1551231941341899323416879511271299 130761.2% MTH092AF184128151312245789115 49.6% MTH092-106489871134675041934345710281390 140332.9%

30. Spring 2012 – Evaluation Update MethodTime Frame Administrative Data – Success Rates and Withdrawal Rates Post Semester Course ObservationsPre-Post Semester Instructor InterviewsPre-Post Semester Supplemental Instructor InterviewsPost Semester Student Focus GroupsPost Semester Self-Efficacy and Subject Matter Confidence Surveys Pre-Post Semester Competing Priorities and Barriers Scales Pre Semester