1. Evaluation EnAct Campuses Sonoma, Humboldt, Chico, San Francisco, San Jose, Fresno, Bakersfield, Pomona Presentation July 11, 2007 Public Works, Inc. Mikala L. Rahn, PhD Patty O’Driscoll, MPA

2. Who Are We? Public Works, Inc. a nonprofit in Pasadena dedicated to working with communities, government, schools and parents by providing services and resources to educate and inform children, youth and families. Our work is in three areas: Education Reform Workforce Development Intervention/Prevention

3. Evaluation Goals  Measure the success of participating faculty and their students with disabilities overtime as well as compared to students without disabilities on a number of measures  Monitor the project based on federal requirements and offer program improvement strategies  Improve the consistency of measurement across the institutions

4. Research Questions 1) Which of the 14 elements of the Universal Design for Learning (UDL) are most important or are they all equal? 2) How does faculty training in UDL impact student resilience (persistence/graduation)? Are there more appropriate courses or faculty to target to affect resilience?

5. Research Questions 3)Why do faculty participate? 4)How is the use of online and other technologies cost-effective, beneficial, and scalable for rural and remote regions serving the target populations? 5)Did the project reach the target segments of the CSU system with information about research and developments in effective teaching of students with disabilities?

6. Balanced Evaluation Approach Process measures focus on how a program is implemented and the outcome measures focus on the results of the program or intervention. Public Works, Inc. is using both quantitative and qualitative data collection methods

7. Methodology Baseline data from participating institutions; annual data collection Attendance data at workshops and institutes Preliminary Assessment of Faculty Implementation of UDL UDL Training Evaluation Faculty Activity Survey/Assessment Course Artifacts Web user statistics

8. Federal Grant Performance Report (GPR) Objectives…Measures...Targets...Performance Data The difference between the rate at which students with documented disabilities complete courses taught by faculty trained in project activities, and the rate at which other students complete those courses. The percentage of faculty trained in project activities that incorporate elements of training into their classroom teaching.

9. Federal Grant Performance Report (GPR) Objectives…Measures...Targets...Performance Data Collaboration to Ensure Access to a Quality Postsecondary Education Technology to Ensure Access to a Quality Postsecondary Education Dissemination of EnACT Content and Processes

10. Program Improvement How do we better serve students with disabilities? How do we better support faculty? What do we need to know and be able to do?

11. Overall Evaluation Goals Keep the grant by meeting the requirements Learn lessons from each other Support faculty and students

12. Summary of Results Annual Performance Report Goal 1: Collaboration EnACT Leadership Team, Advisory Committee, and campus-based Communities of Practice established CSU Fresno, Bakersfield and Pomona to join current collaborative CSU ATI Initiative partnership at 2007 Institute UDL workshops attended by 194 attendees- target was 100

13. Summary of Results Annual Performance Report Goal 2: Technology to ensure access Enhancement of AIM resource specifications for authoring 13 AIM resources proposed or published to date

14. Summary of Results Annual Performance Report Goal 3: Dissemination of EnACT Enhancements of Web site Dissemination at 11 professional conferences or Webcasts

15. Summary of Results Annual Performance Report Highlights Project Specific Performance Measures: Course Completion Rate Students with disabilities completed at 100% (increase of 7% from baseline) n=9 Students without disabilities completed at 96% n=328 GPA for students with disabilities was 3.22 (increase of.22 from baseline) GPA for students without disabilities was 3.26

16. Summary of Results Annual Performance Report Highlights Project Specific Performance Measures: Faculty participation Faculty participation across campuses totaled 34, above target of 25 73% of faculty agreed they would “more likely make changes to their courses” after exposure to UDL workshops (target was 60%) All participating faculty reported making changes to their courses as a result of participation

17. Summary of Results Faculty Interview Highlights Changes to Courses Changes the result of the combination of training they received, not individual components Changes ranged from revisions to teaching strategies to more technologically complex (multi-media, Web CT) Prompted by desire to make course accessible to all students rather than for accommodations to individual students

18. Summary of Results Faculty Interview Highlights Sample of course changes Students acting out lessons or concepts Improvements to posting on the Internet Revisions to Syllabi using EnACT info Revising PowerPoint to be more accessible Study guides, guided lecture notes, assigning all students to be note takers Using rubrics and adding new assessment strategies

19. Summary of Results Faculty Interview Highlights Faculty Feedback about EnACT Each component received positive feedback: overview from UDL I/II, specific information from Institute, and collaboration in Faculty Learning Community Changes to courses had positive impact on teaching and learning; support from EnACT prepared them to continue to make changes

20. Summary of Results Faculty Interview Highlights Faculty Feedback about EnACT Would like support in implementation of technology Would like continued opportunities for collaboration that occurred during FLC

21. Evaluation Recommendations Continue to examine ways to implement EnACT to impact larger numbers of students with disabilities Use lessons learned from EnACT to implement ATI initiative and expand impact on campus Examine the impact, use and usefulness of AIMs

22. Evaluation Next Steps Orient new campuses to evaluation requirements at Leadership meeting Work with campuses to continue to streamline data collection processes Survey students in Fall 2007 Collect Cohort II implementation data in Spring 2008 (Fall 2007 grades) Collect Cohort III baseline data in Spring 2008 (Fall 2007 grades)

23. Contact Us Mikala L. Rahn, PhD President Public Works, Inc. 90 N. Daisy Avenue Pasadena, CA 91017 (626) 564-9890 (626) 564-0657 mrahn@publicworksinc.org

24. Contact Us Patty O’Driscoll Consultant Public Works, Inc. 1191 Loma Ct. Sonoma, CA 95476 (707) 933-8219 (707) 996-8726 fax patty@publicworksinc.org