21 st Century Skills: What Can Be Measured? Beverly Farr, Director of Evaluation MPR Associates

A Tale of Two Projects: Evaluation of 1) ConnectEd Demonstration Sites 2) National Ford PAS Network

3 MPR and ConnectEd Evaluation of the ConnectEd Network of Demonstration Sites (n=16) Provide evidence of effectiveness on a set of student outcome indicators Showcase well-designed examples of multiple pathways or pathway components

4 MPR and ConnectEd Multiple pathways WComprehensive, coherent strategy designed to re-engage high school students in serious learning WPrepare students for postsecondary education and career WConnect academics to real-world applications

5 MPR and ConnectEd Multiple Pathways WLead to the full range of postsecondary opportunities. WImprove student achievement. Designed to produce higher academic and technical achievement and higher rates of high school completion and to support the development of students’ critical- thinking and problem-solving, communication, and collaboration skills.

6 MPR and ConnectEd Evaluation Challenge Gary Hoachlander (2005) points out, “Standardized tests... do not measure students’ diagnostic abilities, capacities for bringing interdisciplinary knowledge to bear on complex problems, understanding of systems, or facility in applying abstract knowledge and academic skills to authentic, real-life situations” (p. 38).

7 MPR and ConnectEd Evaluation Components Collection of ( ) student data Participant data (number, grade level, demographics) CAHSEE and CST scores/proficiency levels Attendance, continuation, PSE readiness and plans Site visits during fall/winter of 2008 Interviews (school and district staff, students, partners) Classroom observations Review of documents

Student Outcomes: Results on Indicator Data  DMD  Palmdale  HPHS

9 MPR and ConnectEd Student Outcomes Attendance, promotion, and continuation rates 10 th -grade CAHSEE pass rates California Standards Tests (CSTs) Student attitudes, behaviors, and career awareness PSE readiness and intentions

10 MPR and ConnectEd Graduation and a-g course fulfillment

11 MPR and ConnectEd Science Achievement

12 MPR and ConnectEd School of Digital Media & Design The School of Digital Media and Design focuses on creating an environment where students interested in multimedia, visual arts and production will master high-level communication, technological and critical thinking skills. Classes emphasize multimedia and collaborative projects to prepare students for lifelong learning, a fulfilling career, and a successful future.

13 MPR and ConnectEd Math Achievement

14 MPR and ConnectEd English Achievement

Other Reported Outcomes  DMD  Palmdale  HPHS

16 MPR and ConnectEd Qualitative Findings Student Attitudes Behaviors and Skills Awareness of Career Options Workplace Readiness College Readiness

Purpose of the Evaluation of Ford PAS: Conduct rigorous study on the effects of the program on student academic achievement, attitudes, and 21st century skills

18 MPR and ConnectEd Research Questions What impact does the program have on students in terms of: a) cognitive performance and academic/business knowledge and skills; b) affective student factors such as attitudes, behavior, future career and vocational aspirations; c) future educational pursuits and success in those settings?

19 MPR and ConnectEd Research Questions In what ways has the program helped students connect classroom learning with real-world experiences? To what extent are students able to apply their academic/business knowledge and skills to real-world settings?

Results of Implementation Survey

21 MPR and ConnectEd Reports by Teachers and Site Coordinators on Student Outcomes

Planning for Measurement of Student Outcomes in Ford PAS Study

23 MPR and ConnectEd Planning Analyses WElaboration of Constructs WCross-walk of Ford PAS modules and core skills WReview of Identification of Essential Knowledge for 21 st Century by Partnership WMatrix of Available Assessments in Use or in Development

Challenges and Next Steps