1. © 2011 Commonwealth Corporation 1 Review of Pilot 1 (2011) Assessment Navjeet Singh, Vice President of Applied Research & Evaluation nsingh@commcorp.org nsingh@commcorp.org Lawren Bercaw, Research and Evaluation Analyst lbercaw@commcorp.org lbercaw@commcorp.org February 28, 2012 Commonwealth Corporation Evaluation of PHCAST

2. © 2011 Commonwealth Corporation 2 Presentation Purpose & Overview The Grant Evaluation Design Year 1 Pilot Review

3. © 2011 Commonwealth Corporation 3 Purpose and Overview To provide an overview of the evaluation required and planned for the grant To review results from evaluation activities for the first year pilot

4. © 2011 Commonwealth Corporation 4 Purpose of evaluation Purpose of the evaluation is to ensure that for the “Core” Curriculum and Curriculum Design is consistent between the PCA and PCHM training and lead to better outcomes: – skills and knowledge gain for PCAs, PCHMs; – improved demonstration of skills on the job by PCAs and PCHMs, and – improved outcomes for clients and consumers.

5. © 2011 Commonwealth Corporation 5 What is this grant about? MA Council - Home Care Services curriculum BCC/BEC PCA curriculum PCAs-no formal training Personal Care HomeMaker (PCHM) Personal Care Attendant (PCA) Existing TrainingPHCAST Grant Direct care worker competencies and PHCAST core curriculum developed Integrated and Transferrable PHCAST Training MA Council PHCM curriculum BCC PCA Curriculum

6. © 2011 Commonwealth Corporation 6 What is the evaluation design? Is the PHCAST curriculum transferrable to different worker types(PCHM or PCA)? Do PCHM and PCA demonstrate positive learning outcomes? – PCAs and PCHMs understand differences in their roles? – PCAs and PCHMs demonstrate positive learning outcomes? – Do employers report greater worker satisfaction and retention? Concerns about evaluation results –various other factors at play: – PHCAST instructor and delivery of training – Context and environment in which training is delivered – Varying needs of independent consumers and agency based clients

7. © 2011 Commonwealth Corporation 7 Formative Evaluation To provide feedback on the process as well as the curriculum: – From students who attended the pilot Year1 Student reaction survey after they completed the training – From PHCAST instructors using the curriculum Instructor feedback Instructor focus group

8. © 2011 Commonwealth Corporation 8 Student Assessment Results Description Pre-Training Assessment Post- Training Assessment Students (No.) 197188 Written Questions 67 Questions Answered Correctly 5259 Change in Score 10.2% Improvement in Score Between Pre- and Post-Test Skills Assessment Students had to demonstrate selected skills (e.g., hand-washing, lifting, transferring) Most students (80%) were described as skilled in most areas and were not classified as “needing review” in any skill Written Assessment

9. © 2011 Commonwealth Corporation 9 Key Issues Identified Are there questions that those with certain characteristics were more likely to answer incorrectly? Characteristics we checked for: – Home Care Council classes compared to Community College classes – Limited English ability – Education: with less than high School compared to those with HS diploma or more education – Previous experience in health care

10. © 2011 Commonwealth Corporation 10 Results of Formative Evaluation Students reported high levels of satisfaction with PHCAST training and strong interest in working as PCAs/PCHMs Instructors also indicated high satisfaction overall with all materials, especially – More information and material resources – More hands-on and interactive activities Potential areas for improvement suggested by instructors: – Need more time or less material for most modules – More time for practicing skills – “Infection Control” had too much material, and at higher language level – Need more visual aids, graphics, perhaps videos