An Integrated Approach to Performance Measurement for the CIC Settlement Program Presented to the Atlantic Region Association of Immigrant Serving Agencies (ARAISA) Conference September 22, /07/2010 4:11 PM

Objectives of the Presentation The objective of the presentation is: 1.To provide an overview of an integrated performance measurement framework for the CIC Settlement Program. 2.To report on the work of the Performance Measurement Advisory Group (PMAG) 3.To gather your comments and suggestions about the use of various data collection methods for immediate outcomes 2

3 Vision Together, we all make a difference in the settlement and integration of newcomers We can measure and assess the impact of our actions Acting upon this information permits on-going improvement of services

Monitoring Policy/Program Development Performance Measurement Research & Evaluation Reporting & Planning Program level Outputs Outcomes Client feedback Public opinion polls on performance The Program-Policy Continuum Departmental planning & performance reports Strategic Review Best Practices Objectives Planned results Improvements Design & delivery Implementation Activities Financials Operations Systems Broader context Impact, efficiency, effectiveness & alternatives Periodic evaluations Evidence Based Decision-Making Performance Measurement has a unique but complementary role in program-policy development 4

5 What is Performance Measurement? It is the systematic gathering of evidence about the outputs (products and services) and outcomes (results) of a program or policy It is a cyclical process of measurement, collection, analysis and reporting of information to assess the achievements of a program or policy and facilitate informed decision-making It answers the key questions: What do we want to accomplish over the life of the program? How will we know how well the program is doing? What evidence is there to support the needs to be addressed by the program? What needs to be undertaken to improve the program?

Performance Measurement On-going collection of information on program performance (outputs & outcomes) to guide decision-making and enable reporting on program results Settlement program Logic Model with expected Outcomes Terms and conditions Funding Settlement Services Based on priorities for services and programming and directives to manage contribution agreements (left side of the logic model) Outputs Immediate Outcomes Intermediate Outcomes Ultimate Outcomes Monitoring Financial & operational activities exercised by settlement officers over contribution agreements and SPOs. Accountability, Risk and Audit Framework (ARAF) 6 Audit To determine whether the Department’s strategy and practices related to risk management, control, and governance processes are adequate. Evaluation Systematic collection and analysis of the performance of a program to make judgements about relevance, progress or success and cost-effectiveness to inform future programming decisions about design and implementation. A visual representation of the interaction between elements of the June 2008 ARAF that relate to performance measurement of the CIC Settlement Program.

7 Outputs Outputs are direct products or services under a policy, program or initiative, delivered to a target group or population, which serve to achieve outcomes. Responsibility for gathering data: Service provider organisations (SPOs) are responsible for providing program outputs to CIC Data is captured through these sources: iCAMS (Immigration-Contribution Accountability Measurement System): collects mostly quantitative data about settlement services provided ARS (Automated Reservation System): collects data from Ontario service providers about language services, which is uploaded to iCAMS ELT Data Collection Tool: collects data about ELT program Annual Reports from BC, Manitoba & Quebec Other methodology that could be used: SPO-initiated client feedback & other data collection strategies Website hits & publications produced and distributed

Immediate Outcomes represent the immediate results achieved, are often observable during or directly following the completion of an activity, and are directly attributable to outputs. 8 Immediate Outcomes Responsibility for gathering the data: Responsibility falls mostly with SPOs, as immediate outcomes relate to projects Data is captured through these sources: Narrative Reporting: to report on project outcomes for monitoring and performance measurement that were indicated in Schedule 1 of Contribution Agreements: used to specify project results and identify links to program results ELT Client Survey: assess the results of ELT programming Annual Reports from BC, Manitoba & Quebec Other methods that could be used: SPO-initiated client feedback & other data collection strategies Client feedback forms (distributed at time of service delivery) CIC-initiated client feedback survey(s) General client survey (telephone, online) Exit Survey Exit Testing (of knowledge and skills)

Intermediate Outcomes are expected to logically occur once one or more immediate outcomes have been achieved. Their impacts extend beyond program participants. 9 Intermediate Outcomes Responsibility for gathering the data: CIC is responsible for collecting data and measuring intermediate outcomes Data is captured through these sources: ELT Client Survey Other methods that could be used: Client Surveys: gather information from clients about intermediate outcomes as a result of receiving services POR (Public Opinion Research): broad-based survey of clients and general public Follow-up surveys Focus Testing

Ultimate Outcomes are the highest-level outcomes that can reasonably be attributed to a policy, program or initiative in a casual manner, and is the consequence of achieving one or more intermediate outcomes. 10 Ultimate Outcomes Responsibility for gathering the data: CIC is responsible for gathering data from existing sources Data is captured through these sources: Existing data sources: Statistics Canada data, Longitudinal Studies, Surveys, etc. Purpose: To measure long-term outcomes of newcomers to Canada Challenges: Difficult to attribute ultimate outcomes to the Settlement Program. Ultimate outcomes should be used as a benchmark

The PMAG was formed in February 2010 at the SIJPPC (Settlement and Integration Joint Policy and Program Council) meeting Mandate: To provide CIC with advice on the development of its performance measurement strategy Membership includes: Service provider representatives from across Canada and CIC staff (Region and NHQ) 11 Show the impact of our actions on clients’ settlement and integration Performance Measurement Advisory Group (PMAG) Note:This group may also be called upon to advise on the development and implementation of data capturing methodology Deliverables and timelines: Recommendation of indicators to assess outcomes – June 2010 Recommendation of outputs – September 2010 Improve management practices and service delivery and refine methods to gather outputs and immediate results (Narrative Reports, Client Feedback)

You are Part of the Success... We Need Your Input to Help Move the Work Forward Do you have samples of data collection tools/mechanisms you currently use that would show results of Settlement programming? What current types of reporting are useful? Not useful? What type of information do you want to share about your projects/services? How can we improve on gathering data about outputs and immediate outcomes? 12

Discussion

Next Steps Continue the work with the PMAG, taking into consideration your comments Consult with methodologists to confirm proposed approach to gathering data for –Outputs –Outcomes Finalize and implement performance measurement strategy Collect data and analyse and report on outputs and outcomes 14

Ultimate Outcomes Settlement Program Logic Model – June 2008 Immediate Outcomes Program Components CIC Strategic Outcome 3 Outputs Intermediate Outcomes Policy Development, Program Design and Management Policy and Program Development Settlement Services Program Implementation and Management Needs Assessment and Referrals Support Services Information & Awareness Services Employment- related Services Initial and on- going needs assessments (including language assessments) Referrals to CIC- funded and community settlement services Enabling services: –Childminding –Transportation assistance –Provisions for disabilities Other support services: –POE reception services –Translation –Interpretation –Settlement/ crisis counselling Information products Orientation sessions Promotion and outreach Labour market bridging Job search skills training Labour market information Workplace orientation 23. Successful integration of newcomers into society and the promotion of Canadian Citizenship 1.Policies and programming align with departmental and government priorities 2.Program models are evidenced-based, informed by stakeholder input and address the barriers & needs of both newcomers and communities 3.Standards, tools, resources and program coordination support the effective delivery of services) 4. Services are efficiently delivered 5. Provision of settlement services across Canada that achieve comparable outcomes 6. Clients, service providers and CIC are aware of newcomer settlement needs 7. Referrals and personalized settlement plans are based on assessed settlement needs 19. Newcomers find employment commensurate with their skills and experience 20. Newcomers enjoy their rights and act on their responsibilities in Canadian society 21. Canadians provide a welcoming community to facilitate the full participation of newcomers into Canadian society 22. Newcomers contribute to the economic, social and cultural development needs of Canada (in PAA) 8. Target population is aware of CIC settlement services 9. Timely, useful and appropriate CIC settlement services are available in the Official Language of choice (in accordance with the Official Languages Act and Policy) 10. Clients obtain the CIC settlement services they need to deal with settlement issues as they emerge Language Learning & Skills Development Language training Other skills/ life-skills training 11. Clients have timely, useful and accurate information needed to make informed settlement decisions 12. Clients understand life in Canada including laws, rights, responsibilities and how to access community resources 13. Clients have the official language skills needed to function in Canadian society 14. Clients have the skills/life-skills needed to function in Canadian society 15. Clients have knowledge of the Canadian work environment and are connected to local labour markets 16. Clients have the skills to find and apply for employment 17. Clients are connected to the broader community and social networks 18. Program participants are aware of newcomers’ needs and contributions and are engaged in newcomer settlement Strategic plans Policy, priorities, standards and outcomes Performance measurement strategy and national reports Horizontal coordination PT consultations Research analysis/ and reports Funding allocation Operational plans Program delivery materials and tools Functional guidance & training Data collection and regional/local/SPO reports Regional, local and SPO coordination Service delivery capacity building Best practices and info sharing Contribution agreements Community Connections Individual and community-level bridging, e.g.: –Host/ mentor matches –Volunteers engaged trained and supported Cultural awareness, anti-racism, and welcoming communities services E ABCD Expected Results A - Orientation B - Language/Skills C - Labour Market Access D – Welcoming Communities E - Program and Policy Development