Presentation on theme: "Clean Air Agenda Horizontal Evaluation Roll-up: Synopsis of Evaluation Results Available Up to September 2010 Final Prepared by Government Consulting Services."— Presentation transcript:
1Clean Air Agenda Horizontal Evaluation Roll-up: Synopsis of Evaluation Results Available Up to September 2010FinalPrepared by Government Consulting Servicesfor Audit and Evaluation Branch, Environment CanadaNovember 8, 2010
2Acronyms ADAPT Adaptation Theme CARA Clean Air Regulatory Agenda CE Clean Energy ThemeCTClean Transportation ThemeDFAITDepartment of Foreign Affairs and International TradeDGDirector GeneralDGTLCCDirector General Theme Leads Coordinating CommitteeDMDeputy MinisterECEnvironment CanadaGHGGreenhouse gasHCHealth CanadaHRSDCHuman Resources and Skills Development CanadaIAQIndoor Air Quality ThemeICIndustry CanadaICCInterdepartmental Coordinating CommitteeINACIndian and Northern Affairs CanadaINTLInternational Actions ThemeM&AManagement & AccountabilityNRCNational Research CouncilNRCanNatural Resources CanadaPHACPublic Health Agency of CanadaTBTreasury BoardTBSTreasury Board SecretariatTCTransport CanadaUNFCCCUnited Nations Framework Convention on Climate Change
3IntroductionThis report has been prepared in fulfillment of a commitment to produce a horizontal roll-up of the results of the evaluations done for the Clean Air Agenda (CAA) themes.The roll-up is intended to help participating departments and agencies report on their CAA results to Parliamentarians and Canadians, and to inform decisions regarding CAA funding and activities beyond 2011.The report comprises four sections:CAA Profile;Methodology;Findings; andObservations.Acknowledgements:The project team was led by Martine Perrault under the direction of the Environment Canada Evaluation Director, Shelley Borys, and included Linda Lee, Gavin Lemieux and Katheryne O’Connor. The team would also like to thank all the individuals from the following organizations who participated in the working group for this project and well as contributed to the development of the evaluation reports that form the basis for this horizontal roll-up: Environment Canada (EC) (including the Clean Air Agenda - Results Management Secretariat), Department of Foreign Affairs and International Trade (DFAIT), Health Canada (HC), Indian and Northern Affairs Canada (INAC), National Research Council (NRC), National Resources Canada (NRCan), Public Health Agency of Canada (PHAC), and Transport Canada (TC). This report was prepared for Environment Canada by James Coflin from Government Consulting Services (PWGSC).
4CAA ProfileThis section provides an overview of the CAA under these headings:Background;Overview of Themes;CAA Program Components;Planned and Actual Expenditures by Theme;Agenda Logic Model; andGovernance.
5BackgroundIn the 2006 Speech from the Throne, the Government committed to taking measures to achieve tangible improvements in our environment, including reductions in air pollutant and greenhouse gas (GHG) emissions.Budget 2006 committed $1.7 billion, later increased to $2.2 billion, to the Clean Air Agenda (CAA) for the four-year period from toThe CAA comprises:the Clean Air Regulatory Agenda (CARA), an integrated regulatory approach to reducing air pollutant (nitrogen oxides, sulphur oxides, volatile organic compounds, and particulate matter) and GHG emissions, anda series of program measures to:support action by governments, individuals, organizations and businesses in reducing emissions of GHGs and air pollutants, andto provide effective responses to climate change.The CAA is a horizontal initiative comprising eight themes and 45 programs. (Note: The Agenda was expanded to include the Clean Energy Dialogue as a component of the International Actions theme in ).A tabular overview of the eight themes is presented on the next page, followed by lists of the program components within each theme, identifying the department(s) that is delivering the program.Detailed information on the Agenda and the program components are available in the Environment Canada Departmental Performance Report, Supplementary Table: Horizontal Initiatives:
6Overview of ThemesThemeLead departmentAllocationClean Air Regulatory Agenda (CARA)aEnvironment Canada$347.3 MClean EnergyNatural Resources Canada$ MClean TransportationTransport Canada$461.7 MIndoor Air QualityHealth Canada/National Research Council$23.0 MAdaptation$115.9 MInternational Actions$50.0 MPartnershipsb$12.0 MManagement and Accountability$5.0 Ma This total does not include additional funds under a reprofiling of CARA that was approved in 2009.b Note: The Partnerships Theme was not launched.Source: Environment Canada, Departmental Performance Report, Supplementary Information, Table : Horizontal Initiatives. Unless otherwise stated, all budget allocation numbers reported in this document are from this source.
7CAA Program Components 1 Clean Air Regulatory Agenda (EC)Industrial Sector Regulatory Actions (EC)Transportation Sector Regulatory Actions (EC, TC & NRCan)Consumer and Commercial Products Regulatory Actions (EC, NRCan)Indoor Air Quality Management Actions (HC)Science in Support of Regulatory Activities & Accountability (EC, HC)Emissions Reporting (EC)Emissions Trading System (EC)Enforcement and Reporting on Progress (EC)Policy Analysis (EC)Clean Energy (NRCan)ecoENERGY for Buildings and Housing (NRCan)ecoENERGY Retrofit (NRCan)ecoENERGY for Industry (NRCan)ecoENERGY for Renewable Power (NRCan)ecoENERGY Renewable Heat (NRCan)ecoENERGY for Technology (NRCan)ecoENERGY for Aboriginal and Northern Communities (INAC)Policy, Communications, Monitoring and Reporting (NRCan)International Actions (EC)International Obligations (EC, DFAIT)International Negotiations (EC, NRCan, DFAIT)Asia-Pacific Partnership (NRCan, EC, IC)PM Annex (EC)Clean Energy Dialogue (EC, NRCan, DFAIT)Partnerships (EC)1. Community Partnerships (EC)Clean Transportation (TC)ecoMobility (TC)ecoTECHNOLOGY for Vehicles (TC)ecoENERGY for Personal Vehicles (NRCan)ecoENERGY for Fleets (NRCan)ecoFREIGHT Partnerships (TC)National Harmonization Initiative for the Trucking Industry (TC)Freight Technology Demonstration Fund (TC)Freight Technology Incentives (TC)Marine Shore Power Program (TC)ecoAUTO Rebate Program (TC, HRSDC)Vehicle Scrappage Program (EC)Analytical & Policy Support (TC)Indoor Air Quality (HC/NRC)Indoor Air R&D Initiative (NRC)Radon Strategy (HC)Adaptation (EC)Assist Northerners in Assessing Key Vulnerabilities and Opportunities (INAC)National Air Quality Health Index and Forecast Program (HC, EC)Climate Change and Health Adaptation in Northern / Inuit Communities (HC)Improved Climate Change Scenarios (EC)Innovative Risk Management ToolsRegional Adaptation Action Partnerships (NRCan)Climate and Infectious Disease Alert and Response System to Protect the Health of Canadians (HC, PHAC)Management and Accountability (EC)Management & Accountability (EC)
8Planned and Actual Expenditures by Theme ($ million)aaThemeApprovedActualClean Air Regulatory Agenda63.842.276.062.6Clean Energy101.581.6206.9202.5Clean Transportation98.892.6219.4160.4Indoor Air Quality18.104.22.168Adaptation7.35.726.020.3International Actions13.27.810.79.7Partnerships b3.01.00.3Management and Accountability1.31.2Total294.4236.7549.7463.4Source: Environment Canada, and Departmental Performance Report, Supplementary Information, Table: Horizontal Initiatives.a Expenditures reported here may differ from those reported by individual departments or CAA themes due to differences in reporting periods.b Although the Partnerships Theme program activities were not launched, some expenditures were incurred during the program’s planning phase.
10GovernanceEC is the lead department on the overall CAA and is responsible for the Clean Air Agenda-Results Management Secretariat (CAA-RMS). In all, eight departments (EC; NRCan; TC; NRC; INAC; HC; PHAC; and DFAIT) are responsible for themes and/or individual programs. Note: Two additional departments (Industry Canada and Human Resources and Skills Development Canada) are partners in specific programs.The CAA accountability structure is composed of these interdepartmental committees:DM Committee.ADM Steering CommitteeDG Theme Management CommitteesDG Theme Leads Coordinating Committee (DGTLCC)The DGTLCC is the primary operational committee responsible for implementing the elements of the CAA, comprising the leads from each of the eight DG Theme Management Committees.The CAA-RMS is the main support structure for the CAA management and accountability theme. Its key functions include:Providing support and coordination for the horizontal management of the CAA across departments;Maintaining a database of financial resource information;Analyzing financial and non-financial information to support an assessment of performance across the CAA, including achievement of results, use, allocation and sufficiency of resources, and risks; andProviding relevant reports, as required, to the governance structure and TBS.
11MethodologyThis section discusses the plan for the CAA evaluation under these headings:Scope and Coverage Limitations;Approach and Methodology;Theme and Program Evaluations Used for the Horizontal Roll-Up;Evaluation Issues and Questions; andMethodological Limitations.
12Scope and Coverage Limitations The roll-up addresses the evaluation questions (page 15) set out in the 2008 horizontal evaluation plan.It focuses on assessing theme and program level outputs and immediate outcomes (as reported in CAA program and theme evaluations) that are contributing to the achievement of the CAA intermediate outcomes (see Logic Model, page 9). The CAA long term outcomes were not assessed as part of this horizontal roll-up because they are expected to occur in a longer time frame.Coverage LimitationsThe roll-up covers 39 (87%) of the 45 CAA program components (see page 14).The combined four-year funding for the programs covered in this roll-up represents about 95% ($2.1 billion) of the $2.2 billion CAA allocation (see page 14).Most of the contributing evaluations were conducted in , thereby only covering 2 to 2½ years of implementation of the CAA activities.Evaluations not included in the current roll-up will be reported separately when available. The current roll-up was finalized without the benefit of those missing components due to a commitment to provide this horizontal report in October 2010 to inform discussions on future CAA program activities.
13Approach and Methodology PurposeThis report, which examines the relevance and performance of the Clean Air Agenda, is intended to meet reporting commitments and to contribute to ongoing clean air and climate change policy development.AudienceThe CAA evaluation roll-up is intended to help meet the information needs of senior management at the departments and agencies that have developed and delivered CAA themes and program components, as well as at the Treasury Board Secretariat.MethodologyThe sole line of evidence for this report is the content of the two thematic roll-up reports and 14 theme and program level evaluations available for review by September 23, This excludes evaluations that are still pending, as specified on page 14.The following methods were used in the 14 theme and program evaluations that contributed information to this roll-up:key informant interviews (14),literature review (6),document review (14),analysis of performance data (5),analysis of secondary data (1),case study (5),survey (2), andexperts group (1).The evidence was extracted from the reports to produce crosswalk matrixes that mapped equivalent data from multiple theme and program evaluations to fields defined by the CAA evaluation questions.
14Theme and Program Evaluations Used for Horizontal Roll-Up Legend: ✓- Evaluation of this theme/program was included in this roll-up; X – Evaluation of this theme/program is in progress or planned.a - This total does not include additional funds under a reprofiling of CARA that was approved in 2009.b - This program was not subject to a discrete evaluation as per NRCan’s evaluation plan.c - These two programs extended their Terms and Conditions for one year and will thus expire in As a result, the evaluation was pushed back until winter 2011.
15Evaluation Issues & Questions Relevance – Are CAA activities consistent with and do they contribute to addressing federal government clean air priorities and the key environmental needs of Canadians?Are CAA activities aligned with federal government priorities?Are CAA activities connected with key environmental needs?Success – Has the CAA achieved its intended outcomes?To what extent has the CAA achieved its intended outcomes?Cost-Effectiveness – Are the most cost-effective and efficient means being used to achieve outcomes?Are there more cost-effective and efficient means of achieving CAA objectives?How could the efficiency of CAA activities be improved?Design & Delivery – Is the CAA designed and delivered in the best possible way?Have each of the CAA activities been implemented, or are they on track to being implemented, as planned and on time?Is the CAA management and accountability structure in place and functioning as anticipated?Is appropriate performance data being collected, captured and safeguarded? If so, is this information used to inform senior management/decision makers?What are the best practices and lessons learned from CAA activities?
16Methodological Limitations Limitations affecting the horizontal roll-upRoll-up vs. horizontal evaluation: The large scale of CAA activities (i.e., $2.2B, 45 programs and 8 departments) dictated the conduct of several theme and program level evaluations, and a subsequent horizontal roll-up of these evaluation findings, rather than an umbrella evaluation of the overall CAA.Program maturity: Most evaluation evidence was collected during early program implementation and therefore relies on limited performance data and/or data focusing on outputs rather than outcomes. Annual performance reports were however developed for the first two years of the CAA.Lines of evidence: Many of the evaluations contributing to this report were heavily reliant on information drawn from key informant interviews. The roll-up of the interview evidence cannot be considered highly reliable given the open-ended nature of the interviews, the variations in questions, and the self-interest of a majority of the respondents (about 46% of the 547 interviews conducted were with federal government employees managing and/or delivering the programs).Level of Focus: It is difficult to draw conclusions about some of the CAA evaluation questions because of the level of detail and focus of the contributing evaluations. Specifically, theme-level evaluations and theme-level roll-ups tend to produce generalized or heavily qualified findings that cannot be quantified or consolidated.Underlying LimitationsThe following methodological and analytical limitations were reported by more than one of the theme evaluations. (The number in parentheses indicates the number of evaluations identifying the limitation) :Program maturity (4): The programs were not completely implemented and it was too early to assess outcomes.Lines of evidence (3): Only two or three lines of evidence were used.Limited performance data (6): The available data were limited to output/activity level indicators.Policy processes (5): It was difficult to access/collect relevant output and outcome data on such processes.Cost data (4): Current and/or sufficiently detailed program and activity level financial data was not available.
17FindingsThis section rolls-up the findings and conclusions from the program and theme-level evaluations that are relevant to each horizontal-level evaluation question.
18Relevance 1. Are CAA activities aligned with federal government priorities? EvidenceOverall CAA activities and outcomes are seen to align with federal priorities as expressed in documents and statements such as:Budgets and speeches from the Throne;Turning the Corner, the April 2007 Government of Canada plan to reduce greenhouse gases and air pollution;UNFCCC commitments, including the commitment under the Copenhagen Accord to a 17% reduction in GHG emissions from 2005 levels;Canada’s Performance 2008–09 that identifies “a clean and healthy environment” and “healthy Canadians” as two of the government’s thirteen outcome areas;A Climate Change Plan for the Purposes of the Kyoto Protocol Implementation Act, May 2010.The evaluations confirmed that the program components they reported on are aligned with departmental priorities and program activity architectures.FindingCAA activities are aligned with federal priorities.
19Relevance 2. Are CAA activities connected with key environmental needs? EvidenceThe CAA evaluation plan listed three indicators for this question: demonstration of the utility/rationale for CAA activities; CAA programs remain relevant as new scientific evidence on air quality and climate change emerges; and, evidence that CAA program architecture has adapted to new evidence and changes of government policy and priorities.The theme and program evaluations, with the exception of the M&A theme evaluation, considered “needs” in the course of assessing the relevance of programs and activities. However, the interpretation of the concept and the kinds of evidence gathered varied. Some looked at key informant opinions about workloads or the responses of beneficiaries and stakeholders as evidence of need, while others referred to research results and/or secondary data sources to document air pollutant and GHG emission levels. The Clean Energy theme report demonstrated a continued need for several of the theme’s program activities.Three of the thematic evaluations examined whether new scientific evidence had emerged that would raise questions about the relevance of CAA activities – none was found.There were no discussions or reports of modified program architecture.FindingSeveral of the theme and program evaluations provided evidence to demonstrate that CAA activities are connected to key environmental needs.
20Success 3. To what extent has the CAA achieved its intended outcomes? ApproachThe roll-up assessed the extent to which each CAA intermediate outcome (see Logic Model, page 9) has been achieved by examining the “success” evidence from the evaluations for the themes that are expected to be the major contributors to each outcome, as identified below:CAA Intermediate OutcomesCARACTCEADAPTIAQINTLCanadians take actions in response to forecasted levels of air quality.International air quality agreements are consistent with Canada’s interests.Reduction measures for indoor air pollutants and radon exposure have been implemented.Targeted industrial sectors have reduced emissions of GHG and air pollutants and improved energy efficiency.Emissions of GHGs and air pollutants have been reduced and energy efficiency was improved in Canadian homes and buildings.Emissions of GHGs, air pollutants energy consumption have been reduced from modes of transportation.Emissions of GHGs and air pollutants have been reduced and energy efficiency was improved from the use of efficient and environment safe practices and products.Emissions of GHGs and air pollutants have been reduced due to the use of clean and renewable energy sources.Canadian communities and user groups use tools and information to assess climate change risks and plan adoption strategies.International climate change agreements are consistent with Canada’s interests.
21Success 3. To what extent has the CAA achieved its intended outcomes? Approach (continued)For each of the CAA intermediate outcomes, success information is presented under three headings:The plans made to achieve the outcome, including which themes/programs were involved and the nature of interventions -- incentives; regulations and agreements; or capacity building -- that are being used.The term “capacity building” is used here to refer to activities that involve obtaining or distributing information with the goal of changing individual and organizational behavior including research, public information products and services, social marketing campaigns, training and educational activities, and demonstration projects.The evidence from the theme and program evaluations about the extent to which planned outputs have been produced and intended outcomes achieved.Findings that can be drawn from the available evidence. It is important to note here that outcome data were not available for most of theme and program level evaluations because programs had not been fully implemented when evaluated, or their outputs were only expected to have the impacts that will contribute to key outcomes some months or years later. The findings sections that follow do not repeat this qualification, but rather highlight where outcomes data are available.continued......
22Success 3. To what extent has the CAA achieved its intended outcomes? Estimate of ProgressAn indicator (see example below) was developed to provide a high level estimation of progress made toward the achievement of each CAA intermediate outcome based on the evidence reported by the theme and program evaluations reviewed:No outputs – no outputs because program implementation was delayed or terminated.Partial outputs – outputs are being produced but not to planned levels, or work has been significantly delayed.Outputs delivered – most planned outputs are being delivered, but insufficient time has elapsed for their contribution to CAA intermediate outcomes to have been experienced.Outcomes likely achieved – given the extent of outputs delivered and program maturity (i.e., sufficient time has elapsed since program implementation), and assuming that the CAA results logic is accurate, it is likely that the intended intermediate outcome is being achieved. There was however no data collected in the program or theme evaluation to assess whether the outcome was being achieved.Outcomes achieved – there is reliable evidence that the intermediate CAA outcome has been achieved.Outcomes likelyOutputs deliveredNo outputsPartial outputsOutcomes achieved
23Outcomes likelyOutputs deliveredNo outputsPartial outputsOutcomes achievedCAA Intermediate Outcome: Canadians take actions in response to forecasted levels of air quality.The PlanThe seven Adaptation theme programs were designed to improve Canada’s capacity to reduce health risks from and adapt to a changing climate and air pollution. The program designers expected increased capacity to be achieved through research and the development of tools, partnerships and standards. Research and tool development is being done directly by the participating departments and by other organizations supported through contribution funding.The theme targets: the North; human health; and infrastructure.EvidenceThe Adaptation programs, with one exception, were launched in , not in , the first year of the CAA. The exception is the AQHI which is the extension of a predecessor program.The theme evaluation, drawing on four program evaluations, reported that the programs were being implemented as planned, although two of them were at risk of partial or delayed implementation.The available evidence showed that where implementation has begun, the programs are beginning to achieve immediate theme-level outcomes: increased capacity to conduct science; greater collaboration; and increased availability of and access to products.FindingsThere is some preliminary, although weak, evidence that the Adaptation theme is making progress toward achieving the theme’s immediate outcomes that are expected to contribute directly to this CAA intermediate outcome.
24Outcomes likelyOutputs deliveredNo outputsPartial outputsOutcomes achievedCAA Intermediate Outcome: International air quality agreements are consistent with Canada’s interestsThe PlanThe PM Annex component of the International Actions theme is working toward the negotiation of an annex to the 1991 Air Quality Agreement. Under such an agreement, the U.S. and Canada would commit to reducing emissions of PM (particulate matter) and its precursors. PM comprises air-bound, microscopic pollutants that have negative impacts on air quality, the environment and human health.EvidenceThe PM Annex initiative has two major outputs: i) partnerships and processes; and ii) formal and informal consultations with domestic stakeholders.The International Actions theme evaluation reports evidence that some progress was made toward developing partnerships and processes with the U.S. to establish the PM Annex.The evaluation found no evidence that stakeholders were consulted directly and it was not clear how Canada’s position was developed or how the views of domestic partners and stakeholders have had an influence or been meaningfully reflected.Changes in policy direction and positions on key issues in both Canada and the US have resulted in delays in achieving the PM Annex.FindingsPM Annex activities have produced outputs related to partnerships and processes but there is little evidence to show that consultations with domestic stakeholders have helped to inform the development of Canada’s position in PM Annex negotiations. Such consultations are necessary to ensure the PM Annex is consistent with Canada’s interests.
25Outcomes likelyOutputs deliveredNo outputsPartial outputsOutcomes achievedCAA Intermediate Outcome: Reduction measures for indoor air pollutants and radon exposure have been implementedThe PlanThe CAA expected to achieve this intermediate outcome through:the Indoor Air Quality theme, which is focusing on capacity building activities including research on improved ventilation, distribution and other indoor air quality solutions, construction of a indoor air research laboratory, creation of a national committee on indoor air quality; the mapping of radon zones; testing radon levels at federal sites and a radon public awareness campaign; andThe CARA theme that planned new federal guidelines (regulations) for indoor air contaminants, and a national radon risk management strategy.EvidenceBecause it was done during the third year of a four-year program and many of the outcomes are not anticipated until the end of the four-year period, the IAQ evaluation examined the progress that had been made under the theme’s two program components. It found that they were generally on track to deliver their planned outputs. The notable exceptions are:the programs have not delivered the planned radon public awareness campaign or the assessments of the indoor air quality public awareness effort due to restrictions on the conduct of public opinion research; and,radon testing of federal buildings is behind schedule.The CARA evaluation reported that federal guidelines for indoor radon, carbon monoxide and ozone levels have been published within planned time frames and that related work is generally on track.FindingsThe Indoor Air Quality Theme and CARA have produced, or are on track to produce, the outputs that are expected to contribute to the achievement of theme and CAA intermediate outcomes, but it is too early to expect that the outputs have had their intended impact.
26No outputsPartial outputsOutputs deliveredOutcomes likelyOutcomes achievedCAA Intermediate Outcome: Targeted industrial sectors have reduced emissions of GHG and air pollutants and improved energy efficiencyThe PlanThe Industrial Regulations component of the CARA theme was designed to develop and implement sector-specific GHG emissions regulations.EvidenceThe CARA evaluation reports that the development of regulations for industrial greenhouse gas emissions, as well as the development of emissions trading and offset systems, have been affected by “directional change in policy.” The program, however, has undertaken many of the planned activities (consultations, research, drafting instructions), but further work has been deferred.FindingsCARA has not yet had the opportunity to produce the outputs needed to achieve the GHG and air pollutant emission reductions required for the achievement of this CAA outcome.
27No outputsPartial outputsOutputs deliveredOutcomes likelyOutcomes achievedCAA Intermediate Outcome: Emissions of GHGs and air pollutants have been reduced and energy efficiency was improved in Canadian homes and buildingsThe PlanThe Clean Energy theme offered homeowners, as well as small and medium –size organizations, incentives to retrofit or construct energy efficient homes and buildings through two programs: ecoENERGY Retrofit ($805 M) and ecoENERGY for Buildings and Houses ($61 M).EvidenceAt the end of its first two years ( and ), the ecoENERGY Retrofit for Homes program had already achieved 64-80% of its targeted range for GHG emission reductions and energy savings. Evidence suggests that 84% of energy savings are directly attributable to the Program. Moreover, homeowners are undertaking double the number of retrofits they had originally planned and 60% are motivated to undertake further measures outside of the Program.At the end of its first two years ( and ), the Buildings and Houses program had achieved 63-71% of its targeted range for energy savings and 80% of its GHG emission reductions and had effectively extended the reach of NRCan’s energy efficiency programs in the residential sector. The program has also exceeded its target of 5,000 trained energy advisor professionals in the construction of energy efficient new homes and buildings.The ecoENERGY Retrofit for Small and Medium Organizations program had achieved GHG emission reductions and energy savings but not to the extent anticipated when targets for the programs were set. Furthermore, attribution of this progress to the program’s financial incentive activities is only partial as some firms would have implemented these changes on their own. For the most part, capacity building activities (e.g., training and design validation), are increasing the capacity of commercial / institutional energy users to adopt energy management best practices and implement energy efficiency measures. In turn, this contributes to the outcome of energy savings and GHG emission reductionsFindingsEvidence shows that, although they have not yet fully reached all of their intended targets, Clean Energy programs are achieving GHG reductions and energy efficiency improvements in the Canadian homes and buildings sector.
28Outcomes likelyOutputs deliveredNo outputsPartial outputsOutcomes achievedCAA Intermediate Outcome: Emissions of GHGs, air pollutants, energy consumption have been reduced from modes of transportationThe PlanTwo themes are contributing to the goals of increasing fuel efficiency and reducing GHG and air pollutant emissions within the personal and commercial transportation sectors:The CARA theme is focused on developing and implementing a range of regulations and related products; andthe Clean Transportation theme comprises capacity building and incentive programs intended to increase the adoption of fuel efficient transportation technologies and behaviours, as well as plans to establish voluntary emission reduction agreements with the air, railway and marine sectors of the transportation industry.EvidenceThe CARA theme evaluation found that the development of transportation-related regulations was happening as planned and within scheduled time frames.The Clean Transportation theme evaluation found that, with small exceptions and some delays, the initiative was delivering its planned activities and outputs. With respect to outcomes, it found evidence that the ecoAUTO Rebate program , an incentive program, had contributed to an increased uptake of fuel-efficient technologies in the personal vehicle market. There was also evidence of some increased awareness of energy efficient transportation technologies among target audiences.With respect to the capacity building programs, both the theme evaluation and NRCan’s Evaluation of the Transportation Sub-Sub Activity found the required outputs had been delivered but that outcomes could not be assessed due to a lack of baseline and outcomes data and the challenges of attributing changes in consumer and business behaviours to specific interventions.The Clean Transportation theme evaluation reports that the voluntary agreements with the air and railway sectors have been renewed or maintained, but a similar agreement for the marine sector has not been developed.FindingsBy and large, the CAA transportation-related activities have delivered the regulatory, incentive and capacity building outputs that are expected to contribute to the achievement of this CAA outcome.
29Outcomes likelyOutputs deliveredNo outputsPartial outputsOutcomes achievedCAA Intermediate Outcome: Emissions of GHGs and air pollutants have been reduced and energy efficiency was improved from the use of efficient and environment safe practices and productsThe PlanIt was expected that this outcome would be realized as a result of capacity building outputs such as public information services, activities promoting smarter energy use, support for the development and deployment of clean energy technologies, and fuel efficient driver training delivered by the Clean Energy and Clean Transportation themes, as well as regulatory actions developed through the CARA theme.EvidenceThe Clean Energy evaluation reports that awareness building activities and adoption of technologies under the ecoEnergy for Industry program contributed to reduced emissions. At the time of the evaluation, the program was within its target range for energy savings and GHG reductions. It however reports that, while the activities and programs funded within the ecoEnergy for Technology Initiative lay the groundwork for increased deployment of clean power generation, reduced emissions from this program will depend on the degree to which these technologies are deployed on a large scale.The Clean Transportation theme evaluation and NRCan’s Evaluation of the Transportation Sub-Sub Activity found that the planned capacity building outputs were being produced. The theme evaluation also highlighted the challenges of attributing changes in consumer and business behaviour to specific capacity building interventions.With respect to regulatory interventions, the CARA theme evaluation reported that the planned development of energy -efficiency standards for consumer products and regulations to reduce the emissions of volatile organic compounds (VOC) in the manufacture and use of consumer and commercial products were on track, but that the impact of the regulations would not be measurable until some years after they had come into effect. NRCan’s evaluation of Equipment programming indicated that its Energy Efficiency Standards and Labelling Program contributes to increasing the minimum energy efficiency of equipment manufactured by suppliers, and therefore purchased by consumers, thereby contributing to reductions in energy use and associated GHG emissions.FindingsThere is evidence that the Clean Energy, Clean Transportation and CARA themes have produced the regulatory and capacity building outputs designed to increase the use of efficient and safe practices and products and some evidence that this outcome is being achieved as a result. It is however too early to conclude on the achievement of this outcome for 2 of the 3 themes involved.
30Outcomes likelyOutputs deliveredNo outputsPartial outputsOutcomes achievedCAA Intermediate Outcome: Emissions of GHGs and air pollutants have been reduced due to the use of clean and renewable energy sourcesThe PlanThe CAA expects to achieve this outcome through the incentives delivered by two Clean Energy theme programs:ecoENERGY for Renewable Power ($276.0 M) provides incentives for the producers of renewable power (e.g., from wind, low impact hydro, biomass and solar photovoltaic); andecoENERGY for Renewable Heat ($36.0 M) uses contributions to commercial, industrial and institutional users to encourage the installation of solar water and air heating systems. As more are installed, prices are expected to fall and the consumption of conventional energy should decline.EvidenceBy April 2009, Renewable Power projects with a total capacity of 3,100 megawatts had been commissioned or were under construction – the program’s target is approximately 4,000 megawatts by March 31, 2011.During its first two years, the Renewable Heat Program supported 334 solar thermal projects – its target is 700 systems by March 31, 2011.FindingsNRCan’s Renewable Energy Evaluation concluded that, after two years, the renewable energy programs have delivered planned outputs and are on track to reach most of their intended outcomes. In addition, it expressed the opinion that many of the programs’ long-term outcomes will be achieved, although it is not possible to directly attribute the achievement of these long-term outcomes to the programs due to interacting factors and considerations.
31Outcomes likelyOutputs deliveredNo outputsPartial outputsOutcomes achievedCAA Intermediate Outcome: Canadian communities and user groups use tools and information to assess climate change risks and plan adaption strategies.The PlanSeveral of the Adaptation Theme programs included capacity development activities (i.e., the development and distribution of tools and information such as guidelines, methodologies, data tables, models, and web pages) to help researchers, government departments and agencies, businesses, community groups and others assess risks and plan adaptive measures.EvidenceThe Adaptation Theme evaluation reported that the evaluations completed for four of the seven programs indicate that the programs were generally being implemented as planned and that Improved Climate Change Scenarios may not be fully implemented within the program timeframe.Available information indicates that the use of adaptation information and products is most likely to be used by groups who have an immediate interest in climate change impacts (e.g., scientists or individuals who are at-risk from adverse health impacts).FindingsThe evidence points to increased use of tools and information by groups with a clear interest in, or an incentive to use, adaptation products and information.
32Outcomes likelyOutputs deliveredNo outputsPartial outputsOutcomes achievedCAA Intermediate Outcome: International climate change agreements are consistent with Canada’s interestsThe PlanThe International Partnerships and Negotiations component of the International Actions Theme is the most important contributor to the achievement of this CAA outcome. Its activities are intended to ensure that Canadian environmental and economic interests are advanced through international negotiations. Note: For the purposes of evaluation, “Canada’s interests” are those goals and interests defined by the Government of Canada in federal negotiating instructions and formal policy statements.EvidenceThe International Actions evaluation reported that the International Partnerships and Negotiations program had produced planned outputs.The evaluation concluded that Canada’s interests are reflected in international climate change agreements, although it is difficult to attribute this directly to the activities under the International Actions theme. As well, there is clear evidence of enhanced consultation activities but they have not necessarily resulted in increased consideration of stakeholder interests in Canada’s positionsFindingsThere is evidence that the International Actions theme has, at least partially, achieved this CAA intermediate outcome.
33Success Progress toward Outcomes: Summary CAA Intermediate OutcomesNo outputsPartial outputsOutputs deliveredOutcomes likelyOutcomes achievedCanadians take actions in response to forecasted levels of air quality.International air quality agreements are consistent with Canada’s interests.Reduction measures for indoor air pollutants and radon exposure have been implementedTargeted industrial sectors have reduced emissions of GHG and air pollutants and improved energy efficiencyEmissions of GHGs and air pollutants have been reduced and energy efficiency was improved in Canadian homes and buildingsEmissions of GHGs, air pollutants and energy consumption have been reduced from modes of transportation.Emissions of GHGs and air pollutants have been reduced and energy efficiency was improved from the use of efficient and environment safe practices and products.Emissions of GHGs and air pollutants have been reduced due to the use of clean and renewable energy sources.Canadian communities and user groups use tools and information to assess climate change risks and plan adoption strategiesInternational climate change agreements are consistent with Canada’s interests.
34Cost-Effectiveness/ Alternatives 4 Cost-Effectiveness/ Alternatives 4. Are there more cost-effective and efficient means of achieving CAA objectives?EvidenceOpinions about alternative more efficient approaches: Key informants were either satisfied that the current program models were as or more efficient than alternatives, or they did not know of any realistic alternatives.Comparison of CAA activities to other comparable programs: Only the Clean Transportation Theme evaluation reported the results of a comparative analysis. It found that comparable programs exist for most Clean Transportation programs, but the availability of cost information for the comparable programs limited the analysis to the ecoAUTO Rebate and the Vehicle Scrappage programs. The Vehicle Scrappage program seems to be more efficient than United States and United Kingdom initiatives based on calculations of incentive cost per vehicle scrapped and total cost per vehicle scrapped. A comparison of the ecoAUTO Rebate program and the U.S. Cash for Clunkers program shows similar unit costs for administration. 3Opinions about whether CAA investments are a good use of public funds: While the evaluations do include key informant suggestions for program refinements and adjustments, the overall tenor of the evidence is that program staff and other stakeholders are generally satisfied that the programs are doing their job at reasonable cost and that no realistic alternatives are available.Costing emission reductions: The Clean Transportation evaluation calculated a cost per tonne of GHG emissions reduced based on forecasted GHG emissions reductions and original program budgets. The results suggest that the Vehicle Scrappage and ecoAUTO Rebate programs are not cost-effective.FindingWhile there is key informant and comparative evidence that CAA activities are efficient, calculations of the cost per tonne of GHG emission reductions suggest that some programs, particularly incentive programs, are not cost-effective.
35Cost-Effectiveness/ Alternatives 5 Cost-Effectiveness/ Alternatives 5. How could the efficiency of CAA activities be improved?EvidenceThe evaluations sought key informant opinions about ways to improve the efficiency of the programs they were associated with.The suggestions that were offered and reported tend to be specific to particular programs, even to particular activities.A comparison of the evaluations failed to identify any suggestions that were repeated across themes and programs.During the first two years of operation CAA theme actual expenditures were under budget (see page 8).FindingThere was no evidence of efficiency improvements that would be relevant to CAA activities generally.
36Design and Delivery 6. Have each of the CAA activities been implemented, or are they on track to being implemented, as planned and on time?EvidenceInformation about the implementation status of 40 of the 45 CAA programs is available.The majority of the programs have been implemented as planned, that is they are producing the expected outputs and, generally, within the expected timeframes:One program, Clean Transportation’s ecoAUTO Rebate program (TC) has been completed; andTwenty-nine programs were identified as on track – delivering their outputs as planned.One (1) CAA program was not implemented as plannedi.e. Community Partnerships (EC).Work on two CARA initiatives – the Emissions Trading System (EC) and the Industrial Sector Regulatory Actions (EC) – has been deferred.Seven programs have experienced challenges resulting in delayed or partial implementation:The Clean Transportation programs ecoMobility (TC); Marine Shore Power Program (TC); National Harmonization Initiative for Trucking Industry (TC), and ecoFREIGHT Partnerships on Freight (TC);The International Actions initiatives PM Annex (EC) and Asia-Pacific Partnership (NRCan); andThe Radon Strategy (HC).FindingThe majority (30 of 40) of the CAA programs evaluated have been implemented as planned, six have been implemented with significant schedule and/ or output changes, two were not implemented and two are described as deferred.
37Design and Delivery 7. Is the CAA management and accountability structure in place and functioning as anticipated?EvidenceThe CAA management and accountability structure is the subject of a separate evaluation that concluded, in part, that:The M&A Theme has brought coherence to the horizontal reporting of CAA activities and results and has emerged as a strong success story, in large part because of the CAA–RMS. Central agency representatives said that it was a key model to follow for reporting on horizontal initiatives.Roles and responsibilities for CAA management have been clearly defined, processes to support decision making and coordination have been established, and results are being reported to Parliament and Canadians in a transparent manner.There was limited evidence, however, of policy and program adjustments being made within the CAA horizontal decision-making structures. When needed, horizontal collaboration has occurred through other more flexible channels.The general tenor of the management and accountability evaluation findings at the theme level tend to validate the M&A theme evaluation conclusion: horizontal management structures have been implemented and play an important role in reporting, but have limited roles in program and policy decision-making, in part due to limited interdependencies between themes (see page 20). Instead, decision making operates on vertical lines of accountability and, where there are interdependencies between programs, bilateral planning and coordination solutions are used.FindingThe agenda and theme level horizontal management structures have been implemented as planned. Their functioning in terms of CAA reporting responsibilities exceeds expectations, but they have not played any significant role in decision-making.
38Design and Delivery 8. Is appropriate performance data being collected, captured and safeguarded? If so, is this information used to inform senior management/decision makers?EvidenceThe CAA evaluation plan listed three indicators for this question:Populated performance data systems: The key informant and document review evidence reported by the theme and program level evaluations suggests that the programs are meeting basic departmental and horizontal reporting requirements.The evaluations do not contain enough specific information to do a program-by-program analysis, but taken together, their observations suggest that a minority of programs have fully functioning systems.The most commonly identified issues were: the lack of an adequately structured and maintained system; processes or systems that focus on activities and outputs, but lack key outcome data; and indicators that are not relevant or appropriate.Six (6) of the nine (9) that examined performance data questions made recommendations for improvements to performance data.Decisions based on performance information: Only two evaluations reported on the use of performance data in decision making: the International Actions evaluation observed that it was used for departmental reporting, but little else, while the Clean Transportation document review and key informant interviews found examples of the use of such data in program management decision making, as well as reporting.Baseline data: Only a few programs have collected or will be able to collect baseline data. As a result it will be difficult, if not impossible, to assess the achievement of intermediate and longer-term outcomes whether the outputs involved incentives, capacity building activities or regulations.FindingIt appears that CAA programs have established systems or processes to gather and maintain activity and output data that supports program management, but some are not collecting the baseline and outcomes data needed to measure and evaluate performance.
39Design and Delivery 9. What are the best practices and lessons learned from CAA activities? EvidenceThe following best practices and lessons learned are those at the Agenda level that cut across several CAA themes or programs. Other best practices and lessons learned can be found in the CAA theme and program evaluation reports.Best Practices: A best practice is a methodology or work process that has been demonstrated to achieve superior results. The evaluations reported a number of best practices, but often fail to provided evidence to demonstrate that the activity described was a superior solution.The best practices at the Agenda level reported by the Management and Accountability evaluation were:Requiring approval of CAA performance reports by DMs of partner departments and agencies before the reports are submitted to TBS.The detailed charter, and the CAA–RMS implementation of the terms of the charter.The approach of the CAA–RMS to ensure effective financial and performance data collection, compilation and reporting (acknowledged by TBS to be a model of reporting on a horizontal initiative).Lessons Learned: A lesson learned describes an aspect of a project or program that did or did not work well, and identifies the risk of ignoring the lesson. In all, six of the evaluations reported a total 19 “lessons learned.”The evaluations tended to provide only a brief description of the lesson without context or identification of risk.While the lessons tended to be specific to a particular program or activity, it is worth noting that four of the evaluations (CARA, Adaptation, Renewable Energy and IAQ) listed nine lessons learned that called for proactive communication to and engagement of external stakeholders, including communities and individuals.FindingThe CAA–RMS led processes for financial and performance data compilation and reporting represents a best practice for horizontal initiatives.
40Overall Observations This section presents observations in two groups: Program observations: Observations related to overall CAA relevance and performanceEvaluation observations: Observations related to technical aspects of horizontal evaluations
41Program Observations Relevance The CAA programs are aligned with government priorities and are connected to key environmental needs.Achievement of OutcomesEarly evidence shows that a large majority of the planned activities are being implemented and delivered.As acknowledged in the 2008 CAA horizontal evaluation plan and the theme and program evaluations, the impacts of most of the CAA programs will not be felt or measureable for some time, even if the program has been fully implemented.Some outcome data for incentives programs (ecoEnergy for Retrofit, Renewable Energy, eco AUTO, and Vehicle Scrappage) suggests that there is increasing uptake of energy efficiency products and services, fuel-efficient technologies and clean energy sources, but there are also outstanding attribution challenges.With the exception of Clean Energy programs, there is mostly anecdotal evidence about the likely impacts of CAA capacity building activities and, without baseline data and given restrictions on survey research, it will be difficult to reliably assess those impacts in the future.Literature reviews have found international evidence about the effectiveness of regulatory interventions, but evaluations of the Clean Air regulations have not yet been done.Cost EffectivenessThere is preliminary evidence that incentive programs, like the ecoAUTO Rebate program and the Vehicle Scrappage program, may not be cost-effective when measured in terms of estimated emission reduction.Design and DeliveryThe CAA horizontal management structures have enabled very successful horizontal reporting but have had limited usefulness for program and policy decision-making in light of the limited interdependencies between CAA themes and the maintenance of vertical lines of departmental accountability.
42Evaluation Observations Horizontal Evaluation FrameworksThe development of the horizontal evaluation plan provided some degree of consistency across the theme and program level evaluations, but not enough to facilitate a strong roll-up of results. Future horizontal evaluation planning might benefit from establishing requirements for all theme and program evaluations to address a small number of core questions, with well defined indicators and comparable/equivalent data sources.Performance MeasurementThe majority of the theme and program evaluations reported that they encountered problems because there was limited outcome data. In many cases this was because the program had not reached the stage of its implementation where outcomes could be measured, or because of methodological issues such as limitations on public opinion research. In other cases, however, it may be that program managers did not collect and maintain outcome data.Evaluation planning would benefit from scheduling early reviews of program performance measurement frameworks and their implementation.Common TerminologyThe roll-up of the theme and program evaluation results was complicated by different interpretations of some concepts and terms such as needs, utility/rationale, unintended outcome, best practice, and lessons learned. While variation in interpretation and understanding of these terms is common and not usually an issue with a single evaluation, a horizontal evaluation plan would benefit from including a terminology lexicon to obtain consistent evidence across several evaluation units and numerous evaluators.
43EndnotesDescriptions of each of the CAA program components are included in the Environment Canada Departmental Performance Report, Supplementary Table: Horizontal Initiatives. The supplementary table is published atThe primary information sources for this evaluation roll-up were the following program and theme-level evaluation reports:CARA: Evaluation of the Clean Air Regulatory Agenda -DRAFT (June 14, 2010), Audit and Evaluation Branch, Environment Canada.CARA: Evaluation of the Equipment Sub-Sub Activity - Draft Final Report (June 28, 2010), NRCan Strategic Evaluation Division. (Note: This evaluation covers NRCan’s Energy Efficiency and Standards Labelling Program, which is a sub-component of CARA under the Consumer and Commercial Products Regulatory Actions program).Clean Energy: NRCan’s Clean Energy Thematic Report - Draft (September 2010), NRCan, Strategic Evaluation Division.Clean Energy: Renewable Energy Evaluation – Final Report (July 2010), NRCan, Strategic Evaluation Division.Clean Energy: Evaluation of the Clean Electrical Power Generation (CEPG) S&T Sub-sub Activity (PAA ) - Final Evaluation Report. (September 17, 2010), NRCan, Strategic Evaluation Division.Clean Energy: Evaluation of the Oil and Gas Sub-sub Activity. Draft Report (September 22, 2010), NRCan, Strategic Evaluation Division.Clean Energy: Evaluation of Energy Efficiency for Industry, Housing and Buildings – DRAFT Report (September 15, 2010), NRCan, Strategic Evaluation Division.Clean Transportation: Government Consulting Services, Report on the Evaluation of the Clean Transportation Theme of the Clean Air Agenda (July 2010), Transport Canada. (Note: This report covers the Clean Transportation theme programs delivered by Transport Canada and Environment Canada).Clean Transportation: Evaluation of the Transportation Sub-Sub Activity - Final Report (July 6, 2010), NRCan, Strategic Evaluation Division. (Note: This evaluation covers NRCan’s participation the transportation sector regulatory actions and six programs, two of which are part of the Clean Transportation theme).Indoor Air Quality: Government Consulting Services, Evaluation of the Indoor Air Quality Theme of the Clean Air Agenda (June 2010), Health Canada and National Research Council.Adaptation: Evaluation of the Clean Air Adaptation Theme: Review of Program Evaluation and Performance Measurement Findings -DRAFT (July, 2010). Audit and Evaluation Branch, Environment Canada.Adaptation: Evaluation of the National Air Quality Health Index Program (February 2010), Audit and Evaluation Branch, Environment Canada.Adaptation: Evaluation of the Improved Climate Change Scenarios Program - Final Draft Report (June 2010), Audit and Evaluation Branch, Environment Canada.International Actions: Evaluation of the International Actions Theme of the Clean Air Agenda - Draft Final Report (June 2010), Audit and Evaluation Branch, Environment Canada.International Actions: Marbek | Stratos. Evaluation of the Canada-U.S. Clean Energy Dialogue - Draft Report, Audit and Evaluation Branch, Environment Canada.Management and Accountability: Evaluation of the Clean Air Agenda Management and Accountability Theme - Draft (Revised July 13, 2010), Audit and Evaluation Branch, Environment Canada.
44EndnotesIt should be noted, though, that the United Kingdom and United States programs were adopted for economic stimulus reasons rather than to reduce transportation emissions.The NRCan Evaluation of the Equipment Sub-Sub Activity (contributing to the CARA Theme) made similar calculations for the two non-CAA programs it reported on (equivalent data for the CAA programs was not available): (page 42)The Accelerated Standards Action Program was expected to reduce GHG emissions by 2.8 Mt/year in In total, $28.4 million was spent under the Accelerated Standards Action Program. Therefore, if the Accelerated Standards Action Program achieved its GHG emissions reductions estimate, it will result in $10.14 per tonne of GHG emissions reduced in 2010, with reductions ongoing past 2010.EnerGuide for Industry was expected to reduce GHG emissions by 1.4 Mt/year in In total, $2 million was spent under EnerGuide for Industry. Therefore, if EnerGuide for Industry achieved its GHG emissions reductions estimate, it will result in $1.43 per tonne of GHG emissions reduced in 2010, with reductions ongoing past 2010.