1. Clean Air Agenda Horizontal Evaluation Roll-up: Synopsis of Evaluation Results Available Up to September 2010
Final
Prepared by Government Consulting Services
for Audit and Evaluation Branch, Environment Canada
November 8, 2010

2. Acronyms ADAPT Adaptation Theme CARA Clean Air Regulatory Agenda CE
Clean Energy Theme CT
Clean Transportation Theme
DFAIT
Department of Foreign Affairs and International Trade DG
Director General
DGTLCC
Director General Theme Leads Coordinating Committee DM
Deputy Minister EC
Environment Canada
GHG
Greenhouse gas HC
Health Canada
HRSDC
Human Resources and Skills Development Canada
IAQ
Indoor Air Quality Theme IC
Industry Canada
ICC
Interdepartmental Coordinating Committee
INAC
Indian and Northern Affairs Canada
INTL
International Actions Theme
M&A
Management & Accountability
NRC
National Research Council
NRCan
Natural Resources Canada
PHAC
Public Health Agency of Canada TB
Treasury Board
TBS
Treasury Board Secretariat TC
Transport Canada
UNFCCC
United Nations Framework Convention on Climate Change

3. Introduction
This report has been prepared in fulfillment of a commitment to produce a horizontal roll-up of the results of the evaluations done for the Clean Air Agenda (CAA) themes.
The roll-up is intended to help participating departments and agencies report on their CAA results to Parliamentarians and Canadians, and to inform decisions regarding CAA funding and activities beyond 2011.
The report comprises four sections:
CAA Profile;
Methodology;
Findings; and
Observations.
Acknowledgements:
The project team was led by Martine Perrault under the direction of the Environment Canada Evaluation Director, Shelley Borys, and included Linda Lee, Gavin Lemieux and Katheryne O’Connor. The team would also like to thank all the individuals from the following organizations who participated in the working group for this project and well as contributed to the development of the evaluation reports that form the basis for this horizontal roll-up: Environment Canada (EC) (including the Clean Air Agenda - Results Management Secretariat), Department of Foreign Affairs and International Trade (DFAIT), Health Canada (HC), Indian and Northern Affairs Canada (INAC), National Research Council (NRC), National Resources Canada (NRCan), Public Health Agency of Canada (PHAC), and Transport Canada (TC). This report was prepared for Environment Canada by James Coflin from Government Consulting Services (PWGSC).

4. CAA Profile
This section provides an overview of the CAA under these headings:
Background;
Overview of Themes;
CAA Program Components;
Planned and Actual Expenditures by Theme;
Agenda Logic Model; and
Governance.

5. Background
In the 2006 Speech from the Throne, the Government committed to taking measures to achieve tangible improvements in our environment, including reductions in air pollutant and greenhouse gas (GHG) emissions.
Budget 2006 committed $1.7 billion, later increased to $2.2 billion, to the Clean Air Agenda (CAA) for the four-year period from to
The CAA comprises:
the Clean Air Regulatory Agenda (CARA), an integrated regulatory approach to reducing air pollutant (nitrogen oxides, sulphur oxides, volatile organic compounds, and particulate matter) and GHG emissions, and
a series of program measures to:
support action by governments, individuals, organizations and businesses in reducing emissions of GHGs and air pollutants, and
to provide effective responses to climate change.
The CAA is a horizontal initiative comprising eight themes and 45 programs. (Note: The Agenda was expanded to include the Clean Energy Dialogue as a component of the International Actions theme in ).
A tabular overview of the eight themes is presented on the next page, followed by lists of the program components within each theme, identifying the department(s) that is delivering the program.
Detailed information on the Agenda and the program components are available in the Environment Canada Departmental Performance Report, Supplementary Table: Horizontal Initiatives:

6. Overview of Themes
Theme
Lead department
Allocation
Clean Air Regulatory Agenda (CARA)a
Environment Canada
$347.3 M
Clean Energy
Natural Resources Canada
$ M
Clean Transportation
Transport Canada
$461.7 M
Indoor Air Quality
Health Canada/National Research Council
$23.0 M
Adaptation
$115.9 M
International Actions
$50.0 M
Partnershipsb
$12.0 M
Management and Accountability
$5.0 M
a This total does not include additional funds under a reprofiling of CARA that was approved in 2009.
b Note: The Partnerships Theme was not launched.
Source: Environment Canada, Departmental Performance Report, Supplementary Information, Table : Horizontal Initiatives. Unless otherwise stated, all budget allocation numbers reported in this document are from this source.

7. CAA Program Components 1
Clean Air Regulatory Agenda (EC)
Industrial Sector Regulatory Actions (EC)
Transportation Sector Regulatory Actions (EC, TC & NRCan)
Consumer and Commercial Products Regulatory Actions (EC, NRCan)
Indoor Air Quality Management Actions (HC)
Science in Support of Regulatory Activities & Accountability (EC, HC)
Emissions Reporting (EC)
Emissions Trading System (EC)
Enforcement and Reporting on Progress (EC)
Policy Analysis (EC)
Clean Energy (NRCan)
ecoENERGY for Buildings and Housing (NRCan)
ecoENERGY Retrofit (NRCan)
ecoENERGY for Industry (NRCan)
ecoENERGY for Renewable Power (NRCan)
ecoENERGY Renewable Heat (NRCan)
ecoENERGY for Technology (NRCan)
ecoENERGY for Aboriginal and Northern Communities (INAC)
Policy, Communications, Monitoring and Reporting (NRCan)
International Actions (EC)
International Obligations (EC, DFAIT)
International Negotiations (EC, NRCan, DFAIT)
Asia-Pacific Partnership (NRCan, EC, IC)
PM Annex (EC)
Clean Energy Dialogue (EC, NRCan, DFAIT)
Partnerships (EC)
1. Community Partnerships (EC)
Clean Transportation (TC)
ecoMobility (TC)
ecoTECHNOLOGY for Vehicles (TC)
ecoENERGY for Personal Vehicles (NRCan)
ecoENERGY for Fleets (NRCan)
ecoFREIGHT Partnerships (TC)
National Harmonization Initiative for the Trucking Industry (TC)
Freight Technology Demonstration Fund (TC)
Freight Technology Incentives (TC)
Marine Shore Power Program (TC)
ecoAUTO Rebate Program (TC, HRSDC)
Vehicle Scrappage Program (EC)
Analytical & Policy Support (TC)
Indoor Air Quality (HC/NRC)
Indoor Air R&D Initiative (NRC)
Radon Strategy (HC)
Adaptation (EC)
Assist Northerners in Assessing Key Vulnerabilities and Opportunities (INAC)
National Air Quality Health Index and Forecast Program (HC, EC)
Climate Change and Health Adaptation in Northern / Inuit Communities (HC)
Improved Climate Change Scenarios (EC)
Innovative Risk Management Tools
Regional Adaptation Action Partnerships (NRCan)
Climate and Infectious Disease Alert and Response System to Protect the Health of Canadians (HC, PHAC)
Management and Accountability (EC)
Management & Accountability (EC)

8. Planned and Actual Expenditures by Theme
($ million) a a
Theme
Approved
Actual
Clean Air Regulatory Agenda
63.8
42.2
76.0
62.6
Clean Energy
101.5
81.6
206.9
202.5
Clean Transportation
98.8
92.6
219.4
160.4
Indoor Air Quality
5.5
4.6
6.4
Adaptation
7.3
5.7
26.0
20.3
International Actions
13.2
7.8
10.7
9.7
Partnerships b
3.0
1.0
0.3
Management and Accountability
1.3
1.2
Total
294.4
236.7
549.7
463.4
Source: Environment Canada, and Departmental Performance Report, Supplementary Information, Table: Horizontal Initiatives.
a Expenditures reported here may differ from those reported by individual departments or CAA themes due to differences in reporting periods.
b Although the Partnerships Theme program activities were not launched, some expenditures were incurred during the program’s planning phase.

9. Agenda Logic Model

10. Governance
EC is the lead department on the overall CAA and is responsible for the Clean Air Agenda-Results Management Secretariat (CAA-RMS). In all, eight departments (EC; NRCan; TC; NRC; INAC; HC; PHAC; and DFAIT) are responsible for themes and/or individual programs. Note: Two additional departments (Industry Canada and Human Resources and Skills Development Canada) are partners in specific programs.
The CAA accountability structure is composed of these interdepartmental committees:
DM Committee.
ADM Steering Committee
DG Theme Management Committees
DG Theme Leads Coordinating Committee (DGTLCC)
The DGTLCC is the primary operational committee responsible for implementing the elements of the CAA, comprising the leads from each of the eight DG Theme Management Committees.
The CAA-RMS is the main support structure for the CAA management and accountability theme. Its key functions include:
Providing support and coordination for the horizontal management of the CAA across departments;
Maintaining a database of financial resource information;
Analyzing financial and non-financial information to support an assessment of performance across the CAA, including achievement of results, use, allocation and sufficiency of resources, and risks; and
Providing relevant reports, as required, to the governance structure and TBS.

11. Methodology
This section discusses the plan for the CAA evaluation under these headings:
Scope and Coverage Limitations;
Approach and Methodology;
Theme and Program Evaluations Used for the Horizontal Roll-Up;
Evaluation Issues and Questions; and
Methodological Limitations.

12. Scope and Coverage Limitations
The roll-up addresses the evaluation questions (page 15) set out in the 2008 horizontal evaluation plan.
It focuses on assessing theme and program level outputs and immediate outcomes (as reported in CAA program and theme evaluations) that are contributing to the achievement of the CAA intermediate outcomes (see Logic Model, page 9). The CAA long term outcomes were not assessed as part of this horizontal roll-up because they are expected to occur in a longer time frame.
Coverage Limitations
The roll-up covers 39 (87%) of the 45 CAA program components (see page 14).
The combined four-year funding for the programs covered in this roll-up represents about 95% ($2.1 billion) of the $2.2 billion CAA allocation (see page 14).
Most of the contributing evaluations were conducted in , thereby only covering 2 to 2½ years of implementation of the CAA activities.
Evaluations not included in the current roll-up will be reported separately when available. The current roll-up was finalized without the benefit of those missing components due to a commitment to provide this horizontal report in October 2010 to inform discussions on future CAA program activities.

13. Approach and Methodology
Purpose
This report, which examines the relevance and performance of the Clean Air Agenda, is intended to meet reporting commitments and to contribute to ongoing clean air and climate change policy development.
Audience
The CAA evaluation roll-up is intended to help meet the information needs of senior management at the departments and agencies that have developed and delivered CAA themes and program components, as well as at the Treasury Board Secretariat.
Methodology
The sole line of evidence for this report is the content of the two thematic roll-up reports and 14 theme and program level evaluations available for review by September 23, This excludes evaluations that are still pending, as specified on page 14.
The following methods were used in the 14 theme and program evaluations that contributed information to this roll-up:
key informant interviews (14),
literature review (6),
document review (14),
analysis of performance data (5),
analysis of secondary data (1),
case study (5),
survey (2), and
experts group (1).
The evidence was extracted from the reports to produce crosswalk matrixes that mapped equivalent data from multiple theme and program evaluations to fields defined by the CAA evaluation questions.

14. Theme and Program Evaluations Used for Horizontal Roll-Up Legend: ✓- Evaluation of this theme/program was included in this roll-up; X – Evaluation of this theme/program is in progress or planned.
a - This total does not include additional funds under a reprofiling of CARA that was approved in 2009.
b - This program was not subject to a discrete evaluation as per NRCan’s evaluation plan.
c - These two programs extended their Terms and Conditions for one year and will thus expire in As a result, the evaluation was pushed back until winter 2011.

15. Evaluation Issues & Questions
Relevance – Are CAA activities consistent with and do they contribute to addressing federal government clean air priorities and the key environmental needs of Canadians?
Are CAA activities aligned with federal government priorities?
Are CAA activities connected with key environmental needs?
Success – Has the CAA achieved its intended outcomes?
To what extent has the CAA achieved its intended outcomes?
Cost-Effectiveness – Are the most cost-effective and efficient means being used to achieve outcomes?
Are there more cost-effective and efficient means of achieving CAA objectives?
How could the efficiency of CAA activities be improved?
Design & Delivery – Is the CAA designed and delivered in the best possible way?
Have each of the CAA activities been implemented, or are they on track to being implemented, as planned and on time?
Is the CAA management and accountability structure in place and functioning as anticipated?
Is appropriate performance data being collected, captured and safeguarded? If so, is this information used to inform senior management/decision makers?
What are the best practices and lessons learned from CAA activities?

16. Methodological Limitations
Limitations affecting the horizontal roll-up
Roll-up vs. horizontal evaluation: The large scale of CAA activities (i.e., $2.2B, 45 programs and 8 departments) dictated the conduct of several theme and program level evaluations, and a subsequent horizontal roll-up of these evaluation findings, rather than an umbrella evaluation of the overall CAA.
Program maturity: Most evaluation evidence was collected during early program implementation and therefore relies on limited performance data and/or data focusing on outputs rather than outcomes. Annual performance reports were however developed for the first two years of the CAA.
Lines of evidence: Many of the evaluations contributing to this report were heavily reliant on information drawn from key informant interviews. The roll-up of the interview evidence cannot be considered highly reliable given the open-ended nature of the interviews, the variations in questions, and the self-interest of a majority of the respondents (about 46% of the 547 interviews conducted were with federal government employees managing and/or delivering the programs).
Level of Focus: It is difficult to draw conclusions about some of the CAA evaluation questions because of the level of detail and focus of the contributing evaluations. Specifically, theme-level evaluations and theme-level roll-ups tend to produce generalized or heavily qualified findings that cannot be quantified or consolidated.
Underlying Limitations
The following methodological and analytical limitations were reported by more than one of the theme evaluations. (The number in parentheses indicates the number of evaluations identifying the limitation) :
Program maturity (4): The programs were not completely implemented and it was too early to assess outcomes.
Lines of evidence (3): Only two or three lines of evidence were used.
Limited performance data (6): The available data were limited to output/activity level indicators.
Policy processes (5): It was difficult to access/collect relevant output and outcome data on such processes.
Cost data (4): Current and/or sufficiently detailed program and activity level financial data was not available.

17. Findings
This section rolls-up the findings and conclusions from the program and theme-level evaluations that are relevant to each horizontal-level evaluation question.

18. Relevance 1. Are CAA activities aligned with federal government priorities?
Evidence
Overall CAA activities and outcomes are seen to align with federal priorities as expressed in documents and statements such as:
Budgets and speeches from the Throne;
Turning the Corner, the April 2007 Government of Canada plan to reduce greenhouse gases and air pollution;
UNFCCC commitments, including the commitment under the Copenhagen Accord to a 17% reduction in GHG emissions from 2005 levels;
Canada’s Performance 2008–09 that identifies “a clean and healthy environment” and “healthy Canadians” as two of the government’s thirteen outcome areas;
A Climate Change Plan for the Purposes of the Kyoto Protocol Implementation Act, May 2010.
The evaluations confirmed that the program components they reported on are aligned with departmental priorities and program activity architectures.
Finding
CAA activities are aligned with federal priorities.

19. Relevance 2. Are CAA activities connected with key environmental needs?
Evidence
The CAA evaluation plan listed three indicators for this question: demonstration of the utility/rationale for CAA activities; CAA programs remain relevant as new scientific evidence on air quality and climate change emerges; and, evidence that CAA program architecture has adapted to new evidence and changes of government policy and priorities.
The theme and program evaluations, with the exception of the M&A theme evaluation, considered “needs” in the course of assessing the relevance of programs and activities. However, the interpretation of the concept and the kinds of evidence gathered varied. Some looked at key informant opinions about workloads or the responses of beneficiaries and stakeholders as evidence of need, while others referred to research results and/or secondary data sources to document air pollutant and GHG emission levels. The Clean Energy theme report demonstrated a continued need for several of the theme’s program activities.
Three of the thematic evaluations examined whether new scientific evidence had emerged that would raise questions about the relevance of CAA activities – none was found.
There were no discussions or reports of modified program architecture.
Finding
Several of the theme and program evaluations provided evidence to demonstrate that CAA activities are connected to key environmental needs.

20. Success 3. To what extent has the CAA achieved its intended outcomes?
Approach
The roll-up assessed the extent to which each CAA intermediate outcome (see Logic Model, page 9) has been achieved by examining the “success” evidence from the evaluations for the themes that are expected to be the major contributors to each outcome, as identified below:
CAA Intermediate Outcomes
CARA CT CE
ADAPT
IAQ
INTL
Canadians take actions in response to forecasted levels of air quality. 
International air quality agreements are consistent with Canada’s interests.
Reduction measures for indoor air pollutants and radon exposure have been implemented.
Targeted industrial sectors have reduced emissions of GHG and air pollutants and improved energy efficiency.
Emissions of GHGs and air pollutants have been reduced and energy efficiency was improved in Canadian homes and buildings.
Emissions of GHGs, air pollutants energy consumption have been reduced from modes of transportation.
Emissions of GHGs and air pollutants have been reduced and energy efficiency was improved from the use of efficient and environment safe practices and products.
Emissions of GHGs and air pollutants have been reduced due to the use of clean and renewable energy sources.
Canadian communities and user groups use tools and information to assess climate change risks and plan adoption strategies.
International climate change agreements are consistent with Canada’s interests.

21. Success 3. To what extent has the CAA achieved its intended outcomes?
Approach (continued)
For each of the CAA intermediate outcomes, success information is presented under three headings:
The plans made to achieve the outcome, including which themes/programs were involved and the nature of interventions -- incentives; regulations and agreements; or capacity building -- that are being used.
The term “capacity building” is used here to refer to activities that involve obtaining or distributing information with the goal of changing individual and organizational behavior including research, public information products and services, social marketing campaigns, training and educational activities, and demonstration projects.
The evidence from the theme and program evaluations about the extent to which planned outputs have been produced and intended outcomes achieved.
Findings that can be drawn from the available evidence. It is important to note here that outcome data were not available for most of theme and program level evaluations because programs had not been fully implemented when evaluated, or their outputs were only expected to have the impacts that will contribute to key outcomes some months or years later. The findings sections that follow do not repeat this qualification, but rather highlight where outcomes data are available.
continued......

22. Success 3. To what extent has the CAA achieved its intended outcomes?
Estimate of Progress
An indicator (see example below) was developed to provide a high level estimation of progress made toward the achievement of each CAA intermediate outcome based on the evidence reported by the theme and program evaluations reviewed:
No outputs – no outputs because program implementation was delayed or terminated.
Partial outputs – outputs are being produced but not to planned levels, or work has been significantly delayed.
Outputs delivered – most planned outputs are being delivered, but insufficient time has elapsed for their contribution to CAA intermediate outcomes to have been experienced.
Outcomes likely achieved – given the extent of outputs delivered and program maturity (i.e., sufficient time has elapsed since program implementation), and assuming that the CAA results logic is accurate, it is likely that the intended intermediate outcome is being achieved. There was however no data collected in the program or theme evaluation to assess whether the outcome was being achieved.
Outcomes achieved – there is reliable evidence that the intermediate CAA outcome has been achieved.
Outcomes likely
Outputs delivered
No outputs
Partial outputs
Outcomes achieved

23. Outcomes likely
Outputs delivered
No outputs
Partial outputs
Outcomes achieved
CAA Intermediate Outcome: Canadians take actions in response to forecasted levels of air quality.
The Plan
The seven Adaptation theme programs were designed to improve Canada’s capacity to reduce health risks from and adapt to a changing climate and air pollution. The program designers expected increased capacity to be achieved through research and the development of tools, partnerships and standards. Research and tool development is being done directly by the participating departments and by other organizations supported through contribution funding.
The theme targets: the North; human health; and infrastructure.
Evidence
The Adaptation programs, with one exception, were launched in , not in , the first year of the CAA. The exception is the AQHI which is the extension of a predecessor program.
The theme evaluation, drawing on four program evaluations, reported that the programs were being implemented as planned, although two of them were at risk of partial or delayed implementation.
The available evidence showed that where implementation has begun, the programs are beginning to achieve immediate theme-level outcomes: increased capacity to conduct science; greater collaboration; and increased availability of and access to products.
Findings
There is some preliminary, although weak, evidence that the Adaptation theme is making progress toward achieving the theme’s immediate outcomes that are expected to contribute directly to this CAA intermediate outcome.

24. Outcomes likely
Outputs delivered
No outputs
Partial outputs
Outcomes achieved
CAA Intermediate Outcome: International air quality agreements are consistent with Canada’s interests
The Plan
The PM Annex component of the International Actions theme is working toward the negotiation of an annex to the 1991 Air Quality Agreement. Under such an agreement, the U.S. and Canada would commit to reducing emissions of PM (particulate matter) and its precursors. PM comprises air-bound, microscopic pollutants that have negative impacts on air quality, the environment and human health.
Evidence
The PM Annex initiative has two major outputs: i) partnerships and processes; and ii) formal and informal consultations with domestic stakeholders.
The International Actions theme evaluation reports evidence that some progress was made toward developing partnerships and processes with the U.S. to establish the PM Annex.
The evaluation found no evidence that stakeholders were consulted directly and it was not clear how Canada’s position was developed or how the views of domestic partners and stakeholders have had an influence or been meaningfully reflected.
Changes in policy direction and positions on key issues in both Canada and the US have resulted in delays in achieving the PM Annex.
Findings
PM Annex activities have produced outputs related to partnerships and processes but there is little evidence to show that consultations with domestic stakeholders have helped to inform the development of Canada’s position in PM Annex negotiations. Such consultations are necessary to ensure the PM Annex is consistent with Canada’s interests.

25. Outcomes likely
Outputs delivered
No outputs
Partial outputs
Outcomes achieved
CAA Intermediate Outcome: Reduction measures for indoor air pollutants and radon exposure have been implemented
The Plan
The CAA expected to achieve this intermediate outcome through:
the Indoor Air Quality theme, which is focusing on capacity building activities including research on improved ventilation, distribution and other indoor air quality solutions, construction of a indoor air research laboratory, creation of a national committee on indoor air quality; the mapping of radon zones; testing radon levels at federal sites and a radon public awareness campaign; and
The CARA theme that planned new federal guidelines (regulations) for indoor air contaminants, and a national radon risk management strategy.
Evidence
Because it was done during the third year of a four-year program and many of the outcomes are not anticipated until the end of the four-year period, the IAQ evaluation examined the progress that had been made under the theme’s two program components. It found that they were generally on track to deliver their planned outputs. The notable exceptions are:
the programs have not delivered the planned radon public awareness campaign or the assessments of the indoor air quality public awareness effort due to restrictions on the conduct of public opinion research; and,
radon testing of federal buildings is behind schedule.
The CARA evaluation reported that federal guidelines for indoor radon, carbon monoxide and ozone levels have been published within planned time frames and that related work is generally on track.
Findings
The Indoor Air Quality Theme and CARA have produced, or are on track to produce, the outputs that are expected to contribute to the achievement of theme and CAA intermediate outcomes, but it is too early to expect that the outputs have had their intended impact.

26. No outputs
Partial outputs
Outputs delivered
Outcomes likely
Outcomes achieved
CAA Intermediate Outcome: Targeted industrial sectors have reduced emissions of GHG and air pollutants and improved energy efficiency
The Plan
The Industrial Regulations component of the CARA theme was designed to develop and implement sector-specific GHG emissions regulations.
Evidence
The CARA evaluation reports that the development of regulations for industrial greenhouse gas emissions, as well as the development of emissions trading and offset systems, have been affected by “directional change in policy.” The program, however, has undertaken many of the planned activities (consultations, research, drafting instructions), but further work has been deferred.
Findings
CARA has not yet had the opportunity to produce the outputs needed to achieve the GHG and air pollutant emission reductions required for the achievement of this CAA outcome.

27. No outputs
Partial outputs
Outputs delivered
Outcomes likely
Outcomes achieved
CAA Intermediate Outcome: Emissions of GHGs and air pollutants have been reduced and energy efficiency was improved in Canadian homes and buildings
The Plan
The Clean Energy theme offered homeowners, as well as small and medium –size organizations, incentives to retrofit or construct energy efficient homes and buildings through two programs: ecoENERGY Retrofit ($805 M) and ecoENERGY for Buildings and Houses ($61 M).
Evidence
At the end of its first two years ( and ), the ecoENERGY Retrofit for Homes program had already achieved 64-80% of its targeted range for GHG emission reductions and energy savings. Evidence suggests that 84% of energy savings are directly attributable to the Program. Moreover, homeowners are undertaking double the number of retrofits they had originally planned and 60% are motivated to undertake further measures outside of the Program.
At the end of its first two years ( and ), the Buildings and Houses program had achieved 63-71% of its targeted range for energy savings and 80% of its GHG emission reductions and had effectively extended the reach of NRCan’s energy efficiency programs in the residential sector. The program has also exceeded its target of 5,000 trained energy advisor professionals in the construction of energy efficient new homes and buildings.
The ecoENERGY Retrofit for Small and Medium Organizations program had achieved GHG emission reductions and energy savings but not to the extent anticipated when targets for the programs were set. Furthermore, attribution of this progress to the program’s financial incentive activities is only partial as some firms would have implemented these changes on their own. For the most part, capacity building activities (e.g., training and design validation), are increasing the capacity of commercial / institutional energy users to adopt energy management best practices and implement energy efficiency measures. In turn, this contributes to the outcome of energy savings and GHG emission reductions
Findings
Evidence shows that, although they have not yet fully reached all of their intended targets, Clean Energy programs are achieving GHG reductions and energy efficiency improvements in the Canadian homes and buildings sector.

28. Outcomes likely
Outputs delivered
No outputs
Partial outputs
Outcomes achieved
CAA Intermediate Outcome: Emissions of GHGs, air pollutants, energy consumption have been reduced from modes of transportation
The Plan
Two themes are contributing to the goals of increasing fuel efficiency and reducing GHG and air pollutant emissions within the personal and commercial transportation sectors:
The CARA theme is focused on developing and implementing a range of regulations and related products; and
the Clean Transportation theme comprises capacity building and incentive programs intended to increase the adoption of fuel efficient transportation technologies and behaviours, as well as plans to establish voluntary emission reduction agreements with the air, railway and marine sectors of the transportation industry.
Evidence
The CARA theme evaluation found that the development of transportation-related regulations was happening as planned and within scheduled time frames.
The Clean Transportation theme evaluation found that, with small exceptions and some delays, the initiative was delivering its planned activities and outputs. With respect to outcomes, it found evidence that the ecoAUTO Rebate program , an incentive program, had contributed to an increased uptake of fuel-efficient technologies in the personal vehicle market. There was also evidence of some increased awareness of energy efficient transportation technologies among target audiences.
With respect to the capacity building programs, both the theme evaluation and NRCan’s Evaluation of the Transportation Sub-Sub Activity found the required outputs had been delivered but that outcomes could not be assessed due to a lack of baseline and outcomes data and the challenges of attributing changes in consumer and business behaviours to specific interventions.
The Clean Transportation theme evaluation reports that the voluntary agreements with the air and railway sectors have been renewed or maintained, but a similar agreement for the marine sector has not been developed.
Findings
By and large, the CAA transportation-related activities have delivered the regulatory, incentive and capacity building outputs that are expected to contribute to the achievement of this CAA outcome.

29. Outcomes likely
Outputs delivered
No outputs
Partial outputs
Outcomes achieved
CAA Intermediate Outcome: Emissions of GHGs and air pollutants have been reduced and energy efficiency was improved from the use of efficient and environment safe practices and products
The Plan
It was expected that this outcome would be realized as a result of capacity building outputs such as public information services, activities promoting smarter energy use, support for the development and deployment of clean energy technologies, and fuel efficient driver training delivered by the Clean Energy and Clean Transportation themes, as well as regulatory actions developed through the CARA theme.
Evidence
The Clean Energy evaluation reports that awareness building activities and adoption of technologies under the ecoEnergy for Industry program contributed to reduced emissions. At the time of the evaluation, the program was within its target range for energy savings and GHG reductions. It however reports that, while the activities and programs funded within the ecoEnergy for Technology Initiative lay the groundwork for increased deployment of clean power generation, reduced emissions from this program will depend on the degree to which these technologies are deployed on a large scale.
The Clean Transportation theme evaluation and NRCan’s Evaluation of the Transportation Sub-Sub Activity found that the planned capacity building outputs were being produced. The theme evaluation also highlighted the challenges of attributing changes in consumer and business behaviour to specific capacity building interventions.
With respect to regulatory interventions, the CARA theme evaluation reported that the planned development of energy -efficiency standards for consumer products and regulations to reduce the emissions of volatile organic compounds (VOC) in the manufacture and use of consumer and commercial products were on track, but that the impact of the regulations would not be measurable until some years after they had come into effect. NRCan’s evaluation of Equipment programming indicated that its Energy Efficiency Standards and Labelling Program contributes to increasing the minimum energy efficiency of equipment manufactured by suppliers, and therefore purchased by consumers, thereby contributing to reductions in energy use and associated GHG emissions.
Findings
There is evidence that the Clean Energy, Clean Transportation and CARA themes have produced the regulatory and capacity building outputs designed to increase the use of efficient and safe practices and products and some evidence that this outcome is being achieved as a result. It is however too early to conclude on the achievement of this outcome for 2 of the 3 themes involved.

30. Outcomes likely
Outputs delivered
No outputs
Partial outputs
Outcomes achieved
CAA Intermediate Outcome: Emissions of GHGs and air pollutants have been reduced due to the use of clean and renewable energy sources
The Plan
The CAA expects to achieve this outcome through the incentives delivered by two Clean Energy theme programs:
ecoENERGY for Renewable Power ($276.0 M) provides incentives for the producers of renewable power (e.g., from wind, low impact hydro, biomass and solar photovoltaic); and
ecoENERGY for Renewable Heat ($36.0 M) uses contributions to commercial, industrial and institutional users to encourage the installation of solar water and air heating systems. As more are installed, prices are expected to fall and the consumption of conventional energy should decline.
Evidence
By April 2009, Renewable Power projects with a total capacity of 3,100 megawatts had been commissioned or were under construction – the program’s target is approximately 4,000 megawatts by March 31, 2011.
During its first two years, the Renewable Heat Program supported 334 solar thermal projects – its target is 700 systems by March 31, 2011.
Findings
NRCan’s Renewable Energy Evaluation concluded that, after two years, the renewable energy programs have delivered planned outputs and are on track to reach most of their intended outcomes. In addition, it expressed the opinion that many of the programs’ long-term outcomes will be achieved, although it is not possible to directly attribute the achievement of these long-term outcomes to the programs due to interacting factors and considerations.

31. Outcomes likely
Outputs delivered
No outputs
Partial outputs
Outcomes achieved
CAA Intermediate Outcome: Canadian communities and user groups use tools and information to assess climate change risks and plan adaption strategies.
The Plan
Several of the Adaptation Theme programs included capacity development activities (i.e., the development and distribution of tools and information such as guidelines, methodologies, data tables, models, and web pages) to help researchers, government departments and agencies, businesses, community groups and others assess risks and plan adaptive measures.
Evidence
The Adaptation Theme evaluation reported that the evaluations completed for four of the seven programs indicate that the programs were generally being implemented as planned and that Improved Climate Change Scenarios may not be fully implemented within the program timeframe.
Available information indicates that the use of adaptation information and products is most likely to be used by groups who have an immediate interest in climate change impacts (e.g., scientists or individuals who are at-risk from adverse health impacts).
Findings
The evidence points to increased use of tools and information by groups with a clear interest in, or an incentive to use, adaptation products and information.

32. Outcomes likely
Outputs delivered
No outputs
Partial outputs
Outcomes achieved
CAA Intermediate Outcome: International climate change agreements are consistent with Canada’s interests
The Plan
The International Partnerships and Negotiations component of the International Actions Theme is the most important contributor to the achievement of this CAA outcome. Its activities are intended to ensure that Canadian environmental and economic interests are advanced through international negotiations. Note: For the purposes of evaluation, “Canada’s interests” are those goals and interests defined by the Government of Canada in federal negotiating instructions and formal policy statements.
Evidence
The International Actions evaluation reported that the International Partnerships and Negotiations program had produced planned outputs.
The evaluation concluded that Canada’s interests are reflected in international climate change agreements, although it is difficult to attribute this directly to the activities under the International Actions theme. As well, there is clear evidence of enhanced consultation activities but they have not necessarily resulted in increased consideration of stakeholder interests in Canada’s positions
Findings
There is evidence that the International Actions theme has, at least partially, achieved this CAA intermediate outcome.

33. Success Progress toward Outcomes: Summary
CAA Intermediate Outcomes
No outputs
Partial outputs
Outputs delivered
Outcomes likely
Outcomes achieved
Canadians take actions in response to forecasted levels of air quality.
International air quality agreements are consistent with Canada’s interests.
Reduction measures for indoor air pollutants and radon exposure have been implemented
Targeted industrial sectors have reduced emissions of GHG and air pollutants and improved energy efficiency
Emissions of GHGs and air pollutants have been reduced and energy efficiency was improved in Canadian homes and buildings
Emissions of GHGs, air pollutants and energy consumption have been reduced from modes of transportation.
Emissions of GHGs and air pollutants have been reduced and energy efficiency was improved from the use of efficient and environment safe practices and products.
Emissions of GHGs and air pollutants have been reduced due to the use of clean and renewable energy sources.
Canadian communities and user groups use tools and information to assess climate change risks and plan adoption strategies
International climate change agreements are consistent with Canada’s interests.

34. Cost-Effectiveness/ Alternatives 4
Cost-Effectiveness/ Alternatives 4. Are there more cost-effective and efficient means of achieving CAA objectives?
Evidence
Opinions about alternative more efficient approaches: Key informants were either satisfied that the current program models were as or more efficient than alternatives, or they did not know of any realistic alternatives.
Comparison of CAA activities to other comparable programs: Only the Clean Transportation Theme evaluation reported the results of a comparative analysis. It found that comparable programs exist for most Clean Transportation programs, but the availability of cost information for the comparable programs limited the analysis to the ecoAUTO Rebate and the Vehicle Scrappage programs. The Vehicle Scrappage program seems to be more efficient than United States and United Kingdom initiatives based on calculations of incentive cost per vehicle scrapped and total cost per vehicle scrapped. A comparison of the ecoAUTO Rebate program and the U.S. Cash for Clunkers program shows similar unit costs for administration. 3
Opinions about whether CAA investments are a good use of public funds: While the evaluations do include key informant suggestions for program refinements and adjustments, the overall tenor of the evidence is that program staff and other stakeholders are generally satisfied that the programs are doing their job at reasonable cost and that no realistic alternatives are available.
Costing emission reductions: The Clean Transportation evaluation calculated a cost per tonne of GHG emissions reduced based on forecasted GHG emissions reductions and original program budgets. The results suggest that the Vehicle Scrappage and ecoAUTO Rebate programs are not cost-effective.
Finding
While there is key informant and comparative evidence that CAA activities are efficient, calculations of the cost per tonne of GHG emission reductions suggest that some programs, particularly incentive programs, are not cost-effective.

35. Cost-Effectiveness/ Alternatives 5
Cost-Effectiveness/ Alternatives 5. How could the efficiency of CAA activities be improved?
Evidence
The evaluations sought key informant opinions about ways to improve the efficiency of the programs they were associated with.
The suggestions that were offered and reported tend to be specific to particular programs, even to particular activities.
A comparison of the evaluations failed to identify any suggestions that were repeated across themes and programs.
During the first two years of operation CAA theme actual expenditures were under budget (see page 8).
Finding
There was no evidence of efficiency improvements that would be relevant to CAA activities generally.

36. Design and Delivery 6. Have each of the CAA activities been implemented, or are they on track to being implemented, as planned and on time?
Evidence
Information about the implementation status of 40 of the 45 CAA programs is available.
The majority of the programs have been implemented as planned, that is they are producing the expected outputs and, generally, within the expected timeframes:
One program, Clean Transportation’s ecoAUTO Rebate program (TC) has been completed; and
Twenty-nine programs were identified as on track – delivering their outputs as planned.
One (1) CAA program was not implemented as planned
i.e. Community Partnerships (EC).
Work on two CARA initiatives – the Emissions Trading System (EC) and the Industrial Sector Regulatory Actions (EC) – has been deferred.
Seven programs have experienced challenges resulting in delayed or partial implementation:
The Clean Transportation programs ecoMobility (TC); Marine Shore Power Program (TC); National Harmonization Initiative for Trucking Industry (TC), and ecoFREIGHT Partnerships on Freight (TC);
The International Actions initiatives PM Annex (EC) and Asia-Pacific Partnership (NRCan); and
The Radon Strategy (HC).
Finding
The majority (30 of 40) of the CAA programs evaluated have been implemented as planned, six have been implemented with significant schedule and/ or output changes, two were not implemented and two are described as deferred.

37. Design and Delivery 7. Is the CAA management and accountability structure in place and functioning as anticipated?
Evidence
The CAA management and accountability structure is the subject of a separate evaluation that concluded, in part, that:
The M&A Theme has brought coherence to the horizontal reporting of CAA activities and results and has emerged as a strong success story, in large part because of the CAA–RMS. Central agency representatives said that it was a key model to follow for reporting on horizontal initiatives.
Roles and responsibilities for CAA management have been clearly defined, processes to support decision making and coordination have been established, and results are being reported to Parliament and Canadians in a transparent manner.
There was limited evidence, however, of policy and program adjustments being made within the CAA horizontal decision-making structures. When needed, horizontal collaboration has occurred through other more flexible channels.
The general tenor of the management and accountability evaluation findings at the theme level tend to validate the M&A theme evaluation conclusion: horizontal management structures have been implemented and play an important role in reporting, but have limited roles in program and policy decision-making, in part due to limited interdependencies between themes (see page 20). Instead, decision making operates on vertical lines of accountability and, where there are interdependencies between programs, bilateral planning and coordination solutions are used.
Finding
The agenda and theme level horizontal management structures have been implemented as planned. Their functioning in terms of CAA reporting responsibilities exceeds expectations, but they have not played any significant role in decision-making.

38. Design and Delivery 8. Is appropriate performance data being collected, captured and safeguarded? If so, is this information used to inform senior management/decision makers?
Evidence
The CAA evaluation plan listed three indicators for this question:
Populated performance data systems: The key informant and document review evidence reported by the theme and program level evaluations suggests that the programs are meeting basic departmental and horizontal reporting requirements.
The evaluations do not contain enough specific information to do a program-by-program analysis, but taken together, their observations suggest that a minority of programs have fully functioning systems.
The most commonly identified issues were: the lack of an adequately structured and maintained system; processes or systems that focus on activities and outputs, but lack key outcome data; and indicators that are not relevant or appropriate.
Six (6) of the nine (9) that examined performance data questions made recommendations for improvements to performance data.
Decisions based on performance information: Only two evaluations reported on the use of performance data in decision making: the International Actions evaluation observed that it was used for departmental reporting, but little else, while the Clean Transportation document review and key informant interviews found examples of the use of such data in program management decision making, as well as reporting.
Baseline data: Only a few programs have collected or will be able to collect baseline data. As a result it will be difficult, if not impossible, to assess the achievement of intermediate and longer-term outcomes whether the outputs involved incentives, capacity building activities or regulations.
Finding
It appears that CAA programs have established systems or processes to gather and maintain activity and output data that supports program management, but some are not collecting the baseline and outcomes data needed to measure and evaluate performance.

39. Design and Delivery 9. What are the best practices and lessons learned from CAA activities?
Evidence
The following best practices and lessons learned are those at the Agenda level that cut across several CAA themes or programs. Other best practices and lessons learned can be found in the CAA theme and program evaluation reports.
Best Practices: A best practice is a methodology or work process that has been demonstrated to achieve superior results. The evaluations reported a number of best practices, but often fail to provided evidence to demonstrate that the activity described was a superior solution.
The best practices at the Agenda level reported by the Management and Accountability evaluation were:
Requiring approval of CAA performance reports by DMs of partner departments and agencies before the reports are submitted to TBS.
The detailed charter, and the CAA–RMS implementation of the terms of the charter.
The approach of the CAA–RMS to ensure effective financial and performance data collection, compilation and reporting (acknowledged by TBS to be a model of reporting on a horizontal initiative).
Lessons Learned: A lesson learned describes an aspect of a project or program that did or did not work well, and identifies the risk of ignoring the lesson. In all, six of the evaluations reported a total 19 “lessons learned.”
The evaluations tended to provide only a brief description of the lesson without context or identification of risk.
While the lessons tended to be specific to a particular program or activity, it is worth noting that four of the evaluations (CARA, Adaptation, Renewable Energy and IAQ) listed nine lessons learned that called for proactive communication to and engagement of external stakeholders, including communities and individuals.
Finding
The CAA–RMS led processes for financial and performance data compilation and reporting represents a best practice for horizontal initiatives.

40. Overall Observations This section presents observations in two groups:
Program observations: Observations related to overall CAA relevance and performance
Evaluation observations: Observations related to technical aspects of horizontal evaluations

41. Program Observations Relevance
The CAA programs are aligned with government priorities and are connected to key environmental needs.
Achievement of Outcomes
Early evidence shows that a large majority of the planned activities are being implemented and delivered.
As acknowledged in the 2008 CAA horizontal evaluation plan and the theme and program evaluations, the impacts of most of the CAA programs will not be felt or measureable for some time, even if the program has been fully implemented.
Some outcome data for incentives programs (ecoEnergy for Retrofit, Renewable Energy, eco AUTO, and Vehicle Scrappage) suggests that there is increasing uptake of energy efficiency products and services, fuel-efficient technologies and clean energy sources, but there are also outstanding attribution challenges.
With the exception of Clean Energy programs, there is mostly anecdotal evidence about the likely impacts of CAA capacity building activities and, without baseline data and given restrictions on survey research, it will be difficult to reliably assess those impacts in the future.
Literature reviews have found international evidence about the effectiveness of regulatory interventions, but evaluations of the Clean Air regulations have not yet been done.
Cost Effectiveness
There is preliminary evidence that incentive programs, like the ecoAUTO Rebate program and the Vehicle Scrappage program, may not be cost-effective when measured in terms of estimated emission reduction.
Design and Delivery
The CAA horizontal management structures have enabled very successful horizontal reporting but have had limited usefulness for program and policy decision-making in light of the limited interdependencies between CAA themes and the maintenance of vertical lines of departmental accountability.

42. Evaluation Observations
Horizontal Evaluation Frameworks
The development of the horizontal evaluation plan provided some degree of consistency across the theme and program level evaluations, but not enough to facilitate a strong roll-up of results. Future horizontal evaluation planning might benefit from establishing requirements for all theme and program evaluations to address a small number of core questions, with well defined indicators and comparable/equivalent data sources.
Performance Measurement
The majority of the theme and program evaluations reported that they encountered problems because there was limited outcome data. In many cases this was because the program had not reached the stage of its implementation where outcomes could be measured, or because of methodological issues such as limitations on public opinion research. In other cases, however, it may be that program managers did not collect and maintain outcome data.
Evaluation planning would benefit from scheduling early reviews of program performance measurement frameworks and their implementation.
Common Terminology
The roll-up of the theme and program evaluation results was complicated by different interpretations of some concepts and terms such as needs, utility/rationale, unintended outcome, best practice, and lessons learned. While variation in interpretation and understanding of these terms is common and not usually an issue with a single evaluation, a horizontal evaluation plan would benefit from including a terminology lexicon to obtain consistent evidence across several evaluation units and numerous evaluators.

43. Endnotes
Descriptions of each of the CAA program components are included in the Environment Canada Departmental Performance Report, Supplementary Table: Horizontal Initiatives. The supplementary table is published at
The primary information sources for this evaluation roll-up were the following program and theme-level evaluation reports:
CARA: Evaluation of the Clean Air Regulatory Agenda -DRAFT (June 14, 2010), Audit and Evaluation Branch, Environment Canada.
CARA: Evaluation of the Equipment Sub-Sub Activity - Draft Final Report (June 28, 2010), NRCan Strategic Evaluation Division. (Note: This evaluation covers NRCan’s Energy Efficiency and Standards Labelling Program, which is a sub-component of CARA under the Consumer and Commercial Products Regulatory Actions program).
Clean Energy: NRCan’s Clean Energy Thematic Report - Draft (September 2010), NRCan, Strategic Evaluation Division.
Clean Energy: Renewable Energy Evaluation – Final Report (July 2010), NRCan, Strategic Evaluation Division.
Clean Energy: Evaluation of the Clean Electrical Power Generation (CEPG) S&T Sub-sub Activity (PAA ) - Final Evaluation Report. (September 17, 2010), NRCan, Strategic Evaluation Division.
Clean Energy: Evaluation of the Oil and Gas Sub-sub Activity. Draft Report (September 22, 2010), NRCan, Strategic Evaluation Division.
Clean Energy: Evaluation of Energy Efficiency for Industry, Housing and Buildings – DRAFT Report (September 15, 2010), NRCan, Strategic Evaluation Division.
Clean Transportation: Government Consulting Services, Report on the Evaluation of the Clean Transportation Theme of the Clean Air Agenda (July 2010), Transport Canada. (Note: This report covers the Clean Transportation theme programs delivered by Transport Canada and Environment Canada).
Clean Transportation: Evaluation of the Transportation Sub-Sub Activity - Final Report (July 6, 2010), NRCan, Strategic Evaluation Division. (Note: This evaluation covers NRCan’s participation the transportation sector regulatory actions and six programs, two of which are part of the Clean Transportation theme).
Indoor Air Quality: Government Consulting Services, Evaluation of the Indoor Air Quality Theme of the Clean Air Agenda (June 2010), Health Canada and National Research Council.
Adaptation: Evaluation of the Clean Air Adaptation Theme: Review of Program Evaluation and Performance Measurement Findings -DRAFT (July, 2010). Audit and Evaluation Branch, Environment Canada.
Adaptation: Evaluation of the National Air Quality Health Index Program (February 2010), Audit and Evaluation Branch, Environment Canada.
Adaptation: Evaluation of the Improved Climate Change Scenarios Program - Final Draft Report (June 2010), Audit and Evaluation Branch, Environment Canada.
International Actions: Evaluation of the International Actions Theme of the Clean Air Agenda - Draft Final Report (June 2010), Audit and Evaluation Branch, Environment Canada.
International Actions: Marbek | Stratos. Evaluation of the Canada-U.S. Clean Energy Dialogue - Draft Report, Audit and Evaluation Branch, Environment Canada.
Management and Accountability: Evaluation of the Clean Air Agenda Management and Accountability Theme - Draft (Revised July 13, 2010), Audit and Evaluation Branch, Environment Canada.

44. Endnotes
It should be noted, though, that the United Kingdom and United States programs were adopted for economic stimulus reasons rather than to reduce transportation emissions.
The NRCan Evaluation of the Equipment Sub-Sub Activity (contributing to the CARA Theme) made similar calculations for the two non-CAA programs it reported on (equivalent data for the CAA programs was not available): (page 42)
The Accelerated Standards Action Program was expected to reduce GHG emissions by 2.8 Mt/year in In total, $28.4 million was spent under the Accelerated Standards Action Program. Therefore, if the Accelerated Standards Action Program achieved its GHG emissions reductions estimate, it will result in $10.14 per tonne of GHG emissions reduced in 2010, with reductions ongoing past 2010.
EnerGuide for Industry was expected to reduce GHG emissions by 1.4 Mt/year in In total, $2 million was spent under EnerGuide for Industry. Therefore, if EnerGuide for Industry achieved its GHG emissions reductions estimate, it will result in $1.43 per tonne of GHG emissions reduced in 2010, with reductions ongoing past 2010.