1. Canadian Aviation Forecast Verification
WMO Workshop on Value-added Services and User-oriented Aerodrome Forecasts
September 9 – 10, 2005
Barry Green
Manager, Performance Assessment
Meteorological Service of Canada
Environnement Canada
Service Météorologique du Canada
Environment Canada
Meteorological Service of Canada

2. Outline Measuring aviation forecast performance Who are our clients?
What are our products?
Special project on user-oriented performance measurement
Performance Measurement TAF Improvements Project (PMTIP)
Environnement Canada
Service Météorologique du Canada
Environment Canada
Meteorological Service of Canada

3. Measuring Aviation Forecast Performance
Performance measurement has two main goals:
Service Quality Assessment
Demonstrate obligations are met
Monitor and analyse quality changes over time
Service Improvement
Identify strengths and weaknesses to aid in improving service
Service delivery
Research and development for enhanced products
Environnement Canada
Service Météorologique du Canada
Environment Canada
Meteorological Service of Canada

4. Who are the Clients? Civil Aviation Authority – NavCanada
Contract for aviation forecast services
Performance targets specified
Department of National Defence
Aerodrome forecasts for military stations
Aviation Forecast Centres
Two centres for civil aviation forecasts
National Defence forecast centres
Environnement Canada
Service Météorologique du Canada
Environment Canada
Meteorological Service of Canada

5. Basic Products Clients receive the same basic statistical products
Forward ART(Amendment Response Time)
Alternate FOH(Frequency of Hits)
At or above alternate limits
Alternate FAR(False Alarm Ratio)
Below alternate limits
VFR FOH(Frequency of Hits)
At or above 1000/3
Each of the clients has its own list of stations that make up the statistics
NavCanada for 175 stations
DND for 15 stations
Forecast centres for 175 stations
Environnement Canada
Service Météorologique du Canada
Environment Canada
Meteorological Service of Canada

6. Statistics
All statistics based on two by two contingency table (minute by minute)
FAR=C/(A+C) range 0 to False Alarm Ratio
FOH=A/(A+C) range 0 to Frequency of hits or Post Agreement
CSI=A/((A+B)+(A+C)-A) range 0 to 1 ---Threat Score or Critical Success Index
POD=A/(A+B) range 0 to Probability of Detection
NPFD=(POD-FAR+1)/2 range 0 to Normalized POD FAR Difference
FOCN=D/(B+D) range 0 to Frequency of correct Null Forecasts
F O R E C A S T O B S
YES NO A
(hits)
(misses) C
(False Alarms) D
(null)
Environnement Canada
Service Météorologique du Canada
Environment Canada
Meteorological Service of Canada

7. Statistics
All statistics based on two by two contingency table (minute by minute)
HPI= range - to 1--- Heidke Skill Score
Measures the usefulness of a forecast compared to a standard
The standard used is chance with the perfect score being 1
No skill is equal to zero
Set T=A+B+C+D
Set E=((A+B)(A+C)+(B+D)(C+D))
Then HPI=((A+D)-(E/T)) / (T-(E/T))
F O R E C A S T O B S
YES NO A
(hits)
(misses) C
(False Alarms) D
(null)
Environnement Canada
Service Météorologique du Canada
Environment Canada
Meteorological Service of Canada

8. Forward ART (Amendment Response Time)
Report of when observations and forecasts do not agree
Provides details if within a two hour period the observation and forecast come back into agreement
Forecast can be amended so as to match observation
observation can change back to match forecast
original forecast can change to match observation
If after two hours observation and forecast still don’t match this is recorded and listed in the report
Main goal is to have forecast and observation to be out of alignment for no longer that 20 minutes
have at least 80 percent of misalignments to be corrected within 20 minutes
Environnement Canada
Service Météorologique du Canada
Environment Canada
Meteorological Service of Canada

9. Web Page Access to Statistics
Internal distribution of current statistics
FOH for above VFR ceilings/visibilities
FOH for above Alternate ceilings/visibilities
FAR for below Alternate ceilings/visibilities
Amendment Response Time
Basic Layout
Tabular reports
Graphs
Interpretive graphs
Environnement Canada
Service Météorologique du Canada
Environment Canada
Meteorological Service of Canada

10. TAF Performance Trends
*Heidke Skill Score (HSS) is Percent Correct (PC) adjusted to account for the proportion of forecasts that would have been correct by chance in the absence of skill.
Ranges from -1 to 1
Zero -> no skill over chance
One is maximum score
Available on internal web site – interpretive graph section
* Forecast Verification A Practitioner’s Guide in Atmospheric Science
Environnement Canada
Service Météorologique du Canada
Environment Canada
Meteorological Service of Canada

11. TAF Performance Trends Heidke Skill Score
175 stations
HSS for Below Alternate Ceilings/Visibilities
Slight upward trend for last few years showing improved skill
Environnement Canada
Service Météorologique du Canada
Environment Canada
Meteorological Service of Canada

12. Performance Measurement TAF Improvements Project
Purpose:
To support the aviation industry’s flight-planning process of ensuring safety and the selection of the most fuel efficient alternate aerodrome, thereby reducing the amount of extra (unnecessary) fuel carriage and subsequent fuel burn, and related Green House Gas (GHG) emissions
Sponsors:
Jointly funded by NavCanada, MSC and PERD
Primary End Users:
Pilots & Flight Dispatchers
Environnement Canada
Service Météorologique du Canada
Environment Canada
Meteorological Service of Canada

13. Client user group formed with representatives from:
PMTIP User Group
Client user group formed with representatives from:
NavCanada
Air Canada
Jazz (Air Canada regional connector airline)
Forecast centres
Environnement Canada
Service Météorologique du Canada
Environment Canada
Meteorological Service of Canada

14. User Impacts
Study by Nav Canada in consultation with airlines established that there is a valid business case to enhancing TAF’s
If a 100% TAF accuracy was achieved conservative estimates of savings were $12 million dollars per year
While 100% TAF accuracy is not possible, by focusing on critical areas even a 20% gain is significant
Environnement Canada
Service Météorologique du Canada
Environment Canada
Meteorological Service of Canada

15. User Impacts Small savings per flight add up quickly….
Average direct aircraft operating cost is $60/minute
Reducing flight time by 1 minute per flight saves $1.6 million/year
Fuel costs ~ $0.50 per kilogram
Reducing fuel burn by 100 kg per flight saves $13 million/year
100 kg of unnecessary fuel boarded per flight costs $2.0 million/year in extra fuel burn
Increasing public concern and impending legislation (Kyoto) require that we focus on reducing aircraft emissions
Every 100 kg of fuel burned releases the following combustion by-products
CO2 – 290 kg
CH4 – 2.73 kg
NO2 – 2.9 kg
Environnement Canada
Service Météorologique du Canada
Environment Canada
Meteorological Service of Canada

16. Potential for savings The best return on investment is achieved by:
Improving TAF accuracy for No Alternate IFR operations to reduce false alarm frequency
Concentrating on airports and their “Preferred” alternates, with the greatest number of air carrier movements to generate maximum economic and environmental value
Improving TAF accuracy for “Preferred” alternates to reduce false alarm frequency
Environnement Canada
Service Météorologique du Canada
Environment Canada
Meteorological Service of Canada

17. Performance Measurement
TAF performance is an essential risk evaluation tool to maintain safety and develop enhanced regulations and airline company procedures
The current performance measurement criteria are sound, basic indicators of TAF performance, but do not meet the needs of the user community when conducting risk assessments
NC/EC and the aviation user community need to work together to develop meaningful TAF performance measurement criteria
Performance measurement information in the correct format, provided to pilots and dispatchers is essential to make the best operational decisions to maximize efficiency
Environnement Canada
Service Météorologique du Canada
Environment Canada
Meteorological Service of Canada

18. Overview of PMTIP Project
Task 1: Development of Statistics
a: TAF & METAR Data Base & No Alternate IFR scenarios
b: Severe Weather
c: Wind
Task 2: Dissemination of Data
Web site
Task 3: Making System Operationally Ready
Environnement Canada
Service Météorologique du Canada
Environment Canada
Meteorological Service of Canada

19. Summary
Understanding user requirements is essential for performance measurement – can also be educational for the forecasters as it may be different from traditional understanding
Performance measurement should be useful for clients and for forecasters for improving service
Understanding how to use performance statistics requires some education of users – a little knowledge can be a dangerous thing!
Environnement Canada
Service Météorologique du Canada
Environment Canada
Meteorological Service of Canada