Canadian Aviation Forecast Verification WMO Workshop on Value-added Services and User-oriented Aerodrome Forecasts September 9 – 10, 2005 Barry Green Manager, Performance Assessment Meteorological Service of Canada Environnement Canada Service Météorologique du Canada Environment Canada Meteorological Service of Canada

Outline Measuring aviation forecast performance Who are our clients? What are our products? Special project on user-oriented performance measurement Performance Measurement TAF Improvements Project (PMTIP) Environnement Canada Service Météorologique du Canada Environment Canada Meteorological Service of Canada

Measuring Aviation Forecast Performance Performance measurement has two main goals: Service Quality Assessment Demonstrate obligations are met Monitor and analyse quality changes over time Service Improvement Identify strengths and weaknesses to aid in improving service Service delivery Research and development for enhanced products Environnement Canada Service Météorologique du Canada Environment Canada Meteorological Service of Canada

Who are the Clients? Civil Aviation Authority – NavCanada Contract for aviation forecast services Performance targets specified Department of National Defence Aerodrome forecasts for military stations Aviation Forecast Centres Two centres for civil aviation forecasts National Defence forecast centres Environnement Canada Service Météorologique du Canada Environment Canada Meteorological Service of Canada

Basic Products Clients receive the same basic statistical products Forward ART(Amendment Response Time) Alternate FOH(Frequency of Hits) At or above alternate limits Alternate FAR(False Alarm Ratio) Below alternate limits VFR FOH(Frequency of Hits) At or above 1000/3 Each of the clients has its own list of stations that make up the statistics NavCanada for 175 stations DND for 15 stations Forecast centres for 175 stations Environnement Canada Service Météorologique du Canada Environment Canada Meteorological Service of Canada

Statistics All statistics based on two by two contingency table (minute by minute) FAR=C/(A+C) range 0 to 1 --- False Alarm Ratio FOH=A/(A+C) range 0 to 1 --- Frequency of hits or Post Agreement CSI=A/((A+B)+(A+C)-A) range 0 to 1 ---Threat Score or Critical Success Index POD=A/(A+B) range 0 to 1 --- Probability of Detection NPFD=(POD-FAR+1)/2 range 0 to 1 --- Normalized POD FAR Difference FOCN=D/(B+D) range 0 to 1 --- Frequency of correct Null Forecasts F O R E C A S T O B S YES NO A (hits) (misses) C (False Alarms) D (null) Environnement Canada Service Météorologique du Canada Environment Canada Meteorological Service of Canada

Statistics All statistics based on two by two contingency table (minute by minute) HPI= range - to 1--- Heidke Skill Score Measures the usefulness of a forecast compared to a standard The standard used is chance with the perfect score being 1 No skill is equal to zero Set T=A+B+C+D Set E=((A+B)(A+C)+(B+D)(C+D)) Then HPI=((A+D)-(E/T)) / (T-(E/T)) F O R E C A S T O B S YES NO A (hits) (misses) C (False Alarms) D (null) Environnement Canada Service Météorologique du Canada Environment Canada Meteorological Service of Canada

Forward ART (Amendment Response Time) Report of when observations and forecasts do not agree Provides details if within a two hour period the observation and forecast come back into agreement Forecast can be amended so as to match observation observation can change back to match forecast original forecast can change to match observation If after two hours observation and forecast still don’t match this is recorded and listed in the report Main goal is to have forecast and observation to be out of alignment for no longer that 20 minutes have at least 80 percent of misalignments to be corrected within 20 minutes Environnement Canada Service Météorologique du Canada Environment Canada Meteorological Service of Canada

Web Page Access to Statistics Internal distribution of current statistics FOH for above VFR ceilings/visibilities FOH for above Alternate ceilings/visibilities FAR for below Alternate ceilings/visibilities Amendment Response Time Basic Layout Tabular reports Graphs Interpretive graphs Environnement Canada Service Météorologique du Canada Environment Canada Meteorological Service of Canada

TAF Performance Trends *Heidke Skill Score (HSS) is Percent Correct (PC) adjusted to account for the proportion of forecasts that would have been correct by chance in the absence of skill. Ranges from -1 to 1 Zero -> no skill over chance One is maximum score Available on internal web site – interpretive graph section * Forecast Verification A Practitioner’s Guide in Atmospheric Science Environnement Canada Service Météorologique du Canada Environment Canada Meteorological Service of Canada

TAF Performance Trends Heidke Skill Score 175 stations HSS for Below Alternate Ceilings/Visibilities Slight upward trend for last few years showing improved skill Environnement Canada Service Météorologique du Canada Environment Canada Meteorological Service of Canada

Performance Measurement TAF Improvements Project Purpose: To support the aviation industry’s flight-planning process of ensuring safety and the selection of the most fuel efficient alternate aerodrome, thereby reducing the amount of extra (unnecessary) fuel carriage and subsequent fuel burn, and related Green House Gas (GHG) emissions Sponsors: Jointly funded by NavCanada, MSC and PERD Primary End Users: Pilots & Flight Dispatchers Environnement Canada Service Météorologique du Canada Environment Canada Meteorological Service of Canada

Client user group formed with representatives from: PMTIP User Group Client user group formed with representatives from: NavCanada Air Canada Jazz (Air Canada regional connector airline) Forecast centres Environnement Canada Service Météorologique du Canada Environment Canada Meteorological Service of Canada

User Impacts Study by Nav Canada in consultation with airlines established that there is a valid business case to enhancing TAF’s If a 100% TAF accuracy was achieved conservative estimates of savings were $12 million dollars per year While 100% TAF accuracy is not possible, by focusing on critical areas even a 20% gain is significant Environnement Canada Service Météorologique du Canada Environment Canada Meteorological Service of Canada

User Impacts Small savings per flight add up quickly…. Average direct aircraft operating cost is $60/minute Reducing flight time by 1 minute per flight saves $1.6 million/year Fuel costs ~ $0.50 per kilogram Reducing fuel burn by 100 kg per flight saves $13 million/year 100 kg of unnecessary fuel boarded per flight costs $2.0 million/year in extra fuel burn Increasing public concern and impending legislation (Kyoto) require that we focus on reducing aircraft emissions Every 100 kg of fuel burned releases the following combustion by-products CO2 – 290 kg CH4 – 2.73 kg NO2 – 2.9 kg Environnement Canada Service Météorologique du Canada Environment Canada Meteorological Service of Canada

Potential for savings The best return on investment is achieved by: Improving TAF accuracy for No Alternate IFR operations to reduce false alarm frequency Concentrating on airports and their “Preferred” alternates, with the greatest number of air carrier movements to generate maximum economic and environmental value Improving TAF accuracy for “Preferred” alternates to reduce false alarm frequency Environnement Canada Service Météorologique du Canada Environment Canada Meteorological Service of Canada

Performance Measurement TAF performance is an essential risk evaluation tool to maintain safety and develop enhanced regulations and airline company procedures The current performance measurement criteria are sound, basic indicators of TAF performance, but do not meet the needs of the user community when conducting risk assessments NC/EC and the aviation user community need to work together to develop meaningful TAF performance measurement criteria Performance measurement information in the correct format, provided to pilots and dispatchers is essential to make the best operational decisions to maximize efficiency Environnement Canada Service Météorologique du Canada Environment Canada Meteorological Service of Canada

Overview of PMTIP Project Task 1: Development of Statistics a: TAF & METAR Data Base & No Alternate IFR scenarios b: Severe Weather c: Wind Task 2: Dissemination of Data Web site Task 3: Making System Operationally Ready Environnement Canada Service Météorologique du Canada Environment Canada Meteorological Service of Canada

Summary Understanding user requirements is essential for performance measurement – can also be educational for the forecasters as it may be different from traditional understanding Performance measurement should be useful for clients and for forecasters for improving service Understanding how to use performance statistics requires some education of users – a little knowledge can be a dangerous thing! Environnement Canada Service Météorologique du Canada Environment Canada Meteorological Service of Canada