1. Free Flight Phase 1 & 2 Performance Measurement Dave Knorr May 16, 2002

2. 2 Outline Free Flight Phase 1 and 2 Measurement Process How and What do we measure? – Mechanisms versus Operational Benefits Free Flight Phase 1 Case Studies Other Exploratory Metrics

3. 3 Free Flight Phase 1 & 2 Established to deliver new automation tools to controllers at limited locations – User Request Evaluation Tool (URET) – Traffic Management Advisor (TMA) – Passive Fianl Approach Spacing Tool (pFAST) – Collaborative Decision Making tools (CDM) Metrics program developed in collaboration with stakeholders Expanded deployment (FFP2) based on FFP1 measured benefits

4. 4 User Request Evaluation Tool Capabilities R-side Controller D-side Controller Assistant Controller Primary User En Route Facility For controllers providing en route services, URET offers the following: Automatic conflict detection Trial planning to assist with conflict resolution or user requests Conformance monitoring of current trajectory Electronic flight data capability

5. 5 Traffic Management Advisor TMA is a time-based system that: Provides a schedule for all arrivals Computes arrival times to the runway and other designated fixes Handles dynamic traffic situations Assists the controllers in meeting schedules by providing advisories Enhances the sequencing process by refining super stream class protocols based on prop and jet engine characteristics

6. 6 Measuring Impacts of New Tools Why is measurement of performance in an operational environment important? Can we measure actual performance of specific tools?

7. 7 Why Measure? Verify projected benefits from models/simulations – Feedback to program management Accountability for capital investments Improve future goal setting and models Model the future Measure Establish goals

8. 8 How Do We Measure? In Actual Operational Environment With: – Real people: controllers, pilots, dispatchers, etc. (not simulated) – Varying weather conditions and seasons – Varying aircraft mix – Varying airport conditions/configurations Partnership With Industry (airlines/airports/GA/others) - share data, share findings and results validation Over significant time periods - before and after implementation

9. 9 What Do We Measure? Quantitative Outcomes Actual Operational Change Mechanisms for Improvement Example: Increasing AmendmentsExample: Distance Savings

10. 10 What Do We Measure? Quantitative Outcomes Mechanisms for Operational Change: Are tools actually being used during study periods? Are tool recommendations being followed? Is there smoother flow into TRACON airspace? Are trial plans being used? Is controller workload reduced? These metrics should correlate with operational benefits measured.

11. 11 What Do We Measure? Quantitative Outcomes Actual Operational Change – Throughput During Peak Activity Periods An indicator of capacity changes – Efficiency Reduction of flight times and/or distances; fuel usage – Predictability Reduction in variation of flight times – Delay A good indicator of system problems Not a good indicator of capacity increases FFP Focus

12. 12 These Measures Interact Demand Delay Capacity Throughput Capacity Reduced Flight Time Increases Variation in Flight Time Increases Throughput Decreases IFR Weather

13. 13 These Measures Interact Demand Delay Capacity Throughput FFP TOOLS Throughput Increases Capacity Increases Demand More Demand Can Be Accommodated and/or Delay Decreases

14. 14 Runway Counts and Flying Times Runway Counts (EA/OA) (OA/IA) (IA/MA) (MA/RW)

15. 15 Key Findings from Free Flight Phase 1 Must be able to attribute operational change to controller actions (mechanisms) Focus on peak periods Include downstream affects of operational improvements

16. 16 Metrics on Operational Performance Changes TMA and URET Case Studies

17. 17 Metered Flow of Arrivals Departure Routes

18. 18 Unmetered Flow of Arrivals Departure Routes Increased Vectoring Increased Conflicts Missed Slots Long Final

19. 19 TMA Supports Efficient Transition Without metering TRACONs can become underutilized or overloaded Metering increases throughput

20. 20 CM Sim Data TMA Details CM_Sim : /export/pb Start Time : Tue May 22 14:49:33 2001 Stop Time : Tue May 22 23:15:22 2001 Total Aircraft in cm_sim (including those without crossing or frozen message) : 567 Meter Fix STA to MF STA-Crossing Crossing Host CTAS MF at Crossing Difference Time ACID MF MF TYPE Freeze Time (seconds) ----------- -------------- ------ ----- ----- ---------- --------- --------- 14:50:08Z AAL1001/TUL.B0573 CIVET CIVET MD8 14:49:53Z 14:50:08Z 15 14:50:11Z KAL062/SBGR.B0328 VISTA VISTA B747 14:50:08Z 14:50:11Z 3 14:50:16Z UAL1769/DEN.B0828 CIVET CIVET B757 14:50:08Z 14:50:16Z 8 14:50:38Z EGF214/SAN.B0074 PEBLE DH8 14:50:25Z 14:50:38Z 13 14:51:11Z AAL2821/DEN.B0519 CIVET CIVET B737 14:51:27Z 14:51:11Z 16 14:52:51Z EGF200/SBP.B0658 VTU VTU DH8 14:49:33Z 14:52:51Z 198 14:53:35Z AAL691/LAS.B0295 CIVET CIVET MD8 14:53:27Z 14:53:35Z 8 14:53:53Z SIA7962/PANC.B0567 FIM FIM B74F 14:53:53Z 14:53:53Z 0 14:56:03Z EGF732/SAN.B0621 PEBLE DH8 14:55:52Z 14:56:03Z 11 14:56:18Z MXA912/GDL.B0161 VISTA VISTA B727 14:56:33Z 14:56:18Z 15 14:59:20Z SWA753/OAK.B0148 FIM FIM B73V 14:59:15Z 14:59:20Z 5 15:00:10Z AMX6490/MEX.B0920 VISTA VISTA B757 15:00:12Z 15:00:10Z 2 15:00:13Z N667CX/DAL.B0897 KONZL KONZL G2 15:00:06Z 15:00:13Z 7 15:00:43Z SKW5112/ONT.B0723 LAHAB IARN E120 15:00:46Z 15:00:43Z 3 ………………………………… …………………. …..

21. 21 CM Sim Data TMA Summary Arrival Rate Accuracy Summary CM_Sim : /export/pb Start Time : Tue May 22 14:49:33 2001 Stop Time : Tue May 22 23:15:22 2001 Total Aircraft : 298 Total Total Aircraft Difference Accuracy Crossed STA/Crossing Percentage ---------- ----------- ------------ 298 8 97.32

22. 22 TMA Operational Benefits Improved Center to TRACON feed supports higher airport throughput – MSP up 4%+ – Denver up 2%+ Metering supports delay distribution to more fuel efficient altitudes – Miami delay distribution improved significantly – Atlanta delay distribution improved at front end of rush Internal Departures receive improved ATC services

23. 23 MSP Actual Arrival Peak-Times

24. 24 MSP Daily Cumulative Arrivals 12-18 March 2000

25. 25 MSP Peak Period Cumulative Arrivals 16 March 2000

26. 26 Before TMA vs After TMABefore TMA vs Oct 01 MSP Throughput Post 9/11 Throughput Higher Than Baseline

27. 27 Impact of TMA at DEN Arrival Rate up 1-2 an hour when stressed DEN only stressed ~2% of the time

28. 28 Arrival Demand & AAR at DEN, 23 OCT 00

29. 29 MIA Arrival Streams Outer Arc 180 nmi. Meter Arc 40 nmi.

30. 30 Percentage of Flight Time Spent in TRACON Time flown from meter arc to runway divided by time flown from outer arc to runway Only aircraft crossing meter arc during peaks included Error bars represent 95% confidence intervals Difference is statistically significant

31. 31

32. 32 Impact of TMA on ZLA internal departures to LAX Gate+Air delay down 23% at smaller airports, 10% at LAS For ZLA-released airports with at least 1 flt/day into LAX Based on ASPM data

33. 33 ATL Internal Departures

34. 34 Measuring Operational Benefits of URET Clear evidence of URET usage and increased controller productivity Measurable improvement in distance reduction through route amendments Insignificant improvement in flight times Measurable improvement in efficient altitudes for descending a/c

35. 35 ZME: January 02 Average TPs, Amendments, and Tracked Aircraft Count by ATC Area Avg TP/dayAvg Accepted AM/day Avg Tracked AC/day Data through 26 January 02

36. 36 ZME: Direct Routing Amendments Notes: - Data Sampling: 2 days/week; between 14Z and 22Z - URET 2-way processing began in July 99 - Includes any Lateral Amendment processed by Host Data through 22 March 02

37. 37 ZID data through 27 Mar; ZME through 22 Mar. Distance Savings for Lateral Amendments Notes: - Data Sampling: 2 days/week - ZID between 13Z and 23Z; ZME between 14Z and 22Z - URET 2-way processing began in July 99 - Includes any Lateral Amendment processed by Host

38. 38 EnRoute Distance; shows consistency

39. 39 EnRoute Time w/out Seasonal Correction

40. 40 EnRoute Time with Seasonal Correction

41. 41 Commercial/NRP metrics at ZME.

42. 42 URET Direct Savings Direct Route Route Structure

43. 43 Average Excess Distances Flown

44. 44 Example: Lifting of Restriction 390 370 350 330 310 290 230 250 230 Flight Level 200 Dayton HI (88)Appleton HI (87) TRACON IND 0 98 NMI Trajectory without restriction Trajectory with restriction LO SECTORS 2/24/00: IND arrivals 87/88 at FL310 ZID Airspace Ultra-HI SECTORS

45. 45 History of Restriction Relaxation Status ZID and ZME

46. 46 Exploratory Metrics Analyzing tool benefits during bad weather

47. 47 Traffic Density and Weather

48. 48 Back-ups

49. 49 Raw Database Logs Data Ingestion NCDC Weather Data Ingestion Database Overview Analysis Database Airport Flight Aircraft Metrics Analysis Host/ARTS Data Ingestion Track Weather Contains information for all flights recorded by the respective computer (ARTS or Host) Contains airport AAR, runway configuration, and restrictions Contains winds, visibility, precipitation, etc. at the airport Data Wrapper

50. 50 FFP Metrics – Lessons Learned/What works Terminal Area Performance Changes are measurable: – Increased throughput during peaks indicates increased capacity – Clear Objective Functions: Increased throughput, decreased flight times – Normalization achievable (demand, conditions, etc.) – Automated analyses possible

51. 51 Analyzing Peak terminal Throughput Focus on peaks where throughput is constrained by capacity – During slow traffic periods, there is little or no benefit with new tools Determine when system is stressed – Demand exceeds capacity – Desire to measure throughput not constrained by demand Determine criteria for minimum peak period – May depend on site

52. 52 MSP Actual Arrival Peak-Times (July 2000)

53. 53 Excess Distance Based on ETMS data Draw 100-nmi ring Calculate excess distance compared to great circle

54. 54 Daily Total Excess Distance between city pairs (Top 31 airports) Look closer at selected pairs

55. Representative Delay Base CaseFFP1 Case

56. Representative Cumulative Delay

57. 57 Meter Fix Arrival Peaks Example Day MIA Jan 14, 2001

58. 58 Arrivals into MSP Average Weekday Runway Count During 5-minute Bins (Jan-Mar 2001) Arrival Banks Peaks indicate capacity of ~5 arrivals during a 5-min period (MSP mixes arrivals and departures on 2 primary runways)

59. 59 Average Excess Distance between city pairs (Top 31 airports)

60. 60 Putting It All Together Drop EA/OA Time Since No Delay Apparent Show When Metering Generally Occurs Runway Counts (OA/IA) (IA/MA) (MA/RW)

61. 61 Runway Counts and Flying Times Full Day Runway Counts (EA/OA) (OA/IA) (IA/MA) (MA/RW)

62. 62 Flights On-Time Avg Delay Per Flight Ground Stop Delay Ground Hold Delay % Flown as Filed Avg Daily Capacity OEP/FFP Measures Ground, Terminal and Enroute for Each Flight Time/Distance Flight Time Variance (Predictability) Capacity/ Throughput PBO Measures Cost Per Flight Safety OE/OD/RIs

63. 63 Delay Distribution Runway Counts (OA/IA) (IA/MA) (MA/RW)

64. 64 Measurement & Reporting Objectives Program Management Focuses Efforts – Establishes Expected Outcomes Exercise Management Oversight Future Decision Making Communicate with stakeholders Feedback to service providers, users Customer Accountability Shareholder Accountability

65. 65 What Do We Measure? There are four measures that cover most of what people want to know about NAS performance: 1. Capacity – How many aircraft are possible? 2. Peak Throughput – How many can get there? 3. Flight Time – How long does it take to get there? 4. Variation of Flight Time – Can we plan on it? These measures can be applied to overall NAS performance as well as Terminal or En Route

66. 66 Runway Counts and Flying Times Full Day Runway Counts (EA/OA) (OA/IA) (IA/MA) (MA/RW)

67. 67 URET at ZKC IDU 12/3/01 As of 3/28/02, Over 55% of Workforce has Completed URET Training (Over 8,000 Training Hours) Benefit Anecdotes: – 20 minute conflict probe works good – Increased time for strategic planning – More direct routings are being given

68. 68 URET Daily Use Metrics at ZID and ZME Count of Directs – Define Direct to be all route amendments which reduced the remaining flying distance – For busy hours (ZID: 13Z - 23Z; ZME 14Z-22Z) – Since May 99, processing 2 days/week Lateral Distance Saved – For all lateral amendments (includes directs as well as penalties), the average of the daily sum of nautical miles changed.

69. 69 ZID: Direct Routing Amendments Notes: - Direct Data Sampling: 2 days/week; between 13Z and 23Z - URET 2-way processing began in July 99 - Includes any Lateral Amendment processed by Host Data through 27 March 02

70. 70 Related Activities ZID – A URET benefits team has been created, includes airline participation. – Any proposed new airspace restrictions will be evaluated by the URET benefits team for validity, prior to implementation. ZME – ZME controllers have started dynamically lifting crossing restrictions for SDF arrivals before the formal evaluation is in place.

71. 71 Arrivals into MSP Average Weekday Runway Count During 5-minute Bins (Jan-Mar 2001) Arrival Banks Peaks indicate capacity of ~5 arrivals during a 5-min period (MSP mixes arrivals and departures on 2 primary runways)

72. 72

73. 73 EnRoute Time -- East/West city-pairs

74. 74 Extrapolated Savings ZID Feb. 24-25 Restriction Relaxation Evaluation – Per Restriction: fuel saved ~113 gallons Evaluation period represents approximately 40% of total traffic for day – Per Day: potential fuel saved: ~282 gallons – Per Year: potential fuel saved: ~102,930 gallons – ZID identified restrictions for possible removal testing (~70-368) – Per Year: potential fuel savings: ~7,205,100 gallons

75. 75 Methodology Delays - Where, When, and How Much Airspace around MSP divided into imaginary rings Data collected on flights entering these rings – Times at which aircraft cross each arc – Flying time through each ring