1. May 15-17, 2002 National Aeronautical and Space Administration and Volpe National Transportation Systems Center, US Dept. of Transportation Performance Measures Approach for NASA Virtual Airspace Modeling and Simulation (VAMS)

2. 2 Table of Contents » VAMS Introduction » Measure Requirements, Message, and Framework » NASA VAMS Measure Approach NASA VAMS Performance Measures Security

3. V IRTUAL A IRSPACE M ODELING AND S IMULATION Security AS Virtual Airspace Modeling & Simulation - Project Readiness Review

4. Project Vision The Virtual Airspace Modeling and Simulation Project provides the technologies and processes for conducting trade-off analyses amongst future air transportation systems concepts and technologies Modeling & Simulation Tools Operational Concepts Evaluation Methods & Techniques AS Virtual Airspace Modeling & Simulation - Project Readiness Review

5. Project Goals & Objectives l Develop the capability to model and simulate behavior of air transportation system concepts and their elements to never- before-achieved levels of fidelity »Develop a set of analytical and computational models and methods to conduct detailed assessments of candidate operational concepts »Establish simulation capability that will enable safe investigation of complex advanced air transportation concepts, and develop a deeper understanding of human performance interaction within it l Develop advanced air transportation concepts »Develop a set of potential operational concepts, concepts of use, and architectures, providing definitions of the future air transportation system and its elements »Develop technology roadmaps to achieve these concepts l Conduct assessments of advanced air transportation concepts »Address potential benefits, identify risks and limits, and evaluate performance, safety, operations, and National Airspace System infrastructure and transition challenges AS Virtual Airspace Modeling & Simulation - Project Readiness Review

6. AS VAMS Technical Objectives Develop and validate modeling and simulation tools providing the multi-objective (safety, capacity, cost) trade space to analyze air traffic management (ATM) concepts to meet forecasted demands of the 2020s Develop operational scenarios, metrics and evaluation methodologies to assess potential operational concepts and technologies to meet the forecasts across the trade space Create operational concepts and conceptual architectures that can be used to define the next generation air transportation system, and develop technology roadmaps, to meet long-term Enterprise goals

7. 7 Table of Contents » VAMS Introduction » Measure Requirements, Message, and Framework » NASA VAMS Measure Approach NASA VAMS Performance Measures

8. 8 Performance Measures Support Decision-Making and Communication Actionable performance measures for VAMS provide knowledge to support: l Decision-making, e.g.: » Concept is worthwhile to pursue » Suggest refinements to a concept l Communications, e.g.: » Communicate with FAA about benefits, cost, and feasibility of concepts » Tell external stakeholders expected benefits and costs of a concept » Convey program results or status to top management » Tell OMB and Congress about program

9. 9 Aviation Performance Measure Problems Some difficulties in determining aviation measures: l Variety of decision-makers and stakeholders l Difficult to define a few measures that capture impacts of interest to all stakeholders l Measures are often not easily understandable l Measures often do not give direct picture of end-benefits l Measures are often not useful for decision-making l Often difficult to utilize qualitative measures (e.g., survey results or anecdotal information) in creditable way

10. 10 Developing Actionable Aviation Performance Measures Keys to developing actionable performance measures: Identify requirements for measures to support decision- making and communication Define the message the measures should tell to support decisions and communication Develop a narrative framework for the measures to present all necessary information and aid understandability Include both quantitative and qualitative measures where appropriate

11. 11 Characteristics of Volpe Approach l Identify measure requirements to provide information desired by all decision-makers & stakeholders l Define message performance measure are to convey that is understandable by all decision- makers & stakeholders l Develop framework to »Present direct impact measures, which are usually calculatable, and relate these to end-user benefits, which are often difficult to calculate »Combine observed data with estimated data »Use survey or anecdotal data where useful to make the case (i.e., include quantitative and qualitative measures).

12. 12 Designing Actionable Performance Measures Develop Actionable Performance Measures Identify decisions Identify information needed Top-Down Requirements: What information is needed? Bottom-Up Requirements: What are the potential impacts and what data is available? Interactive Process Identify decision- makers & stakeholders Identity potential project impacts Identify data sources Design the measure framework and measures to tell the message

13. 13 Decision-Makers and Stakeholders: Requirements What are the desired uses of performance measures by each decision-maker and stakeholder? e.g.: l NASA: Promising concepts and technologies to pursue l FAA: Promising concepts to support l Air carriers: Impacts of potential concepts on their operations, revenues, and costs l Manufacturers: Impacts of potential concepts on their products, revenues, and costs l Pilots: Impacts of potential concepts on their tasks l Air traffic controllers: Impacts of potential concepts on their tasks l General aviation: Impacts of potential concepts on their operations, access to services, and costs l Cargo carriers: Impacts of potential concepts on their operations, revenues, and costs (contd... )

14. 14 Decision-Makers and Stakeholders: Requirements (contd) What are the desired uses of performance measures by each stakeholder? e.g. (contd): l Cargo carriers: Impacts of potential concepts on their operations, revenues, and costs l U.S. Government » Executive Branch: Office of Management and Budget: Benefits and costs; feasibility and directions of concepts; relation to related NASA programs » Congress: Benefits and costs to stakeholders; feasibility and directions of concepts l Other countries: Compatibility impacts l Airport operators: Impacts of potential concepts on their operations, revenues, and costs l Flying public: Air travel service, safety, security, & travel costs l General public: Noise and air pollution

15. Capacity EfficiencyPredictability FlexibilityEnvironment Direct aircraft operator costs AATT Economic Measures AATT Top-Level Performance Measures Domain- Specific (Terminal) Performance Measures Tool-Specific Performance Measures for PFAST Example 1: Measure Hierarchy – NASA AATT Project Total flights flown Total aircraft travel time Total aircraft miles Number of flights more than 15 minutes late from scheduled arrival time # of user requests honored (measured in surveys) Emissions Noise Arrivals – (at defined set of airports) Average number of airport arrivals per hour during peak periods Total number of airport arrivals per year Departures – (at defined set of airports) Average number of airport departures per hour during peak periods Total number of airport departures per year Average number of arrivals per hour, per runway, during peak periods

16. 16 Example 2: Measure Flow - FAA Safe Flight 21 Measures Capabilities Direct Output Measures End-Benefit Outcome Measures End- Benefits Direct Impacts CapabilitiesDirect Impacts Display in cockpit of surrounding traffic/equipment Pilot able to better identify aircraft to follow Pilot awareness of all proximate traffic positions Direct Output Measures Pilot response time for ATC traffic call-out Flight time from final approach fix to touchdown Benefit Impacts Reduced arrival delays Increased predictability of arrival times End-Benefit Outcome Measures Safety Accident rate during final approach maneuvers User Cost Savings/Revenue Enhancement Arrival rate FAA Cost Savings · Voice channel occupancy time

17. 17 Table of Contents » VAMS Introduction » Measure Requirements, Message, and Framework » NASA VAMS Measure Approach NASA VAMS Performance Measures

18. Operational Scenarios Fast-Time Modeling Concepts Real-time Simulations 1. Scope: issues NAS Domain challenges assumptions 2. Top Level Descriptions: core ideas functions 3. Detailed Descriptions: performance roles, responsibilities @ humans & machine human factors user interfaces 4. NAS infrastructure & technology impacts: transition planning architecture technology requirements Empiric Analysis (i.e. expert opinions) Direct Output Measures End-Benefit Outcome Measures Stakeholder Viewpoints (questions to be answered) Number of traffic events (takeoffs, sector crossings, landings, etc.) Number of communication events (requests, clearances, directives, etc.) Throughput (traffic volume) Delay in phases of flight Safety incidents (proximity to minimum separation, incursions, encroachments, etc.) Elapsed flight times Fuel burn by phase of flight Personnel workloads Etc. Scenario Elements: NAS Domain NAS Perturbations (e.g. Wx, Security Incidents) Origin/Destination Demand Assumed Technologies Human/Machine Performance Defined ATM Procedures Assumed Equipage Fleet Mix Etc. Stakeholder Viewpoints (questions to be answered) Average aircraft flight time per air route* Operational cost per flight mile Average airport arrival rate during peak periods Average taxi time from pushback to wheels up during peak traffic periods per specific airports or taxi paths within airports Average voice channel occupancy time per departure from pushback to take off Fuel usage Aircraft maintenance costs per flight mile Etc. * a defined city pair air route Analysis Framework for VAMS Evaluation

19. 19 Constraints to Achieving Capacity To foster a standardized concept evaluation approach, we will develop measures that address common categories of constraints on aviation capacity: »ATC complexity »Choke points »Weather »Pilot/Controller/Operator limitations related to performance and technology »Airport design »User scheduling »Policy on airport/airspace use »Aircraft performance jack perkins: To foster a standardized concept evaluation approach that supports theVAMS primary goal of increasing NAS capacity, we will develop metrics that address …? jack perkins: To foster a standardized concept evaluation approach that supports theVAMS primary goal of increasing NAS capacity, we will develop metrics that address …? jack perkins: To foster a standardized concept evaluation approach that supports theVAMS primary goal of increasing NAS capacity, we will develop metrics that address …? jack perkins: To foster a standardized concept evaluation approach that supports theVAMS primary goal of increasing NAS capacity, we will develop metrics that address …? jack perkins: To foster a standardized concept evaluation approach that supports theVAMS primary goal of increasing NAS capacity, we will develop metrics that address …? jack perkins: To foster a standardized concept evaluation approach that supports theVAMS primary goal of increasing NAS capacity, we will develop metrics that address …? jack perkins: To foster a standardized concept evaluation approach that supports theVAMS primary goal of increasing NAS capacity, we will develop metrics that address …? jack perkins: To foster a standardized concept evaluation approach that supports theVAMS primary goal of increasing NAS capacity, we will develop metrics that address …?

20. 20 Potential Evaluation and Measure Approach 1. Identify capacity constraint(s) being targeted by concept 2. Define concept functionality and map functionality to the targeted capacity constraint 3. Develop/assign measures to concept evaluation »Direct output measures »End-benefit outcome measures 4. Define operational scenario(s) for the concept evaluation 5. Evaluate concept »Fast-time modeling »Real-time simulation »Empirical assessment

21. 21 Example – Terminal Arrival Maneuvering for Weather Avoidance Concept functions: l Onboard weather detection and route planning equipment enable flight crew to autonomously navigate around weather cells and maintain separation from terminal area traffic at ATSP discretion l Unequipped following aircraft would be authorized by ATSP to perform self-spacing from an equipped lead aircraft. l Equipped lead aircraft broadcasts trajectory intent information to all local traffic via datalink. Impacted constraint categories on aviation capacity: l Weather

22. 22 Example – Terminal Arrival Maneuvering for Weather Avoidance (contd) Direct Output measures l Pilot confidence in equipment in weather cell avoidance l Pilot confidence in equipment for accuracy in displaying terminal area traffic positions l Average terminal arrival flight time for equipped aircraft compared non-equipped during peak traffic periods l Average number & duration of ATC arrival vectoring callouts to equipped flights during arrival maneuvers impacted by weather cell avoidance l Percentage of flight crew preferred arrival routes made possible during testing (peak arrival periods) End-Benefit Outcome measures l Airport annual arrival rate during peak periods l Total number of airport arrivals per year l Average/Total voice channel occupancy time per frequency per terminal devoted to communications involving arrival vectoring during peak traffic periods l Total and standard deviation of flight time from 250 miles out to touchdown during peak periods l Total fuel consumption (all flights) per year from 250 miles out to touchdown

23. 23 Addendum Examples of Performance Measures NASA VAMS Performance Measures » NASA Advanced Air Transportation Technology Program » FAA Safe Flight 21 Program » Federal Railroad Administration Safety R&D Program » FAA R&D Program

24. 24 Examples of Actionable Performance Measure Narrative Frameworks Example 1: NASA Advanced Air Transportation Technologies (AATT) Project l Issue: AATT has many projects with different impacts. l Desired Narrative: Show how the AATT projects form a coherent program to support goals. l Solution: Hierarchical measure structure. (contd... )

25. Capacity EfficiencyPredictability FlexibilityEnvironment Direct aircraft operator costs AATT Economic Measures AATT Top-Level Performance Measures Domain- Specific (Terminal) Performance Measures Tool-Specific Performance Measures for PFAST Example 1: Measure Hierarchy – NASA AATT Project Total flights flown Total aircraft travel time Total aircraft miles Number of flights more than 15 minutes late from scheduled arrival time # of user requests honored (measured in surveys) Emissions Noise Arrivals – (at defined set of airports) Average number of airport arrivals per hour during peak periods Total number of airport arrivals per year Departures – (at defined set of airports) Average number of airport departures per hour during peak periods Total number of airport departures per year Average number of arrivals per hour, per runway, during peak periods

26. 26 Examples of Actionable Performance Measure Narrative Frameworks (contd) Example 2: FAA Safe Flight 21 Program l Issue: Decide which Safe Flight 21 projects provide meaningful benefits. l Desired narrative: Show how the results of operational evaluations, while not directly measuring end-process benefits, indicate benefits will be achieved. l Solution: Develop flow of measures showing benefit mechanisms. Describe SF21 Capabilities Develop Direct Output Measures Develop End- Benefit Outcome Measures Identify End- Benefits Identify Direct Impacts Measure during operational evaluations to indicate whether benefits are likely to be achieved Estimate projections (contd... )

27. 27 Example 2: Measure Flow - FAA Safe Flight 21 Measures Capabilities Direct Output Measures End-Benefit Outcome Measures End- Benefits Direct Impacts CapabilitiesDirect Impacts Display in cockpit of surrounding traffic/equipment Pilot able to better identify aircraft to follow Pilot awareness of all proximate traffic positions Direct Output Measures Pilot response time for ATC traffic call-out Flight time from final approach fix to touchdown Benefit Impacts Reduced arrival delays Increased predictability of arrival times End-Benefit Outcome Measures Safety Accident rate during final approach maneuvers User Cost Savings/Revenue Enhancement Arrival rate FAA Cost Savings · Voice channel occupancy time

28. 28 Examples of Actionable Performance Measure Narrative Frameworks (contd) Example 3: Federal Railroad Administration (FRA) Safety R&D Program l Issue: Why are projects selected? l Desired narrative: Present rationale for prioritizing and selecting projects. l Solution: Develop measures to reflect decision criteria for selecting projects in the program portfolio. » Present projects and criteria measures in matrix » Values of criteria measures can be expressed as High, Medium, or Low for quick analysis and easy interpretation. (contd... )

29. 29 Example 3: Measure Criteria Matrix - FRA Safety R&D Program

30. 30 Examples of Actionable Performance Measure Narrative Frameworks (contd) Example – FAA R&D Program (Draft) l Requirements: Help prioritize and select projects; integrate project view; show project contributions to goals l Message: R&D program has a coherent set of projects that support goals l Measure Framework: Hierarchical framework (shows how activities relate to goals)

31. 31 AIR TRAFFIC SVCS.AIRCRAFT SAFETYAIRPORTS SECURITY AEROMED R&D MGMTCOMM. SP. FAA STRATEGIC GOALS FAA SUPPORTING GOALS FAA PERF. OBJ. ------------------------ R&D CHALLENGE R&D STRATEGY R&D PROJECTS STRATEGIC PLAN NARP ENVIRON. SAFETY SECURITYEFFICIENCY ENV. & ENERGY Risk Analysis: Reduce aviation system risks Prevention: Prevent aviation accidents Mitigation: Mitigate aviation system accidents Flight Environment Human Performance Aircraft Systems Terminal Area Commercial Space Risk In-Flight Incidents Crash Survivability Post-Crash Response R&D PROGRAM Example 4: Measure Hierarchy - FAA R&D Program (Draft)

32. 32 Contact Information Sandy Lozito Manager, System Evaluation and Assessment Virtual Airspace Modeling and Simulation (VAMS) Project NASA Ames Research Center M/S 262-4 Moffett Field, CA 94035-1000 650-604-0008 FAX 650-604-3729 slozito@mail.arc.nasa.gov James L. Poage Operations Assessment Division, DTS-43 Volpe National Transportation Systems Center Kendall Square Cambridge, MA 02142 617-494-2371 FAX 617-494-2371 poage@volpe.dot.gov