The Value of Technical Performance Measures Ann Lynn The Boeing Co. (314) 234-9714 ann.b.lynn@boeing.com All Data in this pitch is Notional – for example only

Agenda TPM Definition Typical Types of TPMs Constructing TPM Profiles Performance Thresholds Plan Over Time Uncertainty Management Process Allocation of Margin Roll-up (Parent/Child) TPMs Value to a Program

Technical Performance Measures Represent critical technical thresholds and goals for success of the program Should be variable and responsive to engineering changes Usually exist in a hierarchy corresponding to the spec levels (system, subsystem, component…) Have target values corresponding to specification values that is, your TPM should be a parameter in a spec at some level Are tracked and statused against their defined plan and represent the technical health of the program

Typical Types of TPMs Parameters that flow from and/or support KPPs Key Constraints and Performance Reqmts Weight, power, heat load Reliability/ Maintenance parameters System / subsystem performance Endurance Range Latency Accuracy Efficiency Etc Cost (e.g. AUPC) Parameters associated with Program Technical Risks or areas of technical challenge TPMs should represent a balanced set of key parameters; Don’t monitor something just because “you can”

Example TPM Status Chart Example Monthly / Quarterly Status This view shows: •Title/Definition •Status against spec •Status Color (RYGB) •Maturity of the status

Definition of Performance Thresholds (Generic) Blue – the system exceeds requirements by more than x (and may be subject to re-allocation) Green – the system meets all performance requirements Yellow – the system can meet some objectives but does not meet all performance requirements Red – the system cannot perform satisfactorily TPM margin Spec value TPM margin This is the standard method for defining TPMs. Note that the margin below the spec takes the paramter from “meets” (green) down to the bottom of the yellow zone. That way, when a parameter is not meeting its spec, you ask “how much margin is there?” meaning how much “yellow zone” do you have before that parameter hits red. Where yellow zone is that area where you can negotiate yoru allocated requierments to try to still achieve objectives & goals of the program. Typically setting the spec value to the “hairy edge” of needed performance is considered too risky and doesn’t’ allow the opportunity to use the TPM process Mechanisms for the yellow and red zones. Note, reqmts can be negotiated/updated with corresponding updates to Specs and TPMs. Setup as shown, the parameter has to meet the spec to be green.

TPM Profile Over Time This view of the TPM allows you to see history and trends in performance

Profiles over time (cont.) TPM Spec Status Unit Efficiency 24% 22.2% (Meas.) In this example, the team receives samples from the supplier and expects incremental improvement over time to eventually achieve the spec value. If each sample comes in at its target value, the TPM status should be Green. Therefore the green range changes over time in accordance with the plan. TPMs are set up as “performance to plan”. Note that this type of graph is not intended to reflect changes to spec requirements. TPMs are spec-based, so if the spec changes then the spec line would be adjusted and the threshold values redone so that spec is still the bottom of the green. TPM using current status, not predicted status at end of program For this TPM, performance is planned to improve over time. If actual progress >= planned, you are green, even if you haven’t achieved the end-state target (spec value)

Weight Management Projected weight (projected value at end of program) includes a historical growth factor Current Projected Wt. (w/ Growth Risk) 3971.3 lb Spec NTE Weight 3850 lb TPM if using current status TPM Spec Status System Weight 3850 3588 (current) IPT Weight Allocations 3588 lb Current Baseline Wt. 3588.0 lb TPM if using projected status Baseline weight reflects current configuration sized to loads at NTE weight requirement. Projected weight includes planned design changes and X% weight maturity growth risk for post PDR design maturity. TPM Spec Status System Weight 3850 3971 (projected) Weight TPM usually plotted per the Weight Mgmt process; Show both values, status to Projected; 9 9

TPM Profile with uncertainty bar include an “uncertainty” bar for each reported value Spec value line If your project has significant Risk, or it is early in the project, TPM status can be deceiving unless uncertainty is understood

TPM Management Process (Generic) In the blue: consider adjusting that parameter’s spec value and taking relief elsewhere In the green: press on! TPM margin Spec value In the yellow, or alarming trend: take a program-defined action (put on Risk Watch list, initiate reqmts/design trade, establish new risk mitigation, initiate change to design, reqmts, or plan) TPM margin These process mechanisms allow the program to define set actions based on the severity of concern regarding the status. In the red: take a program-defined action (establish new risk mitigation, initiate change to design, reqmts, or plan) The approach to monitor and control is predicated on correct allocations of margin

Margin

Definition of TPM Margins Factors to consider: Margin may be allocated based on the amount of risk (e.g. Red Risk or low TRL subsystem gets a greater share of margin) Margin may be allocated based on relative impact to System performance (accounts for sensitivities) Margin may be limited by the “hard points” - those parameters that have a hard stop which prevents acceptable operation (e.g. structural limits for weight) TPM margin Spec value TPM margin Margin: How bad can it get before status goes red? How good must it be before status goes blue?

*System Parameter* Tree Power Consumption Energy Collection SPC efficiency Propulsion Consumption Subsys Pwr Consump Energy Storage VMS / MS Prop efficiency AV Weight Cruise consump Subsys Z Aero efficiency VMS ESS RT efficiency maneuvers Subsys Z ESS components Drag A components ESS degradation Stability ESS B components ESS param Power Distribution Subsys A Subsys B EPD efficiency Propulsion PPX efficiency Subsys X Convertor efficiency High Sensitivity Lower sensitivities This diagram maps the subsystem parameters that contribute to the system-level parameter; Sensitivities are noted. Wiring efficiency

Associated Risks *Sys Parameter* This diagram maps Risk to Power Consumption Energy Collection 46-xxx SPC efficiency Propulsion Consumption Avionics & PSS Consump 24-xxx Energy Storage VMS Prop efficiency 1-xxx AV Weight Cruise consump Subsys Z Aero efficiency VMS 57-xxxx ESS RT efficiency 41 - xxx maneuvers Subsys Z ESS components Drag 54-xxx 7-xxx 18-xxx A components ESS degradation 51-xxx Stability ESS 6-xxx B components 50-xxx ESS param Power Distribution Subsys A 2-xxx Subsys B Shows Risk #, Title, and current level (RYG) EPD efficiency Propulsion 56-xxxx PPX efficiency Subsys X 56-xxxx Convertor efficiency High Sensitivity Lower sensitivities Subsys x param Wiring efficiency This diagram maps Risk to the subsystem parameters. Helps show where margin is needed.

Interdependent TPMs Competing for their share of the overall margin Their performance needs to be monitored as a group Change (re-allocation of margin, spec adjustment) has to be managed for the group

KPPs and TPMs KPPs TPMs System level Sub- & Cmpnt Time of Year at which Energy balance is closed TPMs System level Sub- & Cmpnt *System Parameter* (AV Efficiency at End of Mission, AV Performance) AV Weight AV Power Consumption Propulsion eff ESS round trip eff PPX eff SPC eff AV Aero eff Propulsion Power consump VMS Power consump Airframe Power consump ESS Power consump Subsys A Power consump Subsys Z Power consump VMS weight Propulsion weight -Airframe weight ESS weight PPX weight Subsys A weight -Subsys Z weight KPPs represent customer operational objectives; TPMs shown represent measurable design parameters that contribute to the KPP

Relationship of System and subsystem TPMs Example: Command Latency <= 160 ms If supplier specs are more stringent, that means margin is held at subsystem level above supplier Spreadsheets can help account for values and margins at each level Many PMs prefer to monitor Supplier-owned TPMs when available System margin Weapon Lnchr SMS MC PVI 25 5 25 25 50 30 PVI CMPNT1 Spec = 35 ms Weapon Spec = 20 ms Launcher Spec = 5 ms PVI CMPNT2 Spec = 15 ms

Example parent/child TPMs (1 of 3) Owner TPM Title Spec Value Current Status CE System Command Latency 160 ms 136 ms Dsplys ^PVI Cmd Latency 50 ms Msn Cmptr ^MC Cmd Latency 25 ms Stores ^SMS Cmd Latency ^Lnchr Cmd Latency 5 ms ^Weapon Cmd Latency 31 ms System TPM can be green even when child (subsystem) TPM is red; in this case because of margin held at system level

Example parent/child TPMs (2 of 3) Owner TPM Title Spec Value Current Status CE System Command Latency 160 ms 121 ms Dsplys ^PVI Cmd Latency 50 ms 35 ms Msn Cmptr ^MC Cmd Latency 25 ms Stores ^SMS Cmd Latency ^Lnchr Cmd Latency 5 ms ^Weapon Cmd Latency 31 ms System TPM status can be green if child (subsystem) TPMs balance out

Example parent/child TPMs (3 of 3) Example: Command Latency <= 160 ms System margin Weapon Lnchr SMS MC PVI 25 5 25 25 50 30 35 20 Weapon Spec = 25 ms Launcher Spec = 5 ms 35 Over time, System TPM margin may need to be re-allocated to compensate for subsystem over target Subsystem TPM in the blue range may have spec adjusted and “give back” margin to the System level or to another subsystem

TPMs - Takeaways Create a balanced set of key parameters Ensure the full Profile is understood by program technical leadership, even if only the stoplight format is used for periodic reviews Profile over time current vs projected status Basis for margin  maturity / uncertainty Assess the risk and sensitivity of parameters to support appropriate margin allocation Know where your margin is … and isn’t, so you can react to changes in reqmts or in design status TPMs can be used as a predictor of success … and failure

The Value of TPMs Track the “right” things Make sure your TPMs correlate to key technical parameters Track technical progress compared to planned Your monitoring system is only as good as the plan & thresholds you set it up with Spend the time upfront to set up correctly! Your monitoring system is no more accurate than the data you put in maturity / uncertainty of reported data must be evaluated Conisder plotting uncertainty bars on data points to avoid false sense of security