Framework for Performance Metric Development draft-morton-perf-metrics-framework-01.txt Alan Clark IETF 70 PMOL WG

Framework draft Background and motivation behind this draft and PMOL Scope – “PM Entity” however Framework “could be useful” to other WG’s Metric Development –Guidelines, criteria for defining a metric Performance metric development process –Process for proposing, reviewing metrics

Metrics Development View metric from “audience” perspective Definition of “metric” –measure of an observable behavior of an application, protocol or other system Tests of “Usefulness” –Essential to understand performance and find problems? –Correlates with performance as it affects the “user” –Helps with detecting, diagnosing, locating problems

Metrics Development Composed metric –E.g. IPPM work on spatial and temporal composition Metric specification –Process for specifying a metric (tbd) Classes of metrics –Having classes of metrics may simplify definition process Qualifying metric (testing definition of metric) –Ambiguous?, Defined Units?, Measurement Error Defined?, Repeatable?, Implementable?, Assumptions made about measured process?

Metrics Development Reporting models and impact on measurement –E.g. assumptions concerning sampling, averaging, window sizes…. Dependencies – Relationship between variance in “base” metrics and variance in derived or composed metrics

Performance Metric Development Process New proposals –Prepared as I-D’s with metric definitions that conform to previous slides –SHOULD’s be vetted by WG with domain expertize Include assessment of relevance/ interaction with other standards groups (both other WG’s and other standards organizations) Specify intended audience for metric Proposal approval –Process TBD – IETF/ IESG/ AD’s/ Relevant WGs,/ PM Entity…… Interaction with other WGs –Partnership with WG with domain expertize Standards Track performance metrics –Steer drafts through process

Comments, Questions, Next Steps? Potential additional definitions? –Index? –Intermediate? Other comments?