A Deterministic End to End Performance Verification Architecture Jerry Sobieski NORDUnet October 20, 2012

The Problem Emerging Connection Services that offer “guaranteed” performance require a means of: A) determining if a Connection is performing as requested B) determine where (and why) a Connection is failing Guaranteed service require substantially

Deterministic Performance Verification Can we deterministically measure the performance of real traffic across a domain? Can we do so without perturbing the flow? Can we do so in such a fashion that we can determine where along the path performance problems are occuring? Network domain STP-A STP-Z How do we verify the throughput from STP-A to STP-Z?

How do we measure traffic across a domain? Existing models pose specialized performance measurement servers Active measurements are performed that replace or perturb existing flows – the flows we want to understand! Traffic characterization can only be measured between these specific Measurement Points (MPs). the MP to MP path often includes path segments and/or network elements that are not directly part of the path of interest Some tests often transit components with indeterminant performance characteristics E.g. a “Ping” incurs indeterminant latency based upon processor loads.

What do we need to do better? Passive measurement Measure the actual user data flows instead of artificial flows Consistent measurement points Architecturally predictable measurement locations.. Everywhere. Architecturally deterministic measurement points themselves… simple, cheap, and effective. Highly accurate timing What resolution is required to characterize 100 Gbps packet flows? Appropriate software tools Capture/analysis Automated User and/or Operations oriented Secure

A “Performance Flow Correlator” PV Flow correlator 4 5 Flow sample (Realtime/background, step/full sampled) 3 Ingres Flow “tap” (splitter/hdw replication) 2 Circular Flow buffer (time stamped capture servers) 1 Ingress flow Egress flow STP-1 STP-2 user flow Border switch Border switch External domain External domain Intra-Domain Flow Correlation

How the Flow Correlator Works The “flow tap” design can be implemented at every domain boundary, at every interface. Such mirroring capabilities are often already incorporated in switching interfaces – just never used for systematic operational monitoring and fault localization Optical Passive splitters can be easily inserted inline at the interface where the device does not have such capabilities Electrical signals can also be tapped using mirror ports. The tap, when enabled, leads directly to a local flow buffer. The flow buffer can be sized and configured to capture an entire flow, or it can be sized to sample flows according to some rule or policy. The correlation can be done in real time if engineered to do so… or the flow can be captured and stored for later background analysis. … a few seconds later, or a few days later. Correlation can be performed periodically, using short samples or real flows

An inter-Domain “Flow Correlator” PV Flow correlator Flow samples are FTP’d (background processing) or streamed (real time processing) to correlator The inter-domain flow transfer Should use dedicated circuits to avoid affecting other traffic Inter-Domain Transport Infrastructure Flow buffer Flow buffer Egress flow Ingress flow STP-A user flow STP-Z Border switch Border switch Inter-Domain Flow Correlation Source Domain Destination Domain

End to End PV Architecture Aruba Bonaire Curacao Dominica MP-B1 MP-C2 MP-D1 MP-D2 MP-A1 MP-A2 MP-B2 MP-C1 B Stp Z Stp A D C This data plane architecture can deterministically localize faults to a particular domain. Automated agents can perform the analysis NSI Query() primitive can be used to provide both the multi-domain path and the the boundary STPs A simple directory can map STPs to Flowbuffers Or an active agent (NSA?) can be requested to validate path performance.

A look at the border switch Each STP must be able to mirror its specific data stream to the flow buffer Thus, physical channels must be tapped (fiber, UTP, etc.) And virtual channels must be tapped (vlans, LSPs, TDM channels, etc.) The tap should be placed as close to the ingress/egress STP as possible. Outside of elements such as buffers or framers Doing so provides for better fault localization. The mirrored flow must preserve performance fidelity between the tap and the flow buffer capture interface (i.e. no mirror aggregation or buffering prior to capture.) The flow buffer should be sized to process the number and size of simultaneous flows it will be expected to process

Timing Simple packet recording at source and destination can provide accurate packet loss insights even without timing But not performance fidelity Inter-packet arrival times can be measured accurately locally at each end Can reveal jitter characteristics after correlation Presta packet capture cards can do 10 nS time stamping at 10 Gbps. These cards need an external [accurate] clock Other similar cards can also do packet filtering and other intelligent offloading. Could be useful for sourcing streams with very accurate pacing, or payload signatures.

Highly accurate distributed timing Emerging techniques in metrology allow extremely accurate timing to be distributed though the photonic network. Accurate to 10^-17 seconds (!) Advanced NICs can be field programmed to time stamp packets to 10 nanoSecond resolution Driven by external [distributed] clock A 64 byte packet arrives every 512 nS on a 1 Gbps link A 64 B packet arrives no more than every 51 nS on a 10 Gbps link. E2E packet Latency can be computed with distributed global clock to 10 nS intervals…for each packet in a large flow (!)

Interdependence NSI SDN