The ARFTG/NIST Measurement Comparison Program Kate A. Remley and Robert M. Judish NIST John W. Cable and Yeou-Song (Brian) Lee ARFTG Standards Committee

ARFTG/NIST Measurement Comparison Program Why compare measurements? Validate labs’ measurement capabilities Basis for mutual recognition Measurement Assurance – Proficiency testing, accreditation – Validation and confidence ARFTG/NIST Program Compare VNA Performance Ongoing program started in 1990 Administered by: – ARFTG (Automatic Radio Frequency Techniques Group) – NIST (National Institute of Standards and Technology)

Few Questions about Traceability?  Is it correct to say that measurements or standards are traceable?  Only measurement results and values of standards are traceable.  What do I need to support a claim of traceability?  A defined quantity (e.g. power)  Measurement system or working standards  Measurement result or value with a documented uncertainty  Reference standards  Internal measurement assurance program - Source of Information: NIST web site

Process Control for Standards  Measure Standards on more than one system  Accumulate a Large Data Base of Measured Values  Reduce Uncertainty by Averaging Measurement  Track System Performance by recording standards measurement data and error terms  Day-to-day: Measure working standards  System Specifications: Measure Traceable Standards  Inter-laboratory Measurement Comparison (ARFTG)  NIST Traveling Kits

VNA Measurement Traceability Assurance Airline (i.e., Partially Supported Airline) Reference Standards Calibrated ANA Reference Standards Calibration Kit Working Standards Verification Kit Reference Standards NIST Airline (i.e., Beadless Airline) Reference Standards Calibrated ANA Reference Standards Calibration Kit Working Standards Verification Kit Traveling Standards Calibrated ANA Reference Standards Calibration Kit Working Standards Verification Kit Working Standards Users Anritsu National Standards

VNA Measurement Accuracy Quality of standards/connectors Calibration method Hardware stability Environmental stability Operator skill VNA Measurements require calibrations Vector Network Analyzer Calibration Standards

How is accuracy achieved? VNA is a very accurate instrument if it is properly calibrated. Measurement accuracy of a VNA is largely determined by the calibration (accuracy enhancement) process*  Raw (before calibration) errors 2%-80%  Residual (after calibration) errors 0.1%-2%  Connectors are a significant source of errors particularly at the higher frequencies. * Source of information: John Juroshek/NIST

Comparison Program Objectives Compare VNA measurements of S ij Provide confidence to participants Identify measurement problems Developed to assess industry reproducibility Validation of measurement capability Doesn’t provide traceability to NIST: NIST is a blind participant A comparison of S11 measurements from 54 labs

Program Specifics See list at end of presentation for contact names Lab names are confidential: reported only to NIST Artifacts M & F Offset Shorts Airline Mismatched 2-port 20 & 40 dB attenuators Connector types 7-16 GPC-7 N 3.5 mm 2.92 mm (K) 2.4 mm MCP supports 6 connector types Each connector-type kit contains six artifacts Three measurements made, averaged of each artifact 3.5 mm kit

How much data do we have? GPC Type N mm mm mm Connector Type Frequency Range (GHz) # of labs Data per lab As of March, 2003:

Analyzing the Data “Classical” Statistical Bounds with 95% confidence intervals Gaussian Bounds “Average” value Standard deviation Gaussian distribution

Classical Bounds What happens if one lab reports wildly different results? Roughly 95% of measurements should fall between ± 2 

Robust Bounds We develop an automatic method to account for outliers Robust statistics resist the effects of outliers

Robust Statistical Bounds “Median” value MAD Robust Bounds Median and MAD Sample measurements Median = “middle value” MAD = median absolute deviation Robust estimate of  = 1.48xMAD

Effects of Outliers Data “Well Behaved” Data: 4, 5, 2, 6, 5 Mean: 4.4 Median: 5 | x i - m |: 1, 0, 3, 1, 0  :1.5  MAD :1.48 Data: 4, 5, 2, 6, 15 Mean: 6.4 Median: 5 | x i - m |: 1, 0, 3, 1, 10  :5  MAD :1.48 Data “Well Behaved” Classical Classical Robust Robust

Back to our measured data Robust bounds effectively remove effects of outliers Measured S 11 from 54 labsVariance of measured data

One lab’s measurement MCP allows one lab to compare to others One participant compared to the robust bound Absolute deviation from the median compared to the bound This lab should gain confidence in their capability

A second lab’s measurement Outliers can participate without distorting everyone’s data Second participant compared to the robust bound Absolute deviation from the median compared to the bound This lab now realizes it has measurement problems

Typical participant’s report Measurements from a one-port device. This lab is slightly outside the robust bounds.

Summary The ARFTG/NIST Measurement Comparison Program ongoing program uncovers measurement problems provides confidence to participants Robust statistical methods resistant to effects of outliers: useful in measurement comparisons easy to apply to large amounts of data

Standards Committee 2.4 mm connector, Bart Schrijver, Agilent Technologies. Phone (707) , FAX (707) , mm/K connector, Gilbert Perez, Anritsu. Phone (408) ext. 4950, FAX (408) , 3.5 mm connector, Phil Yates, JPL. Phone (818) , FAX (818) , GPC-7 connector, Yeou-Song (Brian) Lee,Anritsu. Phone (408) ext. 4976, FAX (408) , connector, Greg Burns, Northrup Grumman. Phone (410) , FAX (410) , Type ‘N’ connector, John Cable, Kansas City Plant. [ARFTG MCP Committee Chair] Phone (816) , FAX (816) ,

References “Outlier resistant methods for estimation and model fitting”, D.F. Vecchia and J.D. Splett, ISA Transactions, vol. 33, 1994, pp “Measurement program compares automatic vector analyzers,” R.M. Judish and J.G. Burns, Microwaves and RF, May 1991, pp “Robust statisical analysis of vector network analyzer intercomparisons,” R.M. Judish and J.D. Splett, Proc. IEEE Instrumentation and Measurement Society Conference, 1999, pp For more information see :