1. Creating a Calibration Measurement Monitoring System for Many, Ever-changing, Complex Instruments John Wilson Software QA Engineer Agilent Technologies, Inc

2. Learning Objectives: Learn design considerations for building a monitoring system for improving calibration quality for complex instruments. Design goals Enabling technologies/features Learn monitoring systems are needed, provide faster detection and resolution of issues, and can be accomplished with today’s technology. (Case study)

3. Similar Situation – What is Our Situation? Building a measurement monitoring system ISO 17025 section 5.9 “Assuring the quality of test and calibration procedures” ANSI/NCSL Z540.3 section 5.3.2 Measurement assurance procedures Complex electronic instruments Many instruments Legacy instruments Many new instruments added all the time Many calibration platforms Multiple sites Geographically dispersed

4. Design Goals for a Calibration Measurement Monitoring System (CMMS) Monitor the parameters reported on the calibration reports received by customers Electronic data, not pdf or paper

5. Design Goals – Data Collection Data collection/analysis should be frequent and ongoing Like to be real time; but realistically we knew that was not achievable with our constraints in first effort. Targeted daily updates. Data collection should be unobtrusive Collect data on what is already being done Don’t add another test, don’t take away production time Representative of work done for customers, not some idealized “best effort”

6. Design Goals – Analysis Analysis should lead to corrective action Not just detection, but provide trouble-shooting tools Bring the relevant information to bear on the problem. Analysis should communicate Graphics: Language of engineering Adequate engineering labeling –Dynamic labeling – Labels change on context, important with drill down  Fast decisions

7. Enabling Technology – Common Cal Format CalServiceResults XML File Relational Database Visualization Output Test Stations: UUTs XML Schema - A common format for all calibration results DMMs, Oscilloscopes Spectrum Analyzers Signal Generators Timebase Phase Noise Other

8. Enabling Technologies – Setting the Context Well Communicating a message Dynamic Titles Axis labels Reference lines Uncertainty bars Version coloring Date formats

9. Enabling Technologies – Summaries, Drill Down, In Memory Data Visualization and RAM Enables FAST drill down from high level summaries to problem solution With hundreds of thousands of points to analyze, need summaries to highlight trouble-spots. Need to be able to drill down and isolate the core problem quickly Two tools for dealing with query times for lower level data queries –In memory data visualization tools, combined with large RAM (4 - 8 Gbytes) Tend to over-pull data on initial query –Caching (do the query overnight, cache results and examine in the morning) Gives faster updates when exploring later –Data on demand

10. Enabling Technologies – Normalized Measurement Data for Charts Old trick - useful for bringing two different scales together. Upper Spec Limit = +1 Lower Spec Limit = -1 100 mV, 1 V, 10 V, 100 V, and 1000 V points all use same x-axis for histograms, normalized to -100 % (lower limit) at left to + 100 % (upper limit) at right. Upper and lower test limits vary from unit to unit, requiring normalization for control charts.

11. Enabling Features – Correlation: Bringing Relevant Information to Bear Bring in version, cal standards, location, time of day, temp, humidity, cal factor files dates Search for root cause and corrective action

12. Enabling Technologies – Standards and Measurement Uncertainties Inclusion of point-by-point measurement uncertainties (MU) on measurement reports Symbiotic relationship –Monitoring system enhanced by having MUs; –MU estimates improved by monitoring system When process shifts by more than the MU allows, –either the process doesn’t conform to the process defined in the measurement uncertainty equation, or –the MU estimate has left out or underestimated a significant contributor. Overall culture change –Monitoring implants MUs into the culture/process

13. Case Study – DMM Monitoring Discovery, Trouble-shooting, Resolution Tens of Service Centers 100s of calibrations 1000s of tests 100,000s of measurement points  Boiled down to two issues: Daily email summary:

14. Dive into Specific Problem

15. Move from Normalized Data to Measured Data: Problem Exists in Calculation of Test Limits

16. Add Information to Basic Control Chart: Go From Problem  Correlation  Fast Solution

17. Does Anyone Else See the Same Problem? Compare to Other Service Centers Present Exception Previously undetected occurrences

18. Resolution: The characterization of the cal standard (which set the test limits) used a setting that was almost the same as the setting when the cal standard was used in measuring UUTs. Some cal standards were more sensitive to the settings than others. Code was changed so that the settings were exactly the same in the characterization of the cal standard and when used to measure UUTs.

19. Continued Monitoring Verifies the Fix

20. Summary: Through a case study, you will have learned how we can improve our calibrations by implementing process monitoring. My hope is you’ll walk away having learned design considerations to help implement your process monitoring.