1. Measurement Systems Analysis: What is it and why should I care?
Dec. 11, 2012
Barry Kulback
Global Lean Six Sigma Leader
Trane and Thermo King, brands of
Ingersoll Rand

2. Feel free to ask questions at any time!
Agenda
About Ingersoll Rand
Measurement systems and Measurement System Error
What is Measurement System Analysis
Types of Measurement Systems Analysis and examples
Lessons Learned
Feel free to ask questions at any time!

3. About Ingersoll Rand A $14 billion diversified industrial company
Publicly-held; NYSE:IR
Operations in every major geographic region
Strategic brands are #1 or #2 in their markets
Products and services for commercial, industrial and residential markets

4. What do you think about my measurement system?
What’s wrong with a beer can measurement system?
-- cans can be dented
-- cans can be out of alignment
-- can be gaps between cans
-- are these 12 oz or 16 oz tall boys?
-- measuring to nearest can – insufficient discrimination
Many of the measurement ssytems we use day in and day out have similar issues – but we just don’t know it.

5. We do a lot of measuring in our businesses and in any process improvement methodology…
Running the business
Monitor process performance
Improving the business
Baseline
Set the levels of the adjustments
Getting better or worse
Validate improvement results
… and many of the measurement
systems we use for this have
similar problems as the one shown!
To assess or monitor process performance
To establish a baseline
To set the levels of the adjustments we are making to the process
To see if we are getting better or worse
To validate improvement results

6. When not measuring well…
Current process performance may be misjudged
Process improvement results may be misinterpreted
Missed opportunities
Wasted effort, $

7. 5 years of college in physics & chemistry labs….
Not once was it ever discussed… how does your ability to measure influence the results you think you are seeing?

8. My ‘aha’ moment came in my Six Sigma Black Belt Training
DMAIC process improvement methodology
Define
Measure
Analyze
Improve
Control
Measurement System Analysis
Collect data
Six Sigma practitioners are known as Green Belt, Black Belts and Master Black Belts

9. Premise for Six Sigma Methods
Sources of variation can be
Identified
Quantified
Eliminated by control or prevention
Y = f(x)
WAYNE SAYS KEEP THIS ONE
Explain Y=f(x)
My weight
KPOV, I can measure it, track it over time, it varies from day to day, week to week, month to month
Is a function of several KPIVs
How much I eat
How much I exercise
The types of food I eat
Data driven decisions with a known level of confidence…
… we do a lot of measuring in Six Sigma

10. What is an MSA?
When measuring there is always Measurement System Error
An MSA is a procedure to assess a Measurement System
Quantifies the Measurement System Error
Acceptable? Yes or no
If ‘no’  improve the measurement system
MSA output can tell you where to look
When measuring things, we would know that there is always a certain amount of Measurement System Error
An MSA is a procedure to assess a Measurement System
Conducting an MSA will quantify the amount of Measurement System Error, so an assessment can be made as to if it is acceptable or not
If not acceptable, steps should be taken to improve the measurement system
MSA output can help identify how to improve a poor performaing measurement system

11. Conducting an MSA For 2 of the 3 types of MSA’s we’ll cover today
Guidelines
Trained Operator(s)
Proper Method
Representative Samples
Generally two to three operators
Each unit is measured or assessed 2-3 times by each operator
Results are then analyzed
Often with statistical software like Minitab
Analytical and graphical outputs explain the results

12. Types of Data Continuous Data (Quantitative)
Decimal subdivisions are meaningful
Time (seconds)
Pressure (psi)
Conveyor Speed (ft/min)
Rate (inches)
Temperature (degrees)
Attribute Data (Qualitative)
Categories
Good / Bad
Inventory Classification Code A, B or C
Shift number
Counted things (# receipt errors, # units shipped, etc.)

13. Types of Measurement Systems Analysis
Continuous Data
Gage R&R
Attribute Data
Attribute Agreement Analysis
Data Scrub MSA

14. … but first just a wee bit of ‘technical’

15. so… = =

16. Types of Measurement Systems Analysis
Continuous Data
Gage R&R
Attribute Data
Attribute Agreement Analysis
Data Scrub MSA

17. How does measurement system error appear?
LSL
USL
Actual process variation - No measurement error
LSL
USL
Observed process variation With measurement error

18. Why do we care? Can’t really understand the true variation
present! And if it isn’t understood, it can’t
be fixed.
LSL
USL x x x x
A ‘bad’ part
can measure
good
A ‘good’ part
can measure
bad
Observed process variation With measurement error

19. Possible Sources of Process Variation
Observed Process Variation
Actual Process Variation
Long Term
Short Term
Within Sample
Measurement Variation
Due to instrument
Repeatability
Calibration
Stability
Linearity
Due to operators

20. Possible Sources of Process Variation
Observed Process Variation
Actual Process Variation
Long Term
Short Term
Within Sample
Measurement Variation
Due to instrument
Repeatability
Calibration
Stability
Linearity
Due to operators

21. Possible Sources of Process Variation
Observed Process Variation
Actual Process Variation
Long Term
Short Term
Within Sample
Measurement Variation
Due to instrument
Repeatability
Calibration
Stability
Linearity
Due to operators

22. Possible Sources of Process Variation
Observed Process Variation
Actual Process Variation
Long Term
Short Term
Within Sample
Measurement Variation
Due to instrument
Repeatability
Calibration
Stability
Linearity
Due to operators
‘Repeatability’ and ‘Reproducibility’ are the two main contributors to Measurement System Error – hence ‘Gage R&R’

23. -- That depends on what you are using the measurement system for!
If Measurement System Error always exists, when should we be concerned with it?
-- When it it too large.
Too large compared to what?
-- That depends on what you
are using the measurement
system for!

24. Gage R&R – case study Process: Compressor Machining
Project: Six Sigma Black Belt Project
Scrap Reduction
$375,000 in scrap / year
Capacity bound 
lost margins on lost sales
Considering $1M CAPX
to increase capacity

25. Measurement system improved  project complete!
Inspection;
CMM –
measures to
inch
machined casting
assembly into compressor
LSL
USL
Actual
Observed
Define
Measure
Analyze
Improve
Control
Measurement system improved  project complete!
Works in a special environmentally controlled room.
Similar to this one saw used for $145,000.

26. Types of Measurement Systems Analysis
Continuous Data
Gage R&R
Attribute Data
Attribute Agreement Analysis
Data Scrub MSA

27. Attribute Measurement Systems
Assessing attribute data often involved judgment
… sometimes a little
“it’s broke / it isn’t” or “it fits / it doesn’t”
… sometimes a lot
“ it dented too bad, scrap it”

28. Attribute Measurement Systems
It all starts with an Operational Definition:
The ‘spec’
Describes what the defects or categories are
Describes how to perform the appraisal assessment
Used to train those performing the assessment
Should be applied with a high degree of consistency

29. Attribute Agreement Analysis – case study
Process: 2” Copper Tube
Bending
Project: Six Sigma
Green Belt Project
Scrap Reduction

31. Spec = “No Wrinkles”
What’s a wrinkle? How bad is bad?

32. How good are our Operators at assessing
if a tube is wrinkled and should not be used?
Not wrinkled, use it.
Slight wrinkle, use it?
That’s not a wrinkle, it’s a tool mark! Use it?
Not a wrinkle, a stretch mark! Use it?
Wrinkled, scrap it.

33. An Attribute MSA was conducted:
30 Samples
4 Operators
1 ‘Expert’
2 Trials

34. want 90% level of agreement or higher
Ted
Michael
Jim
Danny
Albner
100 90 80 70
Appraiser P e r c n t
Within Appraiser
Assessment Agreement
Date of study:
Reported by:
Name of product:
Misc: [ , ]
95.0% CI
Percent
‘Expert’!
want 90% level of agreement or higher

35. Spec = “No Wrinkles” – OK!
Would use
Would not use ?

36. Visual Aid Added
Prep for future new employees...

37. Types of Measurement Systems Analysis
Continuous Data
Gage R&R
Attribute Data
Attribute Agreement Analysis
Data Scrub MSA

38. Sometimes we get our data out of the ‘system’
… is it right?

39. Data Scrub MSA – case study
Process: Cooling the office
Project: Six Sigma Green Belt Project
Reduce Energy Consumption
Ed’s project
As part of the Office HVAC Control System project an MSA was required. The MSA would consist of verification of system operation. Report logs were generated by the Tracer Summit system stating whether the systems were "on" or "off". Actual system operation would be observed during periods when the Tracer Summit system would cycle the systems "off" to verify that they were actually "off".
The first time the MSA was done, the observed readings did not match the Report Logs that were generated by the system. Units were running even though the computer had them scheduled to be off. It was found that the there was a mechanical problem and the units were switched to a manual mode to perform service work. The units were left in the manual mode after repairs were made which allowed continuous operation. Until the units were switched to automatic mode, the computer could not control the system operation.
The MSA allowed verification of computer generated data.

40. Lessons Learned…

41. An expensive gage does not always mean good measurements

42. Remember this?
Inspection;
CMM –
measures to
inch
$145,000 used
Must operate in a environmentally controlled room
Strict procedures on part handling, cleanliness, controlling local conditions, controlling part temperature…

43. A measurement system is more than a gage or a operational definition (specification)

44. A continuous variable measurement system is composed of:
the gage / measuring device
operator techniques
set-up and handling techniques
the environment in which the measurements
are being done (ex. lighting, access)
recording of measurement results

45. The Attribute Measurement System
Operational
Definition
Training of
Operators
Problems in any of these areas can lead to too high a degree of inconsistent / incorrect assessments
Application

46. Just because a measurement system has been in use ‘forever’ doesn’t mean it is very good

47. Attribute MSA – case study
Process: Invoicing –
Application of
appropriate tax
Project: Six Sigma
Black Belt Project
Improve DSR

48. Tax Codes are applied to invoices being sent out to Customers:
Sales Tax -- a percentage added to invoice,
customer pays
Use Tax -- a percentage of the cost of the
goods, company pays
Non-Tax -- government, hospital, etc., where
neither customer or company
pays
What tax should be applied?

49. (been doing it for years…)
How good are the Accountants at
applying the correct tax code?
(been doing it for years…)

50. An Attribute MSA was conducted:
10 Samples (more would be better)
3 Operators -- who do the job
every day
1 ‘Expert’
2 Trials

53. Just because a it came out of the computer doesn’t mean it is accurate

54. Data Scrub MSA’s – case studies
Focused Management Team monthly published results
Ed’s project
As part of the Office HVAC Control System project an MSA was required. The MSA would consist of verification of system operation. Report logs were generated by the Tracer Summit system stating whether the systems were "on" or "off". Actual system operation would be observed during periods when the Tracer Summit system would cycle the systems "off" to verify that they were actually "off".
The first time the MSA was done, the observed readings did not match the Report Logs that were generated by the system. Units were running even though the computer had them scheduled to be off. It was found that the there was a mechanical problem and the units were switched to a manual mode to perform service work. The units were left in the manual mode after repairs were made which allowed continuous operation. Until the units were switched to automatic mode, the computer could not control the system operation.
The MSA allowed verification of computer generated data.

55. Recap About Ingersoll Rand
Measurement systems and Measurement System Error
What is Measurement System Analysis
Types of Measurement Systems Analysis and examples
Lessons Learned

57. You cannot assume you are measuring well
Final lesson:
You cannot assume you are measuring well

58. Measurement error is always present
You don’t know if it is small enough to ignore unless you assess it!

59. Addendum

60. The P/T ratio quantifies this... A ratio of MSE to the tolerance range
If using the measurement system to see if the item being measured is within the spec, you want the Measurement System Error (MSE) to be small compared to that spec:
specification range, or tolerance
sample
observed
measurements
This..
should be small when
compared to this...
The P/T ratio quantifies this...
A ratio of MSE to the
tolerance range
LSL
USL x
MSE

61. The %R&R quantifies this... A ratio of MSE to process variation
If using the measurement system to analyze the variation present or control the process with Statistical Process Control, you want MSE to be small compared to that variation:
sample
observed
measurements
LSL
USL
This..
should be small when
compared to this...
The %R&R quantifies this...
A ratio of MSE to process
variation x
MSE

62. Measurement System Analysis: What is it and why should I care?
Dec. 11, Professional Development Meeting
TOPIC AREA: MEASUREMENT SYSTEMS
Measurement System Analysis: What is it and why should I care?
SPEAKER – Barry Kulback, Master Black Belt, Six Sigma
For our December Professional Development Meeting we are pleased to have Barry Kulback, Master Black Belt.
Measurement System Analysis (MSA) is an often overlooked critical step in any process improvement process and is the linchpin of Six Sigma’s Measure phase in the DMAIC (Define, Measure, Analyze, Improve, and Control) methodology. This non-technical presentation will review what MSA’s are and why they are important. The major components of measurement error will be discussed along with how measurement error may color your perception of process performance. Examples of the three most common types of MSA’s will be shared along with some lessons learned. Come find out why you should assess your measurement systems the next time you embark on a process improvement journey.

63. Sound interesting? Come join us! We look forward to seeing you!
Reserve your place today by clicking here or by ing
DATE: December 11, 2012
PLACE: Holiday Inn Select Opryland Airport/Briley Parkway
2200 Elm Hill Pike
Nashville, TN 37214
(615)
Click for directions
TIME: 5:30 to 8:00 PM
COST: $20.00 for Dinner - Free if an unemployed member - bring your resume
You do not need to be an APICS member to attend
Remember! Attend Four (4) APICS Nashville Professional Development Meetings from September 2012 thru April 2013 (excluding Tours) and attend the 5th PDM free! New members to our chapter attend their first meeting at no charge when they bring their APICS welcome letter!
Stay Informed! Click to Join our LinkedIn group APICS Middle Tennessee Chapter Nashville

64. Speaker Bio:
Barry Kulback’s 33 year professional career has been entirely with Trane/American Standard now Ingersoll Rand. He is currently the Global Lean Six Sigma leader for the Climate Solutions Sector of Ingersoll Rand and very engaged in IR’s Lean Transformation. He gained his BS in 1979 at Austin Peay State University majoring in Physics and minoring in Computer Science. After spending 20 years in Information Technology Barry joined Operations as a Six Sigma Black Belt and progressed to a certified Six Sigma Master Black Belt in He was named Tennessee Academy of Science’s Industrial Scientist of the year in 2006 and is the holder of 4 patents related to algorithms for delivering to customer request by optimizing the match of demand to supply. He maintains his membership in the American Society for Quality as a Senior Member.
Reserve your place today by clicking here or by ing
Stay Informed! Click to Join our LinkedIn group APICS Middle Tennessee Chapter Nashville
Thank you for your support of APICS.

65. Bring 1 copper tube 1 measuring tape PC Remote mouse
Backup – on data stick, on disk
Pad, pen to take notes re parking lot
45-50 minutes, leaving minutes at the end for question/answer

66. Title
Topic
Sub-topic
Sub-sub-topic

67. Broad and Global Portfolio of HVAC Systems and Services
SERVICES / CONTROLS
TRANSPORT REFRIGERATION
Vision
To make building owner and transport customers more profitable and efficient for life through innovative HVACR systems and services