MSA 7 -- Attributive Measurement System Analysis
The MSA Method 7 is attributive measurement system analysis and serves to evaluate measurement systems in which characteristics are not assessed metrically (as numerical values), but attributively (as categories) -- for example "good/bad", "OK/not OK" or according to a rating scale.
Overview
Purpose and Field of Application
Attributive MSA answers the question: Are inspectors able to consistently and uniformly evaluate parts when no metric measured values are available?
Typical applications:
- Visual inspections -- Surface quality, color deviations, scratches, dents
- Functional tests -- Good/bad decisions (e.g., go/no-go gauge testing)
- Subjective assessments -- Smell, tactile feel, acoustic testing
- Sorting inspections -- Classification into multiple quality levels
Experimental Design
For attributive MSA, the following basic experimental structure applies:
- A selection of parts is assembled. The parts should include both clearly good and clearly bad parts as well as borderline cases.
- Each inspector evaluates each part multiple times (at least 2 runs, 3 recommended).
- The assessments are conducted blind -- the inspectors neither know the "correct" answer (reference assessment) nor the assessments of other inspectors.
- Optionally, a reference assessment (master decision) is established for each part, against which the inspector assessments are compared.
| Parameter | Recommendation |
|---|---|
| Number of parts | At least 20, ideally 50 |
| Number of inspectors | At least 2, recommended 3 |
| Number of runs | At least 2, recommended 3 |
| Proportion of borderline cases | Approximately 30--50% of parts |
| Reference assessment | Recommended (established by experts or specification) |
Important: The quality of an attributive MSA depends significantly on part selection. If only clearly good and clearly bad parts are used, the study will overestimate the capability of the inspection process. Be sure to include sufficient borderline cases.
Input
Configuration
Before data entry, establish the experimental parameters:
| Field | Description |
|---|---|
| Assessment categories | The possible assessments (e.g., "OK" / "not OK" or "1" / "2" / "3") |
| Number of inspectors | How many inspectors participate in the study |
| Number of parts | How many parts are assessed |
| Number of runs | How many times each inspector assesses each part |
| Reference assessment | Optional "correct" assessment for each part (master decision) |
Entering Assessments
Data entry is performed via a table with the following structure:
- Rows: Parts (numbered or named)
- Columns: Inspectors x runs (e.g., "Inspector A / Run 1", "Inspector A / Run 2", ...)
- Cell values: The selected assessment category
- Click in the desired cell.
- Select the assessment category from the dropdown list or enter the value directly.
- Navigate to the next cell with
TaborEnter.
Tip: If a reference assessment is available, enter it in the first column ("Reference"). The agreement of inspectors with the reference is calculated automatically.
Info: You can also import input data via copy & paste from Excel. Make sure that the assessment categories match the defined categories exactly (case-sensitive).
Kappa Values
The central metrics of attributive MSA are the Kappa coefficients, which quantify the agreement of inspectors.
Cohen's Kappa
Cohen's Kappa measures the agreement between two inspectors (pairwise comparison) taking into account the randomly expected agreement.
Formula:
Kappa = (P_o - P_e) / (1 - P_e)
- P_o = Observed agreement (actual proportion of matching assessments)
- P_e = Expected random agreement
Cohen's Kappa is calculated separately for each inspector pair. In my8data, a complete Kappa matrix is displayed, in which each cell contains the Kappa value for a specific inspector pair.
Fleiss' Kappa
Fleiss' Kappa is an extension of Cohen's Kappa for more than two inspectors. It measures the overall agreement of all inspectors simultaneously.
Fleiss' Kappa is displayed in my8data as a single overall value and indicates how well the inspectors overall agree.
Assessment Scale
The interpretation of Kappa values follows the standard classification according to Landis & Koch (1977):
| Kappa Value | Strength of Agreement | Assessment |
|---|---|---|
| < 0.00 | Poor | Worse than chance. Fundamental problem with the inspection process. |
| 0.00 -- 0.20 | Slight | Minimal agreement. Inspection process unsuitable. |
| 0.21 -- 0.40 | Fair | Weak agreement. Considerable improvements needed. |
| 0.41 -- 0.60 | Moderate | Moderate agreement. Improvements recommended. |
| 0.61 -- 0.80 | Substantial | Good agreement. Acceptable for many applications. |
| 0.81 -- 1.00 | Almost Perfect | Nearly perfect agreement. Inspection process excellent. |
Info: A Kappa value of 1.0 means perfect agreement. A value of 0 means that the agreement is no better than pure chance. Negative values indicate systematic contradiction.
Important: In practice, a Kappa value of at least 0.75 is frequently required. In safety-critical areas (e.g., medical devices, aerospace), higher requirements may apply.
Agreement
In addition to Kappa values, my8data provides further analyses of inspector agreement.
Agreement Rates
my8data calculates various agreement rates:
| Metric | Description |
|---|---|
| Within Appraiser | How consistent is each individual inspector with themselves across the various runs? A high value shows that the inspector arrives at the same result when re-assessing the same part. |
| Between Appraisers | How well do the inspectors agree with each other? Compares the assessments of all inspectors for each part. |
| Appraiser vs. Reference | How well does each inspector agree with the reference assessment? Shows the accuracy of each individual inspector. |
| All Appraisers vs. Reference | How well do all inspectors collectively agree with the reference? Only parts where all inspectors agree in all runs are counted. |
Confusion Matrix
The confusion matrix compares the assessments of each inspector against the reference assessments:
| Reference: OK | Reference: not OK | |
|---|---|---|
| Inspector: OK | True positive (correctly accepted) | False positive (incorrectly accepted) |
| Inspector: not OK | False negative (incorrectly rejected) | True negative (correctly rejected) |
The following metrics are derived from the confusion matrix:
| Metric | Formula | Description |
|---|---|---|
| Effectiveness | (True positive + True negative) / Total | Proportion of overall correct decisions |
| Miss Rate | False positive / (True negative + False positive) | Proportion of bad parts incorrectly accepted |
| False Alarm Rate | False negative / (True positive + False negative) | Proportion of good parts incorrectly rejected |
Warning: The miss rate is particularly critical because it indicates how many bad parts are passed as good. In safety-relevant areas, this rate must be as close to 0 as possible.
Tip: Analyze the confusion matrix for each inspector individually. If a particular inspector has a noticeably high miss rate, targeted training should be provided for that inspector. Also examine which parts frequently have errors -- often these are borderline cases where inspector decision is uncertain.
Graphical Evaluations
my8data also presents the agreement analysis graphically:
- Agreement bars: Display the agreement rate per inspector as a bar chart.
- Heatmap: Color-coded matrix showing which parts the inspectors agree on and which they do not.
- Error pattern diagram: Visualizes which parts are frequently assessed incorrectly.