Skip to content

Free template

Updated

Attribute agreement analysis template

An attribute agreement analysis is a measurement system study for pass or fail decisions made by people or a go/no-go gauge. Several appraisers rate the same parts more than once, and the results are compared with each other and with a known reference. This template has a study sheet for 30 parts, three appraisers and two trials, and an Excel file that works out the agreement, kappa, effectiveness, miss rate and false alarm rate for each appraiser, with a verdict against limits you can change.

Free to use: print it, copy it and edit it for your team. Enter your name and work email once to download.

Page 1 of the attribute agreement analysis study sheetTwo blocks of 15 parts each with a reference column and columns for appraisers A, B and C, trials 1 and 2, with fields for the characteristic and the question asked.
Study sheet for 30 partsReference columnAgreement within each appraiserKappa against the referenceEffectiveness, miss rate and false alarm rateBetween-appraiser agreementEditable limits and verdicts
Filled-in Attribute agreement analysis template: 30 housings, three appraisers, two trials
The same sheet filled in with the worked example (page 3 of the PDF, illustrative numbers). Open full size (opens the picture in a new tab)

When to use it

When to use an attribute agreement analysis

  • When quality is judged by eye or by a go/no-go gauge: surface defects, cosmetic grades, visual sorting, a fit check with a gauge.
  • Before you trust a visual inspection station, and again after a change of people, lighting, standard or boundary samples.
  • When two shifts disagree on what is a reject, and you need numbers to show where the disagreement is.
  • As the attribute counterpart to a gage R&R study, which is for measurements that give a number.
  • Not for a measurement with a numeric reading. Use a gage R&R study for that.

How to fill it in

  1. 1

    Write the question the appraisers answer

    One characteristic and one question with two answers: Pass or Fail. If the standard is not clear enough to write as a question, fix that before you run the study.
  2. 2

    Pick 30 parts or more and fix the reference

    Include a few clear Fails, a few clear Passes and several near the limit. Have a master or the customer decide the reference for each part and keep it hidden from the appraisers.
  3. 3

    Run two trials with three appraisers

    Each appraiser rates every part twice, in a different random order each time, without seeing anyone else's calls and without a memory of the first round. Record P or F in the Study sheet.
  4. 4

    Read the agreement within each appraiser

    Count the parts where trial 1 and trial 2 are the same. Low agreement within one person means the standard is unclear to that person.
  5. 5

    Read each appraiser against the reference

    The Excel file works out effectiveness, the miss rate (a Fail called Pass), the false alarm rate (a Pass called Fail) and kappa against the reference, then gives Acceptable, Marginal or Unacceptable.
  6. 6

    Act on the pattern, then run the study again

    A high miss rate means defects get through: fix that first. A high false alarm rate means good parts are scrapped or re-sorted. Retrain, add boundary samples to the standard, and repeat.

The measures

What each measure tells you

Four numbers per appraiser. Read the miss rate first.

  • Agreement within the appraiser

    The question it answers
    Does this person give the same call twice?
    Where the numbers come from
    Parts where trial 1 equals trial 2, divided by the parts
    Watch for
    Low values mean the standard is unclear to that person
  • Effectiveness

    The question it answers
    How often is the call right?
    Where the numbers come from
    Correct decisions divided by all decisions (parts times 2)
    Watch for
    High effectiveness can hide a high miss rate
  • Miss rate

    The question it answers
    How often does a defect get through?
    Where the numbers come from
    Pass called on a part the reference says Fail, divided by all decisions on those parts
    Watch for
    The costly one: it reaches the customer
  • False alarm rate

    The question it answers
    How often is a good part rejected?
    Where the numbers come from
    Fail called on a part the reference says Pass, divided by all decisions on those parts
    Watch for
    Scrap and re-sorting cost
  • Kappa against the reference

    The question it answers
    How much better than chance is the agreement?
    Where the numbers come from
    (Observed agreement minus agreement expected by chance) divided by (1 minus agreement expected by chance)
    Watch for
    Low kappa with high agreement happens when most parts are Pass

A filled-in example

Illustrative, not a benchmark

An example (illustrative): a made-up study of a molded control-knob housing in gloss black, rated Pass or Fail for surface blemishes by three appraisers, A, B and C, twice each, on 30 parts. All decisions are invented.

  • The reference calls 11 parts Fail and 19 parts Pass. That is 22 decisions per appraiser on Fail parts and 38 on Pass parts.
  • Appraiser A agrees with themselves on 29 of 30 parts (96.7%), has a kappa against the reference of 0.96, an effectiveness of 98.3%, a miss rate of 0% (0 of 22) and a false alarm rate of 2.6% (1 of 38). Verdict: Acceptable.
  • Appraiser B agrees with themselves on 27 of 30 parts (90%), has a kappa of 0.89, an effectiveness of 95%, a miss rate of 4.5% (1 of 22) and a false alarm rate of 5.3% (2 of 38). Verdict: Marginal.
  • Appraiser C agrees with themselves on 23 of 30 parts (76.7%), has a kappa of 0.64, an effectiveness of 81.7%, a miss rate of 0% and a false alarm rate of 28.9% (11 of 38). Verdict: Unacceptable.
  • Across all three appraisers, all six decisions agree on 20 of 30 parts (66.7%) and all six match the reference on 20 (66.7%). Pairwise kappa on trial 1: A and B 0.72, A and C 0.60, B and C 0.47.
  • The verdicts use the default limits: effectiveness at least 90% to be acceptable and under 80% unacceptable, miss rate at most 2% and over 5%, false alarm rate at most 5% and over 10%, and kappa at least 0.75 and under 0.40.

A can keep inspecting. B needs the boundary samples shown. C rejects good parts and changes the call between trials, so C does not share the standard: retrain C and run the study again.

Common mistakes

  • Letting appraisers see the reference or each other

    If a person can see the reference or another person's call, the study measures the answer key, not the inspection.

  • Using only obvious parts

    A sample of clear Passes and clear Fails gives near-perfect agreement and tells you nothing. Include parts near the limit, as many as the line really sees.

  • Running the trials back to back in the same order

    Appraisers remember. Shuffle the parts between trials and between appraisers, and leave time between rounds.

  • Looking at agreement without the reference

    Three appraisers can agree with each other and all be wrong. Read agreement together with the results against the reference.

  • Treating the limits as law

    The 90, 80, 2, 5 and 10 percent limits are widely quoted guidelines. Your customer's requirements, or your own risk, may call for different limits. Change them in the sheet.

Download the template

Free to use: print it, copy it and edit it for your team. Enter your name and work email once to download.

Run this template in LeanSuite

In LeanSuite quality checks run as digital checklists with number limits, an out-of-spec answer can raise a tag or start a CAPA automatically, and KPI Builder tracks quality KPIs against targets.

See it in a 45-minute demo

FAQ

Attribute agreement analysis template: common questions

  1. Gage R&R is for a measurement that gives a number, and separates variation by the gauge, the appraiser and the part. Attribute agreement analysis is for a pass or fail call, and compares the calls within and between appraisers and against a reference.

  2. This sheet is built for 30 parts, three appraisers and two trials, which is a common size. More parts give steadier numbers. Include parts near the limit, not only clear Passes and Fails.

  3. Kappa compares the agreement you saw with the agreement you would expect by chance, so it is not fooled when most parts are Pass. Minitab credits the Automotive Industry Action Group's Measurement Systems Analysis manual, 4th edition, for treating a kappa of 0.75 or more as good agreement, and the sheet uses 0.75 and 0.40 as its defaults.

  4. They are widely quoted guideline values for effectiveness, miss rate and false alarm rate, set as editable defaults. They are not a standard. If your customer or your own risk assessment sets other limits, type them in the sheet and the verdicts follow.

  5. A master appraiser, a gauge with a known result or the customer, decided before the study and kept hidden from the appraisers. Without a reference you can measure agreement but not whether the calls are right.

More free templates

All templates
Browse all 277 templates

Can't find the template you need?

Tell us what you're looking for. We'll make it and send it to your inbox, free.

A shift handover sheet, a red tag log, a changeover checklist for one machine: if your team uses it on the floor, ask.

By sending this form you agree to our Terms of Service and Privacy Policy.

Pass it on

We want LeanSuite to be the best place on the internet for lean help. If this was useful, send it to someone on your team or in your network who needs it.