Annotation quality metrics
To judge if labels are fit for training, watch three numbers: gold-set accuracy, IoU for boxes and outlines and how much labelers agree. You'll see what every metric measures, the pass marks people use and the way to calculate it.
- IoU, PASCAL VOC
- At 0.50or more overlap, a box counts as a match.
- IoU, COCO
- Ten marks in 0.05steps run from 0.50 to 0.95, with scores averaged across them.
- Cohen's kappa
- From 0.61 to0.80 means substantial agreement; over 0.80 is almost perfect.
- Krippendorff's alpha
- Firm conclusions usually need 0.800 as the floor.
Reference list: Cohen's kappa (Landis and Koch scale) · COCO detection evaluation · Krippendorff's alpha
Data annotation accuracy: score labels against a gold set
Call a batch of items a gold set when an expert has labeled and checked its right answers. Add those items to the regular workload while keeping labelers unaware of which ones they are (honeypots, as the tools call them); compare afterwards. The portion of gold items labeled correctly is the accuracy score.
A box counts as "right" if its class is right and the overlap sits above your IoU mark (below). Check the score for every labeler and every class; a team can do well overall yet miss most items in one rare class.
A sample calculation
In one week, a labeler sees 200 gold items and labels 188 correctly. That gives 188 ÷ 200, or 94% accuracy.
This check comes built into the tools. One example is CVAT, which offers ground truth and honeypot jobs for comparing labels with known answers during checks.
IoU: how well a box or outline lines up
A drawn box or outline is compared with the gold one by Intersection over union (IoU): shared area divided by total covered area. Shapes that match exactly score 1; no overlap means 0 under the Jaccard index.
A sample calculation
Take two 100 × 100 pixel boxes, with one moved 20 pixels sideways. The overlap is 80 × 100 = 8,000 pixels; together they span 10,000 + 10,000 − 8,000 = 12,000. So IoU comes out at 8,000 ÷ 12,000 = 0.67.
Each of the two public benchmarks uses its own cutoff. For PASCAL VOC, IoU of 0.50 lets a box match; COCO averages its score across ten steps spanning 0.50 through 0.95. Small objects and pixel masks call for a stricter mark than large boxes, so set yours before the pilot and enter it in the label guide.
Inter-annotator agreement: will two labelers respond the same way?
Send identical items to two or more labelers, then compare what they answer. A simple percent agreement makes the team look better, since some matches happen by luck. With two raters, Cohen's kappa discounts agreement expected by luck; the scikit-learn docs frame it as agreement on a classification task."). Beyond two labelers, Fleiss' kappa plays the same role; Krippendorff's alpha accepts however many labelers there are and tolerates gaps from skipped items.
A sample calculation
On 100 photos, two labelers give yes-or-no answers and agree on 85. Yes comes up 60 times for one labeler and 55 times for the other. Agreement by chance works out to 0.60 × 0.55 + 0.40 × 0.45 = 0.51. That gives kappa as (0.85 − 0.51) ÷ (1 − 0.51) = 0.69.
Reading the score: on the Landis and Koch scale, 0.61 to 0.80 counts as substantial and 0.81 to 1.00 as almost perfect, even though the authors offered no evidence behind those cut-offs (Cohen's kappa). Social scientists commonly trust alpha data at 0.800 or above (Krippendorff's alpha); they rate 0.667 to 0.800 fit for tentative conclusions only. Usually, low agreement points to an unclear label guide rather than careless labelers: inspect the items behind the disputes, then improve the guide.
Precision and recall for missed and extra labels
Two kinds of error slip past accuracy on drawn labels: objects that nobody labeled, and labels placed where nothing is. Of the drawn labels, precision is the share matching a gold object; of the gold objects, recall is the share that received one. A labeler boxing just the easy cars gets high precision but weak recall. Use the gold set to measure both.
Which metric fits each label type
| Label type | What to measure |
|---|---|
| Tags for a whole photo or text | Gold set accuracy; agreement (kappa) across twice-labeled items |
| Boxes around objects | IoU for each box, the correct class, plus recall for objects that got missed |
| Outlines and pixel masks | IoU for every outline or mask, and whether the class is right |
| Keypoints | Pixel distance from the gold point, plus any points that are missing |
| Names and phrases in text | Precision and recall on exact span matches |
| Ratings of chatbot answers | Rater agreement (Krippendorff's alpha), because often no single right answer exists |
Putting the numbers into a contract with a vendor
- For every batch, accuracy on your gold set, with the pass mark attached.
- An IoU mark for boxes and outlines, broken out per class when small objects are important.
- Some items get labeled twice, and the agreement score is reported.
- The consequence for a batch under the mark: the vendor pays for relabeling, then it gets scored once more.
These checks are laid out in order in the guide to the QA workflow. Begin with companies that publish a pilot when you want a vendor checked on your own gold set first.
Further guides
Data annotation QA workflow.Check labels one stage at a time, plus which checks belong in a small trial job.
Security and compliance for data annotation.Which audits and laws each company and platform cites, with links to the source, and the documents to request.
Outsourcing data labeling.Costs, small trial jobs and contracts when an outside company labels your data.
Want companies to quote against these rules? Describe the job once, and up to three annotation companies reply with prices.
Get quotes from three companies