Annotation quality metrics
Three numbers tell you whether labels are good enough to train on: accuracy against a gold set, IoU for boxes and outlines, and agreement between labelers. Here is what each one measures, how to work it out and the pass marks in use.
- IoU, PASCAL VOC
- 0.50a box counts as a match at this overlap or above
- IoU, COCO
- 0.50 to 0.95scores averaged over ten marks in steps of 0.05
- Cohen's kappa
- 0.61 to 0.80read as substantial agreement; above 0.80 almost perfect
- Krippendorff's alpha
- 0.800the usual floor for firm conclusions
Sources: COCO detection evaluation · Cohen's kappa (Landis and Koch scale) · Krippendorff's alpha
Data annotation accuracy: score labels against a gold set
A gold set is a batch of items whose right answers an expert has labeled and checked. Mix those items into normal work without telling the labelers which ones they are (tools call them honeypots), then compare. Accuracy is the share of gold items labeled right.
For boxes, "right" means the right class and an overlap above your IoU mark (below). Track the score per labeler and per class: a team that scores well overall can still miss most of one rare class.
A worked example
A labeler meets 200 gold items in a week and gets 188 right. Accuracy is 188 ÷ 200 = 94%.
Tools build this in. CVAT, for one, lists ground truth jobs and honeypots for checking labels against known answers.
IoU: how closely a box or outline matches
Intersection over union (IoU) compares a drawn box or outline with the gold one: the area they share, divided by the area they cover together. Identical shapes score 1; shapes that do not touch score 0 (Jaccard index).
A worked example
Two boxes of 100 × 100 pixels, one shifted 20 pixels to the side. They share 80 × 100 = 8,000 pixels and cover 10,000 + 10,000 − 8,000 = 12,000 together. IoU is 8,000 ÷ 12,000 = 0.67.
The two public benchmarks set the bar differently. PASCAL VOC counts a box as a match at an IoU of 0.50; COCO averages its score over ten marks from 0.50 to 0.95 (COCO). Pick your own mark before the pilot and write it into the label guide: small objects and pixel masks need a stricter one than large boxes.
Inter-annotator agreement: do two labelers give the same answer?
Give the same items to two or more labelers and compare their answers. Plain percent agreement flatters the team, because some answers match by chance. Cohen's kappa removes that chance share for two labelers (scikit-learn calls it a score of "agreement between two annotators on a classification problem"). Fleiss' kappa does the same for more than two (Fleiss' kappa); Krippendorff's alpha works for any number of labelers and copes with items some of them skipped.
A worked example
Two labelers answer yes or no on 100 photos and agree on 85. One says yes 60 times, the other 55. Chance agreement is 0.60 × 0.55 + 0.40 × 0.45 = 0.51. Kappa is (0.85 − 0.51) ÷ (1 − 0.51) = 0.69.
How to read it: the Landis and Koch scale calls 0.61 to 0.80 substantial and 0.81 to 1.00 almost perfect, though its authors gave no evidence for the cut-offs (Cohen's kappa). For alpha, social scientists commonly rely on data at 0.800 or above and treat 0.667 to 0.800 as good only for tentative conclusions (Krippendorff's alpha). Low agreement usually means the label guide is unclear, not that the labelers are careless: read the items they disagree on and fix the guide.
Missed and extra labels: precision and recall
Accuracy on drawn labels misses two kinds of error: objects nobody labeled, and labels drawn on nothing. Precision is the share of drawn labels that match a gold object; recall is the share of gold objects that got a label. A labeler who boxes only the easy cars scores high precision and low recall. Score both on the gold set.
The right metric for each label type
| Label type | What to measure |
|---|---|
| Tags for a whole photo or text | Accuracy against the gold set; agreement (kappa) on items labeled twice |
| Boxes around objects | IoU per box, the right class, and recall for missed objects |
| Outlines and pixel masks | IoU per outline or per mask, and the right class |
| Keypoints | Distance in pixels from the gold point, and missing points |
| Names and phrases in text | Precision and recall on exact span matches |
| Ratings of chatbot answers | Agreement between raters (Krippendorff's alpha), since there is often no single right answer |
Writing the numbers into a vendor contract
- Accuracy on your gold set for each batch, with the pass mark.
- The IoU mark for boxes and outlines, per class if small objects matter.
- A share of items labeled twice, with the agreement score reported.
- What happens to a batch below the mark: relabeled at the vendor's cost, then scored again.
The QA workflow guide sets these checks in order. To test a vendor on your own gold set first, start with the companies that publish a pilot.
More buyer guides
- Data annotation QA workflow: From the label guide to an accepted batch, and what to test in a vendor pilot.
- Data annotation security and compliance: The audits and laws each company and platform names, linked to its source, and the papers to ask for.
- Data labeling outsourcing: Costs, pilots and contracts when another company labels your data.
Want companies to quote against these rules? Describe the job once, and up to three annotation companies reply with prices.
Get quotes from three companies