Data annotation QA workflow
Good labels come from a process with checks at each step, not from a promise. This workflow runs from the label guide to an accepted batch, and shows what to test when you choose a vendor.
The QA workflow, step by step
- Write the label guide. Every class, with pictures of edge cases, what to skip and what to do when unsure.
- Build a gold set. Items an expert labels and a second person checks. These are the answers you score against.
- Run a pilot. A small batch scored against the gold set. Read every error and fix the guide before going further.
- Hide gold items in the work. Mix gold items into normal batches without marking them, so scores reflect everyday work.
- Label some items twice. A share of items goes to two labelers, so you can measure agreement between them.
- Review each batch. A second person checks a sample; rule checks catch boxes off the image or missing classes.
- Accept or return the batch. Compare the scores with the pass marks agreed in advance, and return batches that miss.
- Feed back. Update the guide with each new edge case and tell labelers about the classes they get wrong.
The numbers behind steps 2 to 7 (accuracy, IoU and agreement) are explained in annotation quality metrics.
Review levels: who checks what
| Who checks | What they do | What it catches |
|---|---|---|
| The labeler | Checks their own work before sending it | Slips: a missed object, a wrong class picked |
| A second labeler | Labels the same items, or checks another's | Unclear rules, where two people read the guide differently |
| A reviewer or QA lead | Checks a sample of each batch against the guide | Errors repeated across a whole batch or one labeler |
| Rule checks in the tool | Runs on every label | Boxes outside the image, empty labels, duplicate shapes |
| Your own team | Spot-checks delivered batches against the gold set | Drift from what your model needs |
How companies and tools describe their checks
- Scale AI: software labels the data first, then people review it, with more than one level and kind of review (docs.scale.com).
- CVAT: ground truth jobs, honeypots and consensus between labelers (docs.cvat.ai).
- Label Studio: reviews and agreement scores in its Enterprise edition (labelstud.io).
Most of the 15 companies we compare do not describe their checks on their own pages. Ask each one to show you a batch report from a past project: the scores, the sample size and what happened to batches that missed.
Data labeling vendor selection: what to test in the pilot
- Send every vendor the same small batch of items, with the same label guide, and keep a gold answer for each.
- Score accuracy, IoU and missed objects yourself; do not rely on the vendor's own score.
- Note the questions they ask about the guide. Good questions early mean fewer errors later.
- Time the turnaround, and ask how they would staff the full volume.
- Ask for their audit reports before any real data moves (see data annotation security).
- Compare the price per label after rework, not the headline rate.
These companies publish a free or paid test on their own pages:
- CloudFactory: a few of your files labeled free as a test
- Label Your Data: free trial job
- Keymakr: a few labeled examples you can check first
Batch rules for the contract
- How many items are checked per batch, and who picks them.
- The pass marks: accuracy on gold items, IoU for shapes, agreement on items labeled twice.
- A batch below the mark is relabeled at the vendor's cost and scored again.
- The guide is the reference: changes to it are dated, and older batches are judged by the version they were labeled under.
More buyer guides
- Annotation quality metrics: Accuracy against a gold set, IoU for boxes and masks, and agreement between labelers, with worked examples.
- Data annotation security and compliance: The audits and laws each company and platform names, linked to its source, and the papers to ask for.
- Data labeling outsourcing: Costs, pilots and contracts when another company labels your data.
Want companies to quote against these rules? Describe the job once, and up to three annotation companies reply with prices.
Get quotes from three companies