Jump to the comparison
Data Annotation Services

Data annotation QA workflow

Reliable labels come from checks built into every step of the process, not from promises. The process starts with the label guide and ends at an accepted batch; it also shows what to check as you pick a vendor.

Step-by-step QA workflow

  1. Write down the label guide. Cover each class, with edge cases shown in pictures, the items to skip and the steps for when you're unsure.
  2. Put together a gold set. An expert labels these items and a second person checks them. You score against these answers.
  3. Carry out a pilot run. One small batch, scored against the gold set. Go through every error and correct the guide before moving on.
  4. Keep gold items hidden in the work. Blend them into normal batches with no marking, so the scores reflect everyday labeling.
  5. Double-label some of the items. Send a share of the items to two labelers, which lets you measure how much they agree.
  6. Check over each batch. Someone else checks a sample, while rule checks catch missing classes or boxes off the image.
  7. Accept the batch, or return it. Check the scores against the pass marks agreed in advance; send back any batch that misses.
  8. Pass on feedback. When a new edge case shows up, add it to the guide and tell labelers which classes they mislabel.

Accuracy, IoU and agreement, the figures behind steps 2 to 7, get explained in annotation quality metrics.

Two people reviewing a wall of labeled aerial photos in an office

Who checks what at each review level

Annotation QA workflow review levels
Who checksWhat they doWhat it catches
The labelerChecks their own work before sending itSlips: a missed object, a wrong class picked
A second labelerLabels the same items, or checks another'sUnclear rules, where two people understand the guide in different ways
A reviewer or QA leadUses the guide to check a sample from each batchErrors repeated throughout a batch or across one labeler's work
Rule checks in the toolRuns on every labelBoxes outside the image, empty labels, duplicate shapes
Your own teamSpot-checks delivered batches against the gold setDrift from what your model needs

How companies and tools say their checking works

  • Scale AI: first, software labels the data; then people review it, using more than one level and more than one kind of check (docs.scale.com).
  • CVAT: honeypots, ground truth jobs and consensus among labelers (docs.cvat.ai).
  • Reviews and agreement scores come in the Enterprise edition of Label Studio (labelstud.io).

On their own pages, most of the 15 companies we compare say nothing about their checks. Have each company show you an earlier project's batch report: its scores, sample size and the outcomes for batches that didn't pass.

Data labeling vendor selection: what to test in the pilot

  • Vendors each receive the same items in a small batch and the same label guide; you keep a gold answer per item.
  • Check accuracy, IoU and missed objects on your own instead of relying on the vendor's figures.
  • Note which questions they raise about the guide. Good early questions lead to fewer errors later.
  • Measure the turnaround time and ask how they plan to staff the full volume.
  • Before any real data changes hands, ask for their audit reports (see data annotation security).
  • Don't judge by the headline rate; compare the price per label after rework.

A free or paid test appears on these companies' own pages:

Batch rules to write into the contract

  • The number of items checked in each batch, and who chooses them.
  • The pass marks: IoU for shapes, accuracy on gold items and agreement on twice-labeled items.
  • Batches that miss the mark are relabeled at the vendor's cost, then scored again.
  • The guide stays the reference. Every change to it is dated, and an older batch is judged by the guide version in effect when it was labeled.

Want companies to quote against these rules? Describe the job once, and up to three annotation companies reply with prices.

Get quotes from three companies