RLHF services
With RLHF services, people rank, grade and rewrite what a language model answers, teaching the model what a good answer looks like. Basic ranking work is priced per comparison; grading by experts is priced per hour.
Rough cost for a feedback job
5,000 comparisons overall, drawn as squares of 100 comparisons
For 5,000 comparisons, the going rate abroad comes to roughly
$2,500 to $40,000
Teams abroad list between $0.50 and $8.00 for each comparison.
Get quotes for this jobThe calculation behind this
The published low and high price for that label type gets multiplied by the label count. Another review adds 20 to 50 percent, while a US team costs two to four times the outside-US range.
Prices for answer rankings come from Habile Data.
Prices for sentiment come from GDS Online.
Prices for text entities come from Habile Data.
Picking generalists or experts
Trained generalists can rank chat answers for tone and clarity. Experts are needed to grade code, medical or legal answers; their hourly rate is far higher and pay is per task.
Vendors listing this work
Audit names explained
- SOC 2 a check by an outside auditor of how a company protects its clients' files. Type I checks a single day; Type II checks several months, so it tells you more.
- ISO 27001 a worldwide data-safety standard; an independent auditor confirms the company follows it.
- ISO 27701 adds checks on personal details, such as names and addresses, on top of ISO 27001.
- ISO 9001 checks the steps a company uses to keep its work consistent. It does not cover keeping your files safe.
- HIPAA a US law that protects patients' medical records.
- TISAX a security check that car manufacturers ask their suppliers to pass.
- GDPR the EU's data protection law.
- CCPA a California law on consumer privacy.
| Company | Based in | Kinds of files | Audits and laws it names | How it prices |
|---|---|---|---|---|
| Scale AIscale.com | San Francisco, California | Photos, Video, Text, 3D laser scans (LiDAR), Satellite and map images, Rating AI chatbot answers | SOC 2 Type II | Per project |
| Appenappen.com | Sydney, Australia and Kirkland, Washington | Photos, Video, Text, Sound, 3D laser scans (LiDAR), Medical scans, Rating AI chatbot answers | SOC 2 Type II, ISO 27001, HIPAA, GDPR | Per project |
| TELUS Digitaltelusdigital.com | Vancouver, British Columbia, Canada | Photos, Text, Sound, 3D laser scans (LiDAR), Rating AI chatbot answers | SOC 2, TISAX, ISO 27001 | Per project |
| iMeritimerit.ai | San Jose, California | Photos, Video, Text, Sound, 3D laser scans (LiDAR), Medical scans, Rating AI chatbot answers | SOC 2 Type II, ISO 27001, ISO 9001, HIPAA, GDPR, TISAX | Per project |
| Shaipshaip.com | New York, New York | Photos, Video, Text, Sound, 3D laser scans (LiDAR), Medical scans, Rating AI chatbot answers | ISO 27001, ISO 9001, SOC 2 Type II, HIPAA | Per project |
| Cogito Techcogitotech.com | Levittown, New York | Photos, Text, Medical scans, Rating AI chatbot answers | SOC 2 Type II, ISO 27001, ISO 9001, HIPAA, GDPR, CCPA | Per project |
| Mindy Supportmindy-support.com | Limassol, Cyprus | Photos, Video, Text, Sound, 3D laser scans (LiDAR), Rating AI chatbot answers | SOC 2, GDPR | Per project; teams start at about 5 people working full time on your files |
| Tolokatoloka.ai | Amsterdam, Netherlands | Photos, Video, Text, Sound, Rating AI chatbot answers | SOC 2, ISO 27001, ISO 27701 | Per project |
| Surge AIsurgehq.ai | San Francisco, California | Text, Rating AI chatbot answers | GDPR | Per project |
| Innodatainnodata.com | Ridgefield Park, New Jersey | Photos, Video, Text, Sound, 3D laser scans (LiDAR), Medical scans, Satellite and map images, Rating AI chatbot answers | ISO 9001, ISO 27001 | Per project |
Most companies first look at a sample of your files, then give you one price for the whole job ("per project"). Fill in the short form one time and get prices back from three of them.
Common questions about RLHF services
What do RLHF services cost?
At the low end, generalist preference rankings cost about $0.50 per comparison; at the high end, about $8. Jobs for experts, like grading code or medical answers, can cost $20 to $100 or more per task.
Who labels model feedback for RLHF?
Each vendor lists this work on its own site. The group is iMerit, Shaip, Toloka, Surge AI, Innodata, Scale AI, Appen, TELUS Digital, Cogito Tech and Mindy Support. The vendor table shows, row by row, where each company is based and the way it charges for the work.
Among these vendors, which ones state ISO 27001 or SOC 2?
On their own pages, 8 of the 10 say they have a SOC 2 report; 7 say ISO 27001. An outside audit of client-data protection is SOC 2; the international standard for running information security is ISO 27001.
Send one brief, hear from three vendors
Describe your data once and we pass it to three annotation vendors that label that kind of data, so you can compare quotes side by side.
- One short form instead of three sales calls
- Matched on data type, volume and security needs
- You choose who to talk to