Role Summary
Owns quality controls for human evaluations of complex AI-agent workflows. The role resolves reviewer disagreement and keeps scoring rules consistent. The position is based in San Jose.
This published opportunity is available through Toloka AI. Review the listing carefully for the specific work arrangement and requirements.
Responsibilities
Design audits, sampling and scoring standards. Re-score reviewer output and identify recurring errors. Lead calibration and document disputed decisions. Revise ambiguous evaluation criteria. Train reviewers and report quality trends. Cover delivery operations during the engagement manager's absence.
Qualifications
Bachelor's degree or equivalent experience; four years of relevant quality or senior-review work; AI evaluation and agentic-system experience; audit ownership, calibration and reviewer-training experience; strong writing; spreadsheets, dashboards and enough Python or SQL for analysis.
Preferred Qualifications
Not disclosed
Benefits
Not disclosed
What happens next
Submit your interest online to receive a reference number. The opportunity owner or review team may contact selected applicants about next steps.
For accommodation requests, contact [email protected].