Toloka AI / careers
← All positions

Data quality · Toloka AI

Quality Lead, Agentic AI Workflow Evaluation

San JoseOn-siteNot specified

Role Summary

Owns quality controls for human evaluations of complex AI-agent workflows. The role resolves reviewer disagreement and keeps scoring rules consistent. The position is based in San Jose.

This published opportunity is available through Toloka AI. Review the listing carefully for the specific work arrangement and requirements.

Responsibilities

Design audits, sampling and scoring standards. Re-score reviewer output and identify recurring errors. Lead calibration and document disputed decisions. Revise ambiguous evaluation criteria. Train reviewers and report quality trends. Cover delivery operations during the engagement manager's absence.

Qualifications

Bachelor's degree or equivalent experience; four years of relevant quality or senior-review work; AI evaluation and agentic-system experience; audit ownership, calibration and reviewer-training experience; strong writing; spreadsheets, dashboards and enough Python or SQL for analysis.

Preferred Qualifications

Not disclosed

Benefits

Not disclosed

What happens next

Submit your interest online to receive a reference number. The opportunity owner or review team may contact selected applicants about next steps.

For accommodation requests, contact [email protected].