Role Summary
Lead evaluation methods and systems for assistants and autonomous agents. Measure correctness, grounding, relevance, and task success to guide product decisions. Based in Mountain View or San Francisco.
This published opportunity is available through Toloka AI. Review the listing carefully for the specific work arrangement and requirements.
Responsibilities
Define quality metrics and representative evaluation sets. Calibrate automated graders against human judgments. Integrate regression and production evaluations. Analyze failure patterns and quality tradeoffs. Lead and develop evaluation scientists and engineers.
Qualifications
Ten or more years of relevant technical experience; five or more years managing engineering or applied-AI teams; production evaluation experience; strong programming; MS/PhD in a relevant field.
Preferred Qualifications
Safety testing, public-benchmark knowledge, and cloud/Kubernetes experience.
Benefits
Medical, dental, vision, time off, 401(k), home-office stipend, education/wellness stipends, and daily lunches.
What happens next
Submit your interest online to receive a reference number. The opportunity owner or review team may contact selected applicants about next steps.
For accommodation requests, contact [email protected].