Toloka AI / careers
← All positions

Model evaluation · Toloka AI

Tech Lead Manager, AI Evaluations

United StatesHybridNot specified

Role Summary

Lead evaluation methods and systems for assistants and autonomous agents. Measure correctness, grounding, relevance, and task success to guide product decisions. Based in Mountain View or San Francisco.

This published opportunity is available through Toloka AI. Review the listing carefully for the specific work arrangement and requirements.

Responsibilities

Define quality metrics and representative evaluation sets. Calibrate automated graders against human judgments. Integrate regression and production evaluations. Analyze failure patterns and quality tradeoffs. Lead and develop evaluation scientists and engineers.

Qualifications

Ten or more years of relevant technical experience; five or more years managing engineering or applied-AI teams; production evaluation experience; strong programming; MS/PhD in a relevant field.

Preferred Qualifications

Safety testing, public-benchmark knowledge, and cloud/Kubernetes experience.

Benefits

Medical, dental, vision, time off, 401(k), home-office stipend, education/wellness stipends, and daily lunches.

What happens next

Submit your interest online to receive a reference number. The opportunity owner or review team may contact selected applicants about next steps.

For accommodation requests, contact [email protected].