Role Summary
Assess model quality throughout production post-training runs. Improve measurement reliability and investigate unexpected results that influence training and release decisions. Based in San Francisco or Seattle.
This published opportunity is available through Toloka AI. Review the listing carefully for the specific work arrangement and requirements.
Responsibilities
Optimize the evaluation mix for reliable quality measurement. Monitor evaluations during training runs. Advise researchers on evaluation methods. Investigate regressions and recommend interventions. Build quality dashboards, alerts, and reports.
Qualifications
Experience designing and analyzing scaled ML evaluations; strong Python and production-system skills; rigorous debugging and communication; relevant bachelor's degree or equivalent education, training, or experience.
Preferred Qualifications
LLM post-training, benchmarking research, statistics, experimental design, and evaluation-metric development.
Benefits
Vacation and parental leave; optional equity donation matching.
What happens next
Submit your interest online to receive a reference number. The opportunity owner or review team may contact selected applicants about next steps.
For accommodation requests, contact [email protected].