Toloka AI / careers
← All positions

Model evaluation · Toloka AI

Research Engineer, Post-Training Model Evaluations

United StatesHybridNot specified

Role Summary

Assess model quality throughout production post-training runs. Improve measurement reliability and investigate unexpected results that influence training and release decisions. Based in San Francisco or Seattle.

This published opportunity is available through Toloka AI. Review the listing carefully for the specific work arrangement and requirements.

Responsibilities

Optimize the evaluation mix for reliable quality measurement. Monitor evaluations during training runs. Advise researchers on evaluation methods. Investigate regressions and recommend interventions. Build quality dashboards, alerts, and reports.

Qualifications

Experience designing and analyzing scaled ML evaluations; strong Python and production-system skills; rigorous debugging and communication; relevant bachelor's degree or equivalent education, training, or experience.

Preferred Qualifications

LLM post-training, benchmarking research, statistics, experimental design, and evaluation-metric development.

Benefits

Vacation and parental leave; optional equity donation matching.

What happens next

Submit your interest online to receive a reference number. The opportunity owner or review team may contact selected applicants about next steps.

For accommodation requests, contact [email protected].