Toloka AI / careers
← All positions

Model evaluation · Toloka AI

AI Agent Evaluation Engineer

United StatesRemoteFull-time

Role Summary

Measure conversational-agent quality through direct conversation review and automated evaluation. Build reliable judgments of adherence, task outcomes, and conversation experience across languages.

This published opportunity is available through Toloka AI. Review the listing carefully for the specific work arrangement and requirements.

Responsibilities

Define rubrics and failure categories. Review calls and trace problems to likely causes. Build evaluation pipelines and regression sets. Calibrate automated judges against human labels. Report quality trends, uncertainty, and prioritized fixes.

Qualifications

Three or more years in engineering, applied ML, or data-intensive research, including one to two years with production LLMs; relevant degree or equivalent experience; Python; production evaluation-system experience; professional fluency in another language besides English.

Preferred Qualifications

Not disclosed

Benefits

Medical, dental, vision, flexible PTO, parental leave, retirement plan, home-office setup budget, and monthly wellness/communication stipend.

What happens next

Submit your interest online to receive a reference number. The opportunity owner or review team may contact selected applicants about next steps.

For accommodation requests, contact [email protected].