Role Summary
Measure conversational-agent quality through direct conversation review and automated evaluation. Build reliable judgments of adherence, task outcomes, and conversation experience across languages.
This published opportunity is available through Toloka AI. Review the listing carefully for the specific work arrangement and requirements.
Responsibilities
Define rubrics and failure categories. Review calls and trace problems to likely causes. Build evaluation pipelines and regression sets. Calibrate automated judges against human labels. Report quality trends, uncertainty, and prioritized fixes.
Qualifications
Three or more years in engineering, applied ML, or data-intensive research, including one to two years with production LLMs; relevant degree or equivalent experience; Python; production evaluation-system experience; professional fluency in another language besides English.
Preferred Qualifications
Not disclosed
Benefits
Medical, dental, vision, flexible PTO, parental leave, retirement plan, home-office setup budget, and monthly wellness/communication stipend.
What happens next
Submit your interest online to receive a reference number. The opportunity owner or review team may contact selected applicants about next steps.
For accommodation requests, contact [email protected].