Role Summary
Evaluate model safety and the reliability of systems that investigate AI misuse. Develop representative tests that inform training, safeguard changes, and deployment decisions. Locations include New York City, San Francisco, and Seattle.
This published opportunity is available through Toloka AI. Review the listing carefully for the specific work arrangement and requirements.
Responsibilities
Test safety behavior across conversations and tool use. Measure investigation-agent precision, recall, and robustness. Build datasets representing observed misuse patterns. Validate grading accuracy and identify coverage gaps. Integrate evaluations into regression and release pipelines.
Qualifications
Data-pipeline and LLM experience; strong analysis; research prototyping and production coding; experimental design; relevant bachelor's degree or equivalent education, training, or experience.
Preferred Qualifications
Eight or more years of engineering; evaluation frameworks, abuse detection, red teaming, synthetic data, distributed processing, or prompt engineering.
Benefits
Vacation and parental leave; optional equity donation matching.
What happens next
Submit your interest online to receive a reference number. The opportunity owner or review team may contact selected applicants about next steps.
For accommodation requests, contact [email protected].