Member of Technical Staff - Evaluations

Required

• We need someone who:
• Proactive execution: You pick up use cases and come back with working benchmarks. You parallelize work, make progress visible, and finish without being chased.
• White-box about evals: You volunteer where your own evals or scores were wrong and what you changed, and you present numbers with their caveats attached.
• Work on the internal eval suite and infrastructure: keep it reliable, optimized, and reproducible at scale.
• Design held-out evals for priority capabilities such as agentic workflows, computer use, and cybersecurity, with documented construction and contamination checks.
• Build evals targeted at small edge models, where most public evaluations built for frontier models don't apply.
• Read generated outputs to debug unexpected scores before anyone acts on them.
• Turn customer and open-source feedback into environments usable for data, training, and evaluation, and partner with other teams to design evals for their specific needs.
• Release open-source evals, write papers about them, and publish failure-mode reviews of popular evaluations.
• Experience & Capabilities
• Eval design is where we need you to be unusually strong: you can take a high-level description of a capability and turn it into an evaluation that is accurate, general-purpose, separates good models from bad ones, and is hard to game.
• Evals you have built before that we can recognize or check, and that hold up on coverage, reproducibility, contamination handling, and the signal they were meant to give. At least one built from scratch: a benchmark, an eval paper, or real contributions to lm-eval-harness, lighteval, OLMES, HELM, or Inspect.