Own evaluations across FurtherAI: Build the evaluation frameworks and infrastructure that measure the accuracy, quality, and reliability of our AI agents across workflows, Assistant, document processing, and computer use.

Build high-quality datasets: Develop golden datasets, benchmarks, and regression suites, working closely with Forward Deployed Engineers and domain experts to capture real-world insurance tasks and edge cases.

Evaluate complex agent behavior: Design evaluations for multi-step agent trajectories, tool use, reasoning, and end-to-end task completion — not just individual LLM responses.

Build automated grading systems: Develop LLM-as-judge evaluations, deterministic graders, human evaluation workflows, and rubrics that accurately reflect real-world quality.

Make development evaluation-driven: Integrate offline and online evaluations into development and release processes so engineers can measure improvements, catch regressions, and ship confidently.

Drive model evaluation and selection: Continuously benchmark frontier and open-weight models across accuracy, latency, cost, and reliability. Use results to inform model selection, upgrades, and production releases.

Build continuous improvement loops: Turn production failures, customer feedback, and agent traces into new evaluation cases and measurable improvements in agent quality.

Representative Projects

Build an end-to-end evaluation framework for FurtherAI agents, measuring success across multi-step workflows, tool calls, and complex insurance tasks.

Create a library of golden datasets and benchmarks for common insurance workflows, reusable across customer implementations and out-of-the-box AI skills.

Build an automated benchmarking system that evaluates new frontier and open-weight model releases and identifies the best models for different tasks.

Develop production evaluation pipelines that detect quality regressions, identify failure patterns, and turn them into actionable improvements.

Senior / Staff AI Engineer — Evals & Model Quality

Responsibilities

Qualification

Required

Benefits

Company

Further AI

More roles at Further AI