Outset The Research Engineer at Outset builds and maintains systems to ensure reproducible, scalable AI-powered research experiments. This role supports scientists by implementing evaluation infrastructure and data pipelines.
Responsibilities
Own the evaluation infrastructure that tests competing modeling approaches, so that new tests can be set up quickly, run concurrently and automatically, and be monitored.
Implement measurement pipelines exactly as specified by the scientists, with deterministic and traceable outputs.
Maintain versioned evaluation datasets with held-out sets that are never used for training.
Build and maintain data pipelines for human-participant data, external datasets, and model training data, including linkage of records belonging to the same participant across sources.
Keep research code in step with the production product so that validated improvements ship promptly.
Detect and prevent threats to the validity of evaluations.
Track compute and API spend across experiments and their implications for production-stage costs
Manage repositories, code review, documentation, and the reproducibility of the team's coding workflow.
Identify implementation choices that can be made more correct or more efficient, and propose changes.
Qualification
Strong PythonFundamentals of computingA habit of writing deterministicPrior work on generative agents
Required
Several years of software engineering experience, including experience supporting researchers in a lab or comparable setting.
Experience translating specific scientific requirements and experiment designs into code.
Experience building and operating large-scale data pipelines on human behavior data, and confirming that the pipelines matched the criteria and design the researchers specified.
Strong Python, including the ability to read and audit code produced by agentic coding tools.
Fundamentals of computing: file systems, software libraries, databases, SQL, tabular data, and Git-based workflows.
Experience with cloud infrastructure and with running many jobs concurrently.
Working understanding of machine learning, including how transformer-based language models behave, how model APIs are used, and what fine-tuning involves.
A habit of writing deterministic, reproducible, cost-aware, and well-documented code.
Preferred
Hands-on experience engineering large language model systems: prompt pipelines, evaluation harnesses, embedding computation, agentic systems, and/or fine-tuning infrastructure.
Experience fine-tuning open-weight language models on multi-GPU hardware.
Experience serving open-weight models with an inference framework such as vLLM or SGLang, including batched inference and prefix caching for many requests that share a common prompt.
Experience with experiment tracking and cost monitoring for machine learning workloads.
Experience with survey or panel data and data collection platforms.
Experience extracting and storing model activations for downstream analysis.
Experience with experiment tracking and cost monitoring for machine learning workloads, including GPU provisioning across cloud providers.
Prior work on generative agents, digital twins, or simulation of human respondents.
