Research Scientist

San Francisco · Remotefull timeRemote

Apply on Employer Site

$200k–$240k

H1B: no

Outset Research Scientist at Outset designs and runs studies comparing AI system-generated responses with human responses to improve digital twin fidelity. The role involves applying advanced statistical techniques and designing human rating studies to benchmark and enhance model performance.

Responsibilities

Co-design and run studies that compare system-generated responses with responses from humans, using both qualitative and quantitative measures.
Design, justify, and hone measures of performance.
Apply and adapt state-of-the-art statistical techniques common to survey methodology and large-scale social data analysis, to benchmark and improve sample- and population-level simulation performance.
Design evaluations and experiments with pre-specified analysis plans.
Design and oversee human rating studies, including rater training, coding rubrics, and inter-rater agreement measurement.
Design data collection from third-party data sources and directly from human participants, including longitudinal and repeated-measures studies.
Contribute to the research agenda for new modeling approaches, including evaluation of fine-tuned models and of model calibration.
Specify analyses precisely enough for an engineer to implement.
Co-author papers, preprints, and presentations.

Qualification

Familiarity with machine learningFundamentals of computingPrior work on generative agentsFamiliarity with the startup ecosystem

Required

A completed or nearly completed PhD in computational social science, communication, HCI, sociology, political science, psychology, statistics, or a related field.
Peer-reviewed publications on which you were a primary author.
Working command of graduate-level statistical methodology common to social science, including the assumptions behind each.
Experience analyzing large-scale telemetry and behavioral datasets, with attention to the validity of the resulting inference.
Familiarity with machine learning, spanning classical, nonparametric, and LLM-based approaches, including how transformer-based language models work, what fine-tuning does, and how the major model families differ.
Fundamentals of computing: file systems, software libraries, databases, SQL, and tabular data.
Ability to work in an environment where most code is produced through agentic coding tools, and to read and check that code.

Preferred

Experience fine-tuning open-weight language models on survey, experimental, or behavioral data.
Experience building or evaluating calibrated confidence or error-prediction models, including deriving decision thresholds from human judgments.
Experience measuring similarity of open-ended text, for example with embedding-based methods.
Experience designing or running panel-based data collection with human participants.
Experience presenting research at academic and/or industry conferences.
Prior work on generative agents, digital twins, or simulation of human respondents.
Familiarity with the startup ecosystem
Familiarity with qualitative interviews and ethnographic methods

Company

Outset

View company page

Source: ashby

Active within the past hour

Remote-friendly

More roles at Outset

Data Analyst

San Francisco · Remote

Research Engineer

San Francisco · Remote

Account Executive - Retail

San Francisco · Remote

Account Executive - Pharma

San Francisco · Remote