AI Safety Policy Evaluator, Violence & Threats | Remote US at Handshake · LumiJobs

Responsibilities

As an AI Policy Specialist on the Violence & Fiction team, you will help AI models learn where the line falls between depicting violence and enabling it.
Violence is one of the hardest domains in AI safety because most violent content is legitimate. Novels, games, screenplays, history, journalism, self-defense, and ordinary human frustration all involve violence, and a model that refuses them is broken. A model that helps someone plan real harm is worse. Your job is to tell the difference, case by case, and to explain your reasoning clearly enough that it can train a model.
We are looking for people who already have strong instincts about violence in at least one of these areas: how it works in fiction, how it works in the real world, or how it shows up in people who are struggling. You do not need all three. You need one deep and the judgment to learn the rest.
This is not rote annotation. Policies cannot anticipate every edge case, and good evaluators do not apply them mechanically. You will balance policy text and intent with customer expectations, conversation context, precedent, and team calibration.
Evaluate user requests and AI model responses involving violence, weapons, threats, and dark fiction within the full conversation context
Distinguish fictional, educational, historical, and defensive violence from requests that seek real-world uplift or express real intent to harm
Assess whether a model's response gives meaningful real-world capability, regardless of how the request was framed
Distinguish expressions of anger, frustration, or dark humor from credible threats or crisis indicators
Select the most defensible classification when a case is genuinely ambiguous, and write concise rationales that cite policy language and conversation details
Write and refine adversarial or borderline prompts that probe where a model draws the line
Identify policy gaps, contradictions, and emerging edge cases, and raise them with project leads and policy teams
Participate actively in calibration discussions; challenge interpretations respectfully and update your judgment when stronger reasoning emerges

AI Safety Policy Evaluator, Violence & Threats | Remote US

Responsibilities

Qualification

Required

Preferred

Company

Handshake

More roles at Handshake