Twenty Staff Product Manager - AI Evals at Twenty to lead product roadmap and evaluation capabilities for agentic AI systems. Partner with AI and engineering teams to define performance metrics and improve agent reliability.
Responsibilities
Own or support the product roadmap for agentic AI evaluations, translating product-area goals into clear priorities and outcomes.
Partner closely with engineering and AI teams to understand evaluation needs, technical constraints, and user workflows.
Define how agent performance and product quality should be assessed, including useful measures of task success, reliability, and quality.
Turn technical evaluation needs into clear product requirements, workflows, and acceptance criteria.
Help operationalize evaluation workflows and success metrics so teams can consistently assess agent behavior and product changes.
Use evaluation results and feedback to identify gaps, inform product decisions, and improve the agent experience over time.
Qualification
You're a product-mindedYou take ownershipStrong communicationExperience in cybersecurity
Required
You're a product-minded, technically curious operator who is excited about how agentic systems work and how to make them better.
You enjoy partnering with engineers and AI teams, asking good questions, and working through ambiguity together.
You are rigorous about evidence and metrics, while recognizing that evaluation methods need to reflect real user and product needs.
You take ownership, communicate clearly, and are comfortable balancing learning, execution, and iteration.
Strong product experience with AI or agentic systems, including experience building or managing agentic systems through product work.
Experience directly building evaluations for agents or building the evaluation product, tooling, or system used to assess them.
Ability to translate technical evaluation needs into product requirements, priorities, and workflows in partnership with engineering and AI teams.
Ability to define or operationalize meaningful measures of agent performance and product quality.
Strong communication, collaboration, and product judgment in an evolving, technically complex environment.
Preferred
Experience in cybersecurity, offensive cyber, government, defense, intelligence, or another mission-driven environment.
Familiarity with evaluation approaches such as benchmarks, human or automated evaluation, regression testing, simulation, or analysis of agent traces.
Experience working with products that combine autonomous systems with meaningful human judgment and oversight.
