Glean Product Manager, AI at Glean owns core systems that make AI reliable, efficient, and better over time. The role focuses on agent harness design and evaluation of AI models and benchmarks.
Responsibilities
As a Product Manager on Glean's AI team, you will own the core systems that make our AI reliable, efficient, and measurably better over time. We're looking for deep skills and hands-on experience in one or both of the following areas:
Agent harness: experience designing or shipping the execution layer around the model, including planning, tool calling, context management, sub-agents, skills, memory, and long-running tasks, and making the tradeoffs across intelligence, latency, and cost that determine how well agents perform.
Evaluation: experience building the evalsets, benchmarks, judges, and labeled data that gate launches, compare models and harness changes, and drive continuous hill-climbing on quality.
This is a highly technical, cross-functional IC role. You will work closely with AI engineering, data science, and platform marketing teams, and you will partner with leading model providers and help go-to-market teams turn quality improvements into customer proof points.
Set the product strategy and roadmap for your area, grounded in customer workflows, trace data, and eval results.
Define what "good" looks like for agentic work, including correctness, helpfulness, formatting, efficiency, and reliability, and turn it into clear, measurable launch bars.
Make evidence-based tradeoffs across models, prompts, tools, and harness changes, balancing quality against latency, cost, and capacity.
Decide when to use frontier models versus post-training open-weight models for specific tasks, weighing quality, cost, latency, and control, and define the data and evals needed to make post-trained models competitive.
Build and scale the operating model for evaluation, including datasets, LLM-as-judge systems, human labeling, and benchmarks.
Turn failed runs, poor traces, and user feedback into prioritized product and quality improvements.
Partner with model providers and strategic partners on model evaluation, launches, and joint quality work.
Give the team context, set direction, and build alignment across engineering, data science, and GTM teams without direct authority.
Qualification
You have a learning and growth mindset
Required
You have 4+ years of product management experience, ideally on AI/ML, developer-platform, infrastructure, or quality-tooling products serving large enterprises.
You have built or worked closely on LLM-powered agents or at-scale AI products, and you understand how harness design choices affect agent behavior.
You are fluent in LLM evaluation concepts: evalsets, LLM-as-judge, pairwise/side-by-side judging, judge calibration against human labels, benchmarks, and offline-versus-online alignment.
You are comfortable going deep on agentic systems, including multi-turn tasks, tool use, context and memory management, write actions, and tracing and observability.
You are rigorous with data and can turn ambiguous quality problems into crisp hypotheses, metrics, and experiments.
You have a strong technical background and can engage credibly with engineers and data scientists on system design and experiment results.
You are an excellent written and verbal communicator who enjoys engaging with GTM teams and customers, and you can bridge our technical choices with the outcomes our customers care about.
You have a proven track record of taking ownership, taking initiative, and delivering results.
You have a learning and growth mindset, and you are mission-first.
You are an early adopter of AI in your own product craft, and you use coding agents and AI tools to prototype, analyze, and move faster.
This role is hybrid (4 days a week in one of our SF Bay Area offices)
