Profound Member of Technical Staff, Evals at Profound builds evaluation systems to measure AI model performance and reliability. This role designs benchmarks, scoring systems, and dashboards to ensure trustworthy AI marketing products.
Responsibilities
Build Profound’s eval system: golden datasets, scorers and LLM judges, programmatic verifiers, regression alerts, and dashboards
Write evals for every pipeline behind the product: brand extraction, visibility scoring, sentiment, FactCheck, and citation classification
Define what a great, mediocre, and broken AI Marketer run looks like, and measure it across multi-step tool use, analytical correctness, and content quality
Study how answer engines behave across runs, platforms, regions, and personas, and design the sampling and statistics that keep customer-facing numbers trustworthy
Close the loop from real usage: collect customer signals, cluster failure modes, and turn regressions into permanent guardrails
Build tooling for debugging model and agent behavior, plus the labeling workflows and annotation guidelines behind our ground truth
Make evals part of how we decide things: wired into CI, launch reviews, and model selection
Qualification
Strong Python and data skillsDeep familiarity with LLM evaluationSound experimental designYou believe a good eval is clearWork that shows how you thinkExperience with searchPublications at venues like NeurIPSFor this roleNote
Required
3+ years building evaluation or measurement systems, whether AI evals, experimentation, ranking, or search quality
Strong Python and data skills, with a track record of shipping production systems
Deep familiarity with LLM evaluation: LLM-as-judge, benchmark design, agent harnesses, calibration, and significance testing
Sound experimental design, and the instinct to separate signal from noise in non-deterministic systems
Strong opinions about how models and agents behave, and you keep up with current research
You believe a good eval is clear, reliable, and hard to game
Work that shows how you think: shipped benchmarks, datasets, open-source tools, or publications
Comfortable operating in a fast-paced startup environment with rapid iteration cycles
Preferred
Experience with search, retrieval, citation, or fact-verification systems
Publications at venues like NeurIPS, ICML, ICLR, ACL, or COLM
This is an on-site role in our NYC or SF office, designed for builders who thrive on speed, iteration, and meaningful impact.
For this role, the expected base salary range is $215,000 to $300,000. Profound’s total compensation package is designed to be competitive and includes base salary, equity, and a full range of benefits and perks. Final compensation will depend on factors such as your skills, experience, qualifications, and location, and will be determined during the interview process. Our recruiting team will share more details about the full compensation package and benefits as you move through hiring.
Note: All official communication from Profound will come from a @tryprofound.com email address. If you're contacted by anyone using a different domain, please disregard and report it as spam.
