Manager, AI Engineering and Vendor Operations

Required

• 5+ years of experience building and shipping production AI or machine learning systems, with deep hands-on knowledge of large language models and the systems built around them.
• 2+ years of experience managing or mentoring engineers.
• Experience setting engineering practices for a team, such as code review, testing, or deployment standards.
• Experience working with technology vendors and managing software tools, including security and data privacy reviews.
• Experience building automated testing, including methods for evaluating the quality of AI model outputs.
• Hands-on experience building in a major cloud environment (Azure, AWS, or GCP).
• Experience setting up automated testing, including methods for evaluating the quality of AI model outputs.
• Experience working across technical and non-technical teams, including explaining technical and cost trade-offs to people without a technical background and working with product managers to gather requirements from stakeholders.
• Strong Python skills and working knowledge of retrieval-augmented generation (RAG) and semantic search.
• Proficiency in design and architecture of AI systems and systems of AI evaluation
• Proficiency in SQL and cloud data warehouses such as BigQuery or Snowflake.
• Familiarity with model evaluation methods, prompt engineering, and API integration, including NPR's approved model families such as Claude, Gemini, Gemma, and Llama.