Ayahealthcare Product Manager IV at Aya Healthcare owning platform capabilities for AI systems focusing on quality, safety, and trustworthiness. Connects evaluation, governance, observability, risk management, and product delivery.
Responsibilities
Understand needs and friction points of AI developers, product teams, evaluators, data teams, and operators.
Define the platform contract: what every squad can rely on, written down, versioned and enforced by the pipeline.
Prioritize reusable capabilities for model and tool integration, context access, CI/CD, environments, feature flags, observability, evaluation, and incident response.
Prioritize scenario libraries, automated evaluations, human review, red teaming, monitoring and incident investigation. Replace release gates with thresholds the pipeline enforces.
Partner with Engineering and Architecture to define APIs, platform contracts, documentation, onboarding, and operational standards.
Ensure secure access, least privilege, data protection, auditability, and separation of development, staging, and production.
Define measures including time to first deployment, deployment frequency, change failure rate, reliability, and developer satisfaction.
Make the decision record and the evaluation suite the documentation: intended behavior, limitations, risk controls, provenance and model or prompt changes live there and nowhere else.
Manage tradeoffs between standardization, autonomy, speed, cost, and maintainability.
Create feedback loops through internal research, usage analytics, support data, and platform reviews.
Communicate platform strategy and investment needs to technology and product leaders.
Publish platform health and adoption on the live status page. No status meetings.
Qualification
Working understanding of AI evaluationStrong analyticalExperience with LLM evaluation
Required
Bachelor’s degree or equivalent experience in product management, engineering, data, quality, risk, security, or a related field.
5+ years of product management or technology experience with quality, platform, governance, analytics, or enterprise systems.
Working understanding of AI evaluation, software quality, observability, privacy, security, and risk management.
Ability to convert principles and policy requirements into usable workflows and measurable controls.
Strong analytical, communication, facilitation, and cross-functional leadership skills.
Comfort working with ambiguity and making risk-based prioritization decisions.
Preferred
Experience with LLM evaluation, red teaming, model monitoring, responsible AI, MLOps, or AI governance.
Experience establishing enterprise quality or trust platforms.