Idme Staff Software Engineer at ID.me owning the architecture and strategy for evaluation frameworks, automation tools, and testing infrastructure. Drive testing practices across functional, end-to-end, LLM/eval, and security-focused testing.
Responsibilities
Own Test Architecture & Strategy: Set the architecture and long-term strategy for automation frameworks and testing infrastructure (Playwright, Python, Jest, API automation) across ID.me's services, and ensure comprehensive, reliable coverage org-wide.
Lead Testing Strategy for Distributed Systems: Define end-to-end and integration testing strategy for complex, asynchronous, event-driven, and eventually-consistent systems, including workflow orchestration platforms such as Temporal.
Drive Testing-in-Production as an Org Standard: Build and champion canary deployments, progressive/phased rollouts, synthetic monitoring, and automated rollback triggers to validate real-world behavior safely before full release.
Own Security-Focused Test Automation: Set the standard for testing authentication/authorization flows (OAuth 2.0, OIDC), RBAC/ReBAC, secrets management, and tenant isolation.
CI/CD Quality Gates: Integrate automated testing into CI/CD pipelines (e.g., GitHub Actions) org-wide to give every team fast, reliable feedback.
Drive AI & Evals Culture: Build robust evaluation pipelines for AI-assisted features and LLM agents, while applying AI tools to accelerate test authoring, test plan generation, and code review across the org.
Cross-Team Collaboration: Partner with Security, Platform, and Product teams as a testing strategist, advising on testable code structure, coverage gaps, and quality checks throughout the development lifecycle.
Mentorship: Guide senior and mid-level engineers on test architecture, testing strategy, and quality practices.
Debug & Root Cause: Debug and root-cause issues across frontend, backend, APIs, and distributed services, and drive fixes to closure.
Reliability & Eval Metrics: Define and track reliability, quality, and eval accuracy metrics (e.g., time to detect, time to remediate, output quality scores) for the systems you evaluate and test.
Qualification
10+ years of experience (or equivalentHands-on experience with PlaywrightAPI & Systems TestingIdentity & Access ManagementSecurity & ComplianceTesting Strategy & OpsEngineering & PlatformAI-Augmented TestingAI Agent EvalsBackground in identity verification
Required
10+ years of experience (or equivalent: 10+ years with a Master's degree, or a PhD with 5+ years) as a software engineer in modern web applications and distributed systems.
Proficiency in Python/TypeScript/JavaScript/Java.
Hands-on experience with Playwright, Jest, API automation, or evaluation frameworks for software/AI systems.
Engineering & Platform: Java/Spring Boot or Ruby on Rails; GCP, Docker, Kubernetes; observability and chaos engineering.
AI-Augmented Testing: Using/building agentic tools for test generation, code review, and test planning.
AI Agent Evals: Experience building eval frameworks for LLM agents (e.g., correctness graders, LLM-as-judge, human-in-the-loop evals, benchmark dataset curation); familiarity with agentic frameworks (Claude API/Anthropic SDK, BrainTrust, LangChain, LangGraph, CrewAI, or similar); production monitoring for AI systems (behavioral drift detection, output sampling, shadow scoring); red-teaming or adversarial testing experience for AI models or agents; experience with RAG pipelines, including data chunking and pre-retrieval data quality gates.
