Senior Software Development Engineer in Test, ML

USA-WI-Madisonfull time

Apply on Employer Site

On Site

senior

$115k–$172k

H1B: yes

Sony Interactive Entertainment Senior Software Development Engineer in Test at Sony Interactive Entertainment focusing on ML/AI quality and automation. Lead quality initiatives and develop automation frameworks for ML-powered products.

Responsibilities

Define and complete quality strategies, test plans, and automation coverage for ML-powered services and platform components.
Use LLMs and other AI-assisted techniques to generate, expand, and maintain high-value test cases for ML-powered workflows.
Design scenario-based test suites for AI features, including adversarial prompts, edge cases, ambiguous inputs, and underrepresented user scenarios.
Lead QE efforts for multi-functional projects, driving risk assessment, dependency management, and release readiness.
Design, develop, and maintain scalable automation frameworks for backend services, APIs, and ML inference systems using Python and/or Java.
Build automated validation for ML and LLM outputs, including ranking behavior, score distributions, prompt/response quality, hallucination indicators, and probabilistic model evaluation.
Debug test failures, service anomalies, model inconsistencies, and AI behavior regressions to identify root causes and drive resolution.
Perform functional, integration, regression, API, end-to-end, performance, and reliability testing for distributed systems.
Improve automation reliability, reduce flakiness, and optimize execution efficiency.
Partner with engineering and ML teams to integrate automated testing into CI/CD pipelines and release workflows.
Collaborate across teams to establish scalable quality standards, tooling, and guidelines.

Qualification

Experience using LLMs to generateExperience with AI evaluation toolingStrong experience testing RESTful APIsProficiency in PythonFamiliarity with databasesStrong understanding of SDLCExcellent problem-solvingFamiliarity with ML infrastructurePrior work in content moderation MLExperience testing high-scaleFamiliarity with NodeExperience testing mobile

Required

Bachelor’s degree in Computer Science or equivalent
5+ years of experience as an SDET or QE engineer focused on backend and distributed systems.
Experience using LLMs to generate, transform, and prioritize test cases for AI-powered experiences.
Experience with AI evaluation tooling, prompt evaluation frameworks, model monitoring, or human-in-the-loop review workflows.
Strong experience testing RESTful APIs, microservices, and distributed architectures.
Proficiency in Python, Java, JS or similar languages for automation development.
Hands-on experience with automation frameworks such as pytest, JUnit, Selenium, Playwright, Cypress, or Appium.
Experience with CI/CD systems and test pipelines (Jenkins, GitHub Actions, etc.).
Experience with cloud and container technologies (AWS, GCP, Kubernetes, Docker).
Familiarity with databases, monitoring, and observability tools.
Strong understanding of SDLC, Agile methodologies, and release processes.
Excellent problem-solving, debugging, and communication skills.

Preferred

Experience validating ML outputs using statistical analysis or scenario-based testing approaches.
Familiarity with ML infrastructure, data pipelines, or model-serving platforms (Seldon, KServe, Ray Serve, etc.).
Prior work in content moderation ML, security, fraud detection, or adversarial ML.
Experience testing high-scale, low-latency online services.
Experience with Databricks or similar ML platform tooling.
Familiarity with Node.js, React, or modern frontend technologies.
Experience testing mobile, console, or other non-PC platforms.
What sets you apart

Benefits

Dental
Vision
401(k)
PTO
Wellness

Company

Sony Interactive Entertainment

View company page

Source: greenhouse

Active within the past hour

More roles at Sony Interactive Entertainment

Staff Software Engineer, Data Governance & Trust

USA-CA-San Mateo

IT Support Specialist (Contract)

USA-CA-Los Angeles-Jefferson

Staff Technical Program Manager - AI/ML

USA-CA-San Mateo

Staff Technical Program Manager

USA-CA-Telecommute