Senior Software Engineer, Production

United States · Remotefull time

Senior Software Engineer, Production at Matter Labs · LumiJobs

Remote

senior

H1B: no

Matter Labs Senior Software Engineer in Production at Matter Labs responsible for running production environments for institutional customers and reducing incident load through automation and fixes. Owns environment safety, incident restoration, and service level agreements.

Responsibilities

You join the Production Unit, the three-person team that owns ZKsync OS in production for our customers: detection, incidents, rollouts and the service objectives each customer is measured against. Today the work is mostly reactive: on-call, incidents, rollouts, RCAs. The job is to change that. Every page and every manual step is a candidate for a fix in the server, a better alert or an automation. About 30 percent of your time goes to code now, and that share grows as the reactive load shrinks.
You own the environments you are assigned end to end: whether a change is safe to roll out, how an incident gets restored and explained, and whether the customer's service objectives are being met. Component teams own their code; you read it, find the cause, and land the fix.
Run production for institutional customers. Hold the First Responder rotation, coordinate incidents to restoration, and write the RCA the customer's engineers read.
Decide what ships. Go/no-go on changes to your environments, verification before rollout and validation after, including executing sequencer rollouts.
Remove the cause. Turn pages, runbooks and manual steps into fixes in the sequencer, provers or tooling, so the same incident does not come back.
Measure the service. Own the SLA card for your customers, measured monthly against their contracts.
Finish across teams. Drive diagnosis and fixes through the component teams that own the code, without waiting to be asked.
Later, support customers who run the chain themselves, where the fix has to work in someone else's hands.
Within a year, on-call load in your environments is measurably lower because of what you shipped, and the unit can take on the next customer without adding people.
What we look for

Qualification

You are willing to hold a 24/7 rotation

Required

You find causes in code you did not write. Strong in a systems language such as Rust, Go, C++ or Java; Rust is learnable here.
You verify before you roll out and you can say which changes are reversible and which are not.
You make other teams finish the work without escalating.
You are willing to hold a 24/7 rotation.

Preferred

If you meet most of this and the rest is learnable, apply anyway.
This is not a Forward Deployed Engineer role, which builds and deploys for a customer from discovery to production readiness; this role operates what is in production. It is not DevOps (clusters, network, L1 nodes, backups and disaster recovery stay with DevOps), not partner management, and not a people-management role.