Site Reliability Engineer - Fleet Command & Control at Etched · LumiJobs

Responsibilities

Define what a healthy inference system looks like. In collaboration with software teams building out management and telemetry systems, identify the signals that most clearly show an unhealthy site, rack, node or chip, keep refining them as the fleet changes, and decide which actions (drain, cordon, restart, reroute, roll back) run on their own and which page a person.
Set a common alerting standard. Define SLOs with each team (latency such as time to first token, error rate, availability), alert on user-facing symptoms, and verify that the metrics behind each alert measure what the alert assumes.
Cut alert noise. Suppress expected alerts during deploys and maintenance automatically, correlate related alerts into one incident, and retire pages nobody acts on.
Automate recovery. Turn runbooks into agentic hardware health recovery: service restarts, node drain and traffic shift, spare swaps. Build it with guardrails (scoped permissions, rate limits, blast-radius checks, an audit trail)..
Route every event to the right engineer. Classify events by domain, page the right domain SRE with context attached, and dispatch on-site technicians for physical work such as tray swaps, reseats and leak response.
Shape the fleet data. Work with Supercomputing and Infrastructure, who build the telemetry pipeline, so it captures the key signals. Set the tracing standard (OpenTelemetry) so every service can be followed end to end, and report uptime and reliability trends that drive engineering and capacity decisions.
Run operationally excellent incident management. Own severity levels, incident channels, communications, postmortems and corrective-action tracking. Design on-call rotations, handoffs and onboarding so on-call is sustainable.
Raise the reliability bar for services. Review service designs with their owners for failure modes and graceful degradation before they ship, and partner with Supercomputing on canaries, rolling node-by-node deploys and graceful drain, so updates stop causing downtime.
Share on-call in a follow-the-sun rotation with SREs in other time zones.
Be the first line of defense. Acknowledge, size the user impact, open the incident, and decide fast between restart, drain, rollback and escalation.
Find where requests are stuck. Trace a latency or throughput regression through the serving stack (queueing, prefill, decode, handoffs, cache) to the stage that's backing up.
Triage hardware on live racks. Work through errors: chips dropping off PCIe, firmware panic resets, chip-to-chip link errors, HBM ECC errors and power faults; run recovery tools and decide when a node comes out.

Site Reliability Engineer - Fleet Command & Control

Responsibilities

Benefits

Company

Etched

More roles at Etched