Own monitoring architecture and signal quality across GPU and host telemetry (DCGM, Prometheus/Grafana) and facility telemetry (BMS, EPMS, DCIM, SCADA, CDU / direct liquid cooling): what we alert on, suppress, and trust. Consume NOC noise-disposition feedback to drive suppression and redesign. Treat alert noise as a design failure, not an operator failure

Provide SEV command support: technical incident leadership, bridge coordination with the NOC, paging and escalation (e.g., PagerDuty), and timeline and severity hygiene

Run blameless postmortems and drive corrective actions to closed, not filed

Gate high-risk change: fleet-scale firmware/BIOS/BMC rollouts (Redfish, canaries, rollback) and SOP/MOP/EOP-governed work on power and cooling systems through a disciplined change management process

Lead cross-functional reliability projects spanning compute, network (InfiniBand/RoCE), storage, and facility signal boundaries, including grey failures that cross those boundaries (e.g., NVLink flapping, thermal throttling, power sequencing)

Partner with Hardware Failure Analysis on systemic failure classes: turn fleet data (MTBF/AFR, repeat-RMA trends) into monitoring, stop-rules, and vendor RCA inputs

Build and maintain playbooks, run game days, and keep cross-discipline dependency maps current. Own runbook quality jointly with the NOC (SRE designs; NOC operates and corrects)

Define error budgets and availability objectives at campus and service boundaries as adopted by the business

Participate in on-call rotations and incident response for SEV-class events in the Memphis / Southaven data center campus

Demonstrate monitoring and observability design at fleet or campus scale, including alert hygiene, suppression, and signal quality

Treats alert noise as a design failure, not an operator failure

Improve mean time to detect (MTTD) and mean time to resolve (MTTR) for severity-class events, and ensure blameless postmortems with closed action items.

Site Reliability Engineer - Memphis

Responsibilities

Qualification

Required

Preferred

Company

xAI

More roles at xAI