xAI The Director of Site Operations owns node and rack uptime for SpaceXSI's AI supercompute cluster across 5+ sites operating 24/7. This role leads a 250+ person team and site reliability engineering to ensure exceptional cluster health and uptime.
Responsibilities
Own cluster uptime by serving as extreme owner of node, rack, and cluster health across 5+ sites running 24/7, accountable for customer Service Level Agreements and consistently exceptional uptime on SpaceXSI's supercompute cluster
Lead a large operations organization, directing a 250+ person team spanning site managers, shift supervisors, and technicians across four 24/7 shifts, building a culture of excellence and accountability at every layer of the org
Drive node and rack remediation by ensuring systematic recovery of failed nodes and racks through command-line and physical intervention, driving mean time to repair to the feasible minimum
Partner across functions by coordinating with facilities operations to limit downtime from power and cooling faults and proactive maintenance; with network engineering on cluster upgrades; and with tenant representatives on node remediation and planned and unplanned downtime
Own vendor execution, directing vendors through hardware rework and field operations so repairs, replacements, and capacity work happen at the speed the cluster requires
Lead site reliability engineering, owning the SRE organization responsible for proactive cluster health monitoring, reactive fault mitigation at scale, root cause analyses for node, rack, and cluster issues, and site-wide reliability procedures and fault documentation
Run data-driven improvement by leading continual improvement and efficiency initiatives, using operational data to balance team resources and raise uptime, repair time, and SLA performance across sites
Command incidents at scale and set the standard for incident response during cluster-impacting events, providing clear direction, fast recovery, and tight communication with internal and external partners
Scale operations by standardizing best practices across sites and grow the organization in step with cluster expansion, keeping operations consistent as SpaceXSI's footprint scales
Qualification
Proven ability to lead largeDeep expertise in server hardwareA track record of owning uptime
Required
Bachelor’s degree and 7+ years of experience working in a large scale operations with 5+ years leading people leaders of technical teams; OR 10+ years of experience working in a large scale operations with 5+ years leading people leaders of technical teams
Preferred
Proven ability to lead large, multi-site, 24/7 operations organizations in fast-paced, high-responsibility settings
Deep expertise in server hardware, cluster reliability, and data center technologies, from deployment through lifecycle management
Experience supporting compute-heavy environments like AI, machine learning, or high-performance computing at scale
A track record of owning uptime, Service Level Agreements, or reliability metrics for large compute clusters
Experience leading site reliability engineering or equivalent reliability-focused teams, including root cause analysis and procedure ownership
Strong analytical skills and the ability to explain technical concepts clearly to diverse audiences, from technicians to executive and tenant partners
A history of partnering with vendors at scale, driving mean time to repair down, and scaling operations across multiple sites
Familiarity with tooling and automation (e.g., Jira, Python, Bash) used to monitor cluster health and improve team efficiency
