xAI Controls Engineer at SpaceXAI responsible for designing and operating networks powering AI supercomputer campuses. Supports GPU training and inference clusters with a focus on high availability and low latency.
Responsibilities
SpaceXAI is looking for an exceptional network engineer with experience in mission-critical, large-scale production environments to support the design, build-out, and operation of networks that power our AI supercomputer campuses. As a member of the Supercomputer Infrastructure / Network Engineering team, you will provide design and operational support for the fabrics used by GPU training and inference clusters, site operations, automation and controls, and facilities teams. The ideal candidate thrives in intense, high-flux environments, brings a strong sense of urgency balanced with operational excellence, communicates clearly, and demonstrates high technical acumen.
Design and implement highly available, low-latency, high-bandwidth networks, carefully balancing routing, congestion control, and redundancy technologies for AI training fabrics, inference front-ends, storage, and site/OT networks.
Design and maintain supercomputer data center and campus networks in accordance with company network standards. Collaborate with adjacent infrastructure, compute, storage, SiteOps, and enterprise teams.
Evaluate, procure, and deploy network hardware including data-center class switches, NICs, firewalls, optical multiplexers, and related appliances supporting 400G/800G and beyond.
Contribute to maturing network automation tooling; implement configuration analysis, linting, validation, and scalable deployment frameworks (GitOps / IaC).
Plan and coordinate network change windows with stakeholders to perform software updates, hardware refreshes, cluster expansions, and general maintenance (including evenings and weekends when required by compute schedules).
Troubleshoot and resolve network-related issues affecting cluster health and job performance; publish root cause analysis (RCA) documentation and host retrospectives.
Provide direct networking support during cluster bring-up, expansion, and production training/inference campaigns; serve as on-call or networking responsible engineer during operations.
Proactively tailor network monitoring and telemetry (fabric health, congestion, packet loss, NCCL/collective performance) so issues are detected before they impact training or inference.
Continuously create and update network documentation, including architecture overviews, design drawings, fiber/cable plant records, and operational procedures.
Collaborate with cross-functional teams to identify and resolve potential design issues, especially systemic or cascading failure modes and false redundancy in AI fabrics and site networks.
Perform job walks with customers, vendors, and contractors to gather requirements and produce implementation plans for new halls, rows, and campus interconnects.
Qualification
Bachelor’s degree in computer scienceExtensive hands-on experience designingHands-on experience with CiscoExperience with switch port security
Required
Bachelor’s degree in computer science, computer engineering, or other STEM discipline and 3+ years of professional network engineering experience;
OR 5+ years of professional network engineering experience in lieu of a degree.
Extensive hands-on experience designing, deploying, supporting, and troubleshooting Layer 2 and Layer 3 networks in latency-sensitive and/or industrial / data-center environments.
Functional experience with multiple network vendors in production or lab environments.
Experience with GitOps and Infrastructure as Code frameworks, both as a user and contributor.
Preferred
Strong understanding of the OSI model and network standards.
Hands-on experience with Cisco, Arista, Juniper, and/or NVIDIA Spectrum-X data-center class switches.
Experience with RoCEv2 Ethernet AI/HPC fabrics; InfiniBand experience is a plus.
Working knowledge of AI training and inference traffic patterns and how they behave on the network (collectives, congestion, ECMP, adaptive routing). Familiarity with NCCL is a plus.
Experience with WDM and large-scale single-mode / multimode fiber plants, including OTDR and acceptance testing.
Experience with switch port security, network segmentation, QoS, multicast, and redundancy protocols.
Familiarity with network monitoring and Layer 1 test tools; experience building operational telemetry and dashboards.
Proficiency in scripting (Bash / PowerShell / Python) and automation frameworks (Terraform, Ansible, etc.).
