Fal AI AI Fabric Architect at Fal AI owns the end-to-end network fabric for GPU clusters, tuning performance and debugging fabric issues. The role includes vendor evaluation, collaboration on automation, and mentoring network engineers.
Responsibilities
You are the architect of the network that our GPU clusters run on. You own the fabric end to end, from the NIC in each GPU server through the spine and leaf across the AI fabric, and you know that in AI infrastructure the network is core to the machine. As fal brings its own GPU capacity online across multiple sites, you will define the fabric architecture every site is built from, and then personally make sure it gets built and performs. This is not an architecture-only role. We are a small team moving fast, and the right person is as comfortable writing a port map, tracing a bad optic, or tuning congestion control on the data center floor as they are setting technical direction.
Own the fabric architecture for every site: GPU backend fabric (InfiniBand and/or RoCEv2), and GPU frontend Fabric
Choose topologies and make the key calls: rail-optimized vs. other designs, InfiniBand vs. Ethernet, oversubscription ratios, and how the fabric scales from one pod to many
Turn the architecture into a repeatable network blueprint: cabling plans, port maps, fabric addressing and routing design, and a fabric bill of materials (switches, NICs, optics, cables)
Get hands-on with bring-up at each site: be on the floor for the first racks, validate cabling and firmware, configure the fabric manager and switches, and run performance acceptance (NCCL tests, burn-in) before capacity goes live
Tune for real workload performance: congestion control, PFC/ECN, adaptive routing, and NCCL settings for both inference and training traffic
Debug fabric problems wherever they show up, from a flapping link to a job that’s mysteriously slow
Evaluate and select AI Fabric vendors and hardware, and push back with data when something doesn’t meet spec
Collaborate across automation and software to help build network monitoring, telemetry, and automation that let a small team run a large fleet, and be the top escalation point for AI Fabric incidents
Document as you go so each new site is faster than the last, and help hire and mentor the network engineers who join after you
Qualification
Deep expertise in InfiniBand (NDR/XDRComfortable automating with PythonNVIDIA Quantum/Spectrum-X400G/800G OSFP opticsExperience at a neocloud
Required
7+ years in data center or HPC networking, including direct responsibility for production GPU cluster fabrics
A track record of taking GPU fabrics from empty racks to production, with your own hands on the work, not only through vendors or integrators
Deep expertise in InfiniBand (NDR/XDR, UFM, subnet management) and/or RoCEv2 lossless Ethernet
Strong command of rail-optimized and Clos/fat-tree designs, BGP/ECMP, and EVPN/VXLAN
A clear understanding of how fabric design and tuning affect collective communication and real workload performance
Comfortable automating with Python, Ansible, or similar
Strong Linux skills and the ability to debug across the boundary between host, NIC, and fabric
Sound judgment on tradeoffs between cost, performance, reliability, and time to deploy