Physicalintelligence ML Infra Engineer responsible for scaling and optimizing training systems and core model code. Own infrastructure for large-scale training including job orchestration and reusable JAX training pipelines.
Responsibilities
This is a hands-on, high-leverage role at the intersection of ML, software engineering, and scalable infrastructure.
The ML Infrastructure team supports and accelerates PI’s core modeling efforts by building the systems that make large-scale training reliable, reproducible, and fast. The team works closely with research, data, and platform engineers to ensure models can scale from prototype to production-grade training runs.
Own training/inference infrastructure: Design, implement, and maintain systems for large-scale model training, including scheduling, job management, checkpointing, and metrics/logging.
Scale distributed training: Work with researchers to scale JAX-based training across TPU and GPU clusters with minimal friction.
Optimize performance: Profile and improve memory usage, device utilization, throughput, and distributed synchronization.
Enable rapid iteration: Build abstractions for launching, monitoring, debugging, and reproducing experiments.
Partner with researchers: Translate research needs into infra capabilities and guide best practices for training at scale.
Contribute to core training code: Evolve JAX model and training code to support new architectures, modalities, and evaluation metrics.
What We Hope You’ll Bring
Strong software engineering fundamentals and experience building ML training infrastructure or internal platforms.
Hands-on large-scale training experience in JAX (preferred), PyTorch.
Familiarity with distributed training, multi-host setups, data loaders, and evaluation pipelines.
Qualification
Deep ML systems background (eBackground in robotics
Required
Strong software engineering fundamentals
Experience building ML training infrastructure or internal platforms
Hands-on large-scale training experience in JAX (preferred) or PyTorch
Familiarity with distributed training and multi-host setups
Experience managing training workloads on cloud platforms (SLURM, Kubernetes, GCP TPU/GKE, AWS)
Ability to debug and optimize performance bottlenecks
Preferred
Deep ML systems background (e.g., training compilers, runtime optimization, custom kernels).
Experience operating close to hardware (GPU/TPU performance tuning).
Background in robotics, multimodal models, or large-scale foundation models.
Experience designing abstractions that balance researcher flexibility with system reliability.
Pursuant to the San Francisco Fair Chance Ordinance, we will consider for employment qualified applicants with arrest and conviction records.