Mercor AI Infrastructure Engineer builds and operates GPU and accelerator platforms for AI workloads including clusters, schedulers, training, and inference serving. The role involves end-to-end system ownership from design through on-call support.
Responsibilities
Design, deploy and operate GPU clusters across AWS, GCP, and other hyperscalers/neoclouds/on-premise data centers, including custom Kubernetes and Slurm operators and the metadata services behind them.
Build scheduling that keeps accelerators busy: gang scheduling for distributed jobs, bin-packing to reduce fragmentation, and quotas and fair share for multi-tenant teams.
Run distributed training infrastructure on PyTorch Distributed, Ray, or Megatron, with fast checkpointing and model loading.
Deploy and tune high-throughput LLM serving with vLLM or similar: tensor and model/expert parallelism, prefill/decode disaggregation, KV cache offload, FP8/FP4 quantization.
Build benchmarking and traffic-replay tools that measure TTFT, TPOT and latency percentiles, and set per-config performance baselines.
Build health checks across hardware, network and software layers, and automated remediation that handles fault storms without human intervention.
Instrument fleet utilization with DCGM, Prometheus and Grafana. Write recording rules and alerts that teams rely on.
Ingest utilization, occupancy and billing data into a warehouse, attribute spend to teams and workloads, and support capacity planning.
Drive autoscaling, rightsizing and idle-capacity recovery programs with measurable cost results.
Write clear runbooks and documentation, and explain infrastructure tradeoffs to researchers, finance and leadership.
Build internal tools or dashboards that researchers and non-engineers use daily.
