horizon3ai The Staff Software Engineer on the OTA Services team leads the design and operation of NodeZero's over-the-air update system. This role involves writing production code, producing architecture designs, and mentoring engineers.
Responsibilities
The Expert Engineer on the OTA Services team is the technical anchor for how NodeZero updates itself across thousands of customer deployments — without customer intervention, without downtime, and without introducing new vulnerabilities in the process. NodeZero runs as a container inside customer infrastructure. When a new image is ready, the OTA system has to find every deployed instance, negotiate the update, deliver it safely, validate it, and roll back gracefully if something goes wrong. Getting that right at scale, across diverse customer environments including air-gapped and restricted networks, is a genuinely hard systems problem.
This is a hands-on staff-level role. You will write production code, produce architecture designs, set the technical direction for update delivery and version management, and mentor P4 engineers on the team. The OTA Services team works in close partnership with the Image Build & Release team — your counterpart on the build side produces the artifacts you deliver. This role reports to the Client Platform Engineering Manager.
OTA Architecture: Design and own the over-the-air update system — the delivery protocol, version negotiation, rollout orchestration, health verification, and automatic rollback
Deployment Fleet Management: Own the systems that track the version state of every NodeZero deployment — know what's running where, what's pending, and what's stuck
Rollout Strategy: Design safe rollout strategies for updates — phased rollouts, canary deployments, customer pinning, and emergency hold mechanisms
Constrained Environment Delivery: Design delivery mechanisms that work reliably across diverse customer environments including air-gapped networks, strict egress policies, and minimal-footprint deployments
Reliability & Incident Response: Own OTA system reliability end-to-end — on-call for delivery failures, stuck updates, and version drift incidents
Mentorship: Mentor P4 engineers through design review, code review, and technical pairing; be the person the team turns to when delivery semantics get complicated
What You'll Bring
Distributed Systems & Delivery
8+ years of professional software engineering experience with deep specialization in distributed systems, remote update systems, or fleet management infrastructure
Deep understanding of remote update system design — delivery protocols, version negotiation, delta delivery, health verification, and rollback semantics
Qualification
We are a fully remote company
Preferred
Experience with Kubernetes operators or controller patterns — OTA delivery in Kubernetes environments often benefits from operator-style lifecycle management
Background in cybersecurity or experience building software for regulated or high-security environments
Familiarity with Fleet (or similar open-source fleet management tools) and where custom infrastructure is preferable
Experience with signed artifact delivery and chain-of-custody verification end-to-end
We are a fully remote company, and this job may require up to 5% of travel to be successful.