Responsibilities

The Sr. Staff Platform / Data Reliability Engineer is responsible for making Databricks a reliable, secure, scalable, and operationally mature platform for enterprise use. This role focuses on the platform itself rather than domain-specific business modeling, ensuring the lakehouse can support multiple domains, regulated data handling, and a growing set of production workloads without becoming fragile, expensive, or hard to govern.
This role partners closely with the Senior Staff Data Engineer, who focuses more heavily on ingestion and medallion implementation patterns, and with the existing Cloud & Infrastructure team, which owns cloud accounts, networking, and foundational infrastructure. The Platform / Data Reliability Engineer owns the Databricks operational layer that sits above that foundation: reliability, observability, deployment standards, compute and job policies, and platform enablement patterns for internal users.
Own operational excellence for the Databricks platform, including monitoring, alerting, observability, incident response support, and production runbook patterns for data jobs and platform services.
Define and maintain CI/CD and promotion standards for Databricks assets, including workflows, jobs, notebooks, code packages, infrastructure configuration, and environment promotion from dev to prod.
Design and maintain platform standards for job orchestration, cluster and compute policies, service principal usage, environment isolation, and production execution reliability.
Establish reusable operational templates and enablement patterns for new domains onboarding to Databricks, including logging conventions, job tagging, metadata capture, and support handoff expectations.
Partner with the Senior Data Engineer to ensure ingestion and medallion patterns are implemented in a way that is observable, recoverable, cost-aware, and secure in production.
Work with the cloud/infrastructure team to align Databricks configuration and usage patterns with broader enterprise cloud standards, especially where commercial and future government-hosted environments are involved.
Help enforce technical controls for data segregation, access boundaries, and operational compliance in a highly regulated environment.
Track and improve platform health metrics such as job success rates, incident trends, data pipeline reliability, cost efficiency, and environment drift.
Document platform standards, operational expectations, and support models so the Databricks platform can scale beyond a small founding team.
Mentor internal engineers who are growing into platform responsibilities, helping expand Databricks operational knowledge within the team.

Sr. Staff Platform/Data Reliability Engineer, Databricks (R5537) at Shield · LumiJobs

Sr. Staff Platform/Data Reliability Engineer, Databricks (R5537)

Responsibilities

Qualification

Required

Preferred

Benefits

Company

Shield

More roles at Shield