Okta Staff Observability Site Reliability Engineer specializing in Splunk to own and evolve the Splunk ecosystem. Responsible for designing scalable observability infrastructure and automating deployment of agents across distributed systems.
Responsibilities
Automated Infrastructure: Design, build, and maintain scalable observability infrastructure using tools like Terraform.
Splunk Engineering: Optimize the collection, processing, and storage of log data to ensure high reliability and low latency of our Splunk services
Incident Response: Participate in on-call rotations and lead post-incident reviews to drive systemic improvements and "observability-driven development."
Automation: Eliminate "toil" by automating the deployment and scaling of observability agents and collectors.
Qualification
Log ManagementSRE MindsetProgramming ProficiencyDistributed SystemsProblem SolvingBonus Skills (The "Nice-to-Haves")Telemetry StandardsCharge-back appCloud PlatformsAdditional requirementsExperience with OpenTelemetry
Required
Log Management: Minimum 5+ Experience scaling and managing Splunk Cloud at scale (1000+ SVCs), including Workload Management (WLM) and HEC optimization. Visualization: Expertise in creating intuitive, actionable Splunk dashboards that correlate data across multiple sources.
SRE Mindset: Minimum 5+ years of experience in an SRE, DevOps, or Systems Engineering role with a focus on high-availability systems.
Programming Proficiency: Strong coding skills in SPL, Go for building internal tools and automating workflows.
Distributed Systems: Deep understanding of Linux internals, networking (TCP/IP, DNS, Load Balancing), and container orchestration (Kubernetes/EKS).
Problem Solving: A data-driven approach to debugging complex, cross-service performance bottlenecks.
Bonus Skills (The "Nice-to-Haves")
Telemetry Standards: Hands-on experience with OpenTelemetry (OTel), Vector, or similar frameworks for instrumenting applications.
Charge-back app: Experience in implementing Splunk charge-back app for usage reporting
Cloud Platforms: Experience managing observability native tools within AWS or GCP.
Additional requirements:
This position requires the ability to access federal environments and/or have access to protected federal data. As a condition of employment for this position, the successful candidate must be able to submit documentation establishing U.S. Person status (e.g. a U.S. Citizen, National, Lawful Permanent Resident, Refugee, or Asylee. 22 CFR 120.15) upon hire.
This person must attend in person onboarding in our San Francisco/Chicago office the first week of employment.
Preferred
Experience with OpenTelemetry, Vector, or similar frameworks
Experience implementing Splunk charge-back app
Experience managing observability tools within AWS or GCP
