8451 Senior Data Engineer at 84.51° designs and develops Databricks and cloud-based data solutions. Collaborates with cross-functional teams to build reliable data pipelines and improve data engineering practices.
Responsibilities
Take ownership of features and drive them to completion through all phases of the entire 84.51° SDLC
Participate in the design and development of Databricks and Cloud-based solutions
Implement automated unit and integration testing
Collaborate with architecture and lead engineers to ensure consistent development practices
Provide mentoring to junior engineers
Collaborate within a cross-functional team of data scientists, Product partners, and engineers to translate analytical needs into reliable data pipelines and scalable solutions
Participate in retrospective reviews
Participate in the estimation process for new work and releases
Collaborate with other engineers to solve and bring new perspectives to complex problems
Drive improvements in data engineering practices, procedures, and ways of working
Embrace new technologies and an ever-changing environment
Prepare, validate, and organize governed data and metadata to support reliable analytics and downstream data consumers
Qualification
Bachelor’s Degree (Computer ScienceExperience with Pipeline OrchestrationExperience with CI/CDExperience with AzureExperience with Databricks Delta TablesExposure to BI Tooling (Tableau
Required
Bachelor’s Degree (Computer Science, Management Information Systems, Mathematics, Business Analytics, or STEM)
3+ years proven ability of professional Data Development experience
3+ years proven ability of developing with Databricks
3+ years of experience with PySpark/Spark
3+ years of experience developing with either Python or Java
Full understanding of ETL concepts and Data Warehousing concepts
Experience with Pipeline Orchestration
Experience with CI/CD
Experience with version control software
Strong understanding of Agile Principles (Scrum)
Experience preparing and validating data for downstream consumers, with an emphasis on data quality, governance, and clear metadata documentation.
Preferred
Experience with Azure
Experience with Databricks Delta Tables, Delta Lake, Delta Live Tables
Proficient with Relational Data Modeling
Experience with Python Library Development
Experience with Structured Streaming (Spark or otherwise)
Experience with Kafka and/or Azure Event Hub
Experience with Github SaaS / Github Actions
Exposure to BI Tooling (Tableau, Power BI, Cognos, etc.)
