Responsibilities

Design, implement, and maintain scalable, fully serverless data pipelines on AWS using Lambda, Glue, Athena, Step Functions, and S3 to support reporting, analytics, and AI use cases.
Build and evolve data models and schemas that enable performant querying and downstream consumption by AI, Analytics and engineering teams. Build versioned data models, feature stores, with semantics, reproducible backfills, and safeguards against leakage or inconsistent definitions.
Develop ETL/ELT workflows to ingest, cleanse, transform, and load data from internal applications, third‐party sources, and event streams into our data lake and analytical layers.
Partner closely with Product, Underwriting, Analytics, and AI stakeholders to understand data requirements and translate them into robust data structures, contracts, and SLAs.
Implement data quality controls, monitoring, and alerting to ensure accuracy, completeness, timeliness, and lineage of critical datasets and features used by models. Include automated controls for schema change, validity, reconciliation, claims maturity, and model leakage.
Optimize serverless workloads for cost, performance, and scalability, including query tuning in Athena and efficient storage formats/partitioning in S3.
Contribute to and enforce data engineering best practices, including version control, code review, CI/CD for data pipelines, and Infrastructure as Code with AWS CDK.
Collaborate with AI and Analytics teams to design and maintain feature stores and other reusable data assets that accelerate experimentation and model deployment. Partner on batch or API scoring and capture model/data versions, recommendations, actions, overrides, and claims to close the learning loop.
Experience designing and operating event-driven or streaming data pipelines for near-real-time scoring or decisioning, using Kafka, Amazon Kinesis, or comparable technologies; strong understanding of event schemas, idempotency, ordering, retries, observability, and failure recovery.
Troubleshoot pipeline issues, resolve data‐related incidents, and provide ongoing support for production data workflows and model‐driven applications.
Document data models, pipelines, and data contracts, and help evangelize data literacy and self‐service analytics across the organization.

Qualification

Experience with MLOps practicesKnowledge of data qualityMinimum Requirements

Required

Strong experience with AWS data and serverless services (Lambda, Glue, Athena, S3, Step Functions or similar orchestration tools).
Advanced proficiency in Python (required) and SQL, with experience in relational and NoSQL databases and in building reusable ETL/ELT libraries. Hands-on experience with PySpark for distributed data processing is strongly preferred; TypeScript experience for AWS CDK required.
Experience with MLOps practices, including versioned feature and training datasets, orchestration of training and scoring workflows, CI/CD, model/data monitoring, and reproducible deployments in partnership with AI.
Solid understanding of data modeling principles for AI workloads. Apply schema evolution, incremental processing, partitioning, and columnar or open-table formats such as Parquet and Iceberg.
Experience designing and operating data pipelines at scale, including batch and near‐real‐time ingestion. Hands-on experience with Kafka is preferred.
Familiarity with SQL and query optimization in columnar data stores and engines.
Knowledge of data quality, governance, and security for sensitive healthcare and financial data, including least-privilege access, retention, classification, tokenization, and de-identification.
Hands‐on experience with Infrastructure as Code, preferably AWS CDK, for provisioning and managing data infrastructure.
Ability to partner with AI and Analytics teams to design data solutions for model experimentation and production; Kedro or similar workflow experience is a plus.
Knowledge of responsible AI and data-use practices, including permitted-use controls, privacy, fairness considerations, documentation, and governance for sensitive healthcare and financial data.
Strong communication skills to explain complex data concepts clearly to business stakeholders; regulated-domain experience, preferably healthcare claims or insurance, is a plus.
Minimum Requirements:

Senior Data Engineer at Paretocaptiveservicesllc · LumiJobs

Senior Data Engineer

Responsibilities

Qualification

Required

Benefits

Company

Paretocaptiveservicesllc

More roles at Paretocaptiveservicesllc