Data Platform Engineer, Ingestion at Minerva · LumiJobs

Responsibilities

Every new data source starts as someone else's problem: an API with rate limits, a vendor file that arrives late and gets corrected twice, a schema that changes without warning. Today, adding a new vendor feed still takes hand-built work.
You make new sources dependable and make the next one cheap. You build the ingestion pipelines for our customer and vendor data, automate onboarding from configuration to first delivery, and turn what you build into modules other Minerva teams can use without running platform infrastructure themselves.
This is a mid-level individual contributor seat on the data platform team, reporting to our Head of Data Platform. Our data product team decides which sources matter and what good data looks like. You own how it arrives and how it stays dependable.
Build pipelines for messy sources. APIs, files, database extracts and approved scraping sources, at high volume, with inconsistent formats and schemas that change.
Make reruns safe. Incremental loads, backfills and retries that never duplicate or corrupt data. Bad records get quarantined, not loaded. Throughput and processing cost stay under control.
Automate onboarding. The path from source configuration and access checks through provisioning, validation, deployment and first delivery runs without hand-holding.
Build modules other teams use. Reusable connectors and storage components with tests, examples and clear interfaces, so a team building an API integration or a scraper gets managed ingestion, storage and monitoring without building them.
Keep the origin of every value. Carry source metadata through the pipeline (which vendor, which delivery, when) so we can always say where a value came from. Enforce privacy controls so restricted data can't come back through a new load or a replay.
Own your integrations in production. Connect each one to shared monitoring, write its recovery runbook and fix source-specific problems when they break.
Orchestration: Dagster, with Airflow still running the workloads that haven't moved.
Transformation: dbt and SQLMesh.
Storage: Snowflake, plus a lakehouse on Spark, Iceberg, Trino and AWS Glue.

Data Platform Engineer, Ingestion

Responsibilities

Qualification

Preferred

Benefits

Company

Minerva

More roles at Minerva