Design, build, and maintain scalable, reliable, and high-throughput data ingestion pipelines for structured and semi-structured data.

Implement robust and secure data lake and SQL-based storage architectures optimized for performance and cost.

Develop and maintain internal tools and frameworks for data ingestion using Python, Golang, and SQL.

Build and maintain inference pipelines to support model serving and near real time prediction workflows

Develop and maintain internal tools and deployment processes to support ML development workflow , including experimentation, testing, deployment and monitoring.

Collaborate cross-functionally with Cloud, Edge, Product, and AI teams to define data contracts, schemas, and retention policies.

Use AWS cloud infrastructure (including Argo Workflows, S3, Lambda, Glue, Kinesis, Athena, and RDS) to support end-to-end data workflows.

Employ Infrastructure-as-Code (IaC) practices using Terraform to manage data platform infrastructure.

Monitor data pipelines for quality, latency, and failures using tools such as CloudWatch, SumoLogic, or DataDog.

Build and maintain ingestion pipelines that land curated sensor and operational data into Delta tables on Databricks to support ML experimentation and model training.

Continuously optimize storage, partitioning, and query performance across large-scale datasets.

Participate in architecture reviews and ensure the platform adheres to security, compliance, and best practice standards.

Data Platform Engineer

Responsibilities

Qualification

Required

Preferred

Company

Bright AI

More roles at Bright AI