Responsibilities

In your first 90 days, map the current estate end to end: every source, pipeline, store, consumer and delivery path, with its volume, cost, owner and failure history. Publish a target-state data architecture and a sequenced migration plan that engineering and product leadership sign off.
Design and build the new ingestion and processing backbone. Replace bespoke pipelines with a standard pattern for streaming and batch ingestion, enrichment, and delivery, running at millions of documents per day with clear SLOs.
Lead the data platform workstream of our GCP migration. Turn "lift and shift" into real modernization: decide what moves as is, what gets rebuilt, and what gets retired, and own the technical design of each wave.
Rationalize storage and search. Define when we use the warehouse (BigQuery), search and index (Elasticsearch/OpenSearch), object storage, vector and graph stores, then consolidate toward that pattern and cut the cost of our most expensive workloads.
Build the data foundation for AI. Deliver agent-callable data services, hybrid retrieval (keyword, analytical, vector, graph), and provenance and lineage on every record so our AI products can cite their evidence.
Build the governance and compliance platform. Centralize and serve governed and compliant data (e.g. privacy laws) as part of AI-readiness, setting parameters, exclusions, and retention for optimal performance of AI.
Make the platform operable. Put in place data contracts, quality checks, observability and cost attribution so every pipeline has an owner, an SLO and a known unit cost.
What you will own
Pipelines and ingestion. Architecture and reference implementations for collection, ingestion, parsing, enrichment (translation, NER, entity resolution) and delivery. Event-driven and queue-based processing, idempotency, backfills and replay, schema evolution, and exactly-once or effectively-once guarantees where they matter.
Storage, compute and search. The physical data platform across warehouse, lakehouse, search and object storage. Partitioning, indexing, tiering and retention strategies at petabyte scale. Query and aggregation performance, and the hardware and cost trade-offs behind them.
Cloud migration and environments. Technical design for moving data workloads from AWS to GCP with minimal disruption to customers. Patterns that also work in FedRAMP, disconnected and air-gapped deployments, where managed cloud services may not be available.
Retrieval and AI enablement. APIs and query layers that serve both analysts and agents. Hybrid retrieval across structured, unstructured, vector and graph data, built with our AI and Graph teams to support RAG and agentic workflows.

Qualification

12+ years building data systemsYou still write production codeClear written communicationExperience with vector searchExperience with entity resolutionBackground in OSINTExperience building for FedRAMPExperience with data observabilityEducation and locationBachelor's degree in Computer Science

Required

12+ years building data systems, with at least 5 as the lead architect or principal engineer on a large-scale data platform.
You still write production code. Strong in Python plus at least one of Java, Scala or Go, and fluent in SQL.
You have personally led a major re-architecture or cloud migration of a data platform and can walk us through the design, the sequencing, what broke and what you would change.
Deep hands-on experience with distributed processing and streaming, for example Spark, Beam/Dataflow, Flink, Kafka, Pub/Sub or Pulsar.
Production experience with GCP data services such as BigQuery, Dataflow, Pub/Sub, GCS and Spanner. AWS experience is also expected, since you will be migrating from it.
Expert-level operation of search and index systems (Elasticsearch or OpenSearch) at scale, including cluster sizing, mappings, aggregation performance and cost tuning.
Experience with lakehouse table formats (Iceberg, Delta or Hudi) and orchestration (Airflow/Composer, Dagster or similar).
A track record of cutting platform cost materially without hurting performance, and the numbers to show for it.
Clear written communication. You can write a design doc that a CEO and a senior engineer both understand.
Ability to obtain a US security clearance

Preferred

Experience with vector search, embeddings pipelines and retrieval architectures for LLM and agentic applications.
Experience with entity resolution, knowledge graphs or graph databases, ideally across multilingual or messy real-world data.
Background in OSINT, national security or law enforcement, risk intelligence, and financial data
Experience building for FedRAMP, IL5 or air-gapped environments, including running data platforms without managed cloud services.
Experience with data observability, lineage and governance tooling (for example OpenLineage, Great Expectations, Monte Carlo, Dataplex).
Experience combining large volumes of scraped multimedia and structured third-party data.
Education and location
Bachelor's degree in Computer Science, Engineering or a related technical field, or equivalent hands-on experience. A Master's or PhD is a plus but not required.

Principal Data Architect at Babelstreet · LumiJobs

Principal Data Architect

Responsibilities

Qualification

Required

Preferred

Benefits

Company

Babelstreet

More roles at Babelstreet