Responsibilities

CTG is seeking a Lead Data Engineer to design, build, and maintain scalable, efficient data pipelines and systems following modern data engineering best practices. The Lead Data Engineer will partner with other Data Engineers to evaluate and prototype new tools and technologies, assess associated risks and benefits, and deliver exceptional value to our clients.
Design, build, and maintain scalable data pipelines, ETL/ELT workflows, and data models using Python, Apache Spark (PySpark), dbt, SQL (PostgreSQL), and AWS Glue.
Develop and optimize AWS-native data platforms leveraging AWS Glue, Amazon EMR, Amazon MWAA (Apache Airflow), Lambda, Step Functions, Amazon S3, Redshift, RDS, DMS, and CloudWatch.
Build high-performance ingestion, transformation, and orchestration workflows for structured and semi-structured data using Apache Iceberg, Parquet, ORC, and Avro.
Design and optimize analytical data platforms using Amazon Athena, Trino, Hive, OpenSearch, and enterprise data catalog technologies.
Integrate enterprise and external data sources across relational and NoSQL platforms including PostgreSQL, Oracle, Redshift, GraphDB, and other NoSQL databases.
Build AI-enabled data solutions using Amazon Bedrock, RAG pipelines, and vector search technologies including Amazon S3 Vectors and OpenSearch vector indexes.
Develop cloud infrastructure using CloudFormation (Infrastructure as Code), GitHub, Harness, and enterprise CI/CD pipelines while leveraging SNS, SQS, and EventBridge for event-driven architectures.
Improve the reliability, scalability, performance, and maintainability of enterprise data platforms through monitoring, troubleshooting, automation, and continuous optimization.
Support mission-critical analytics and reporting solutions within large-scale AWS-based federal data environments, implementing solutions that comply with FedRAMP and NIST SP 800-53 security controls.
Lead modernization initiatives migrating legacy platforms including IBM DataStage, Hadoop, Rundeck, and shell-based workflows to cloud-native AWS services.
Mentor junior engineers through technical guidance, architecture discussions, and code reviews while promoting engineering best practices.

Qualification

A strong communicatorCollaborativeBachelor's degree in Computer ScienceStrong hands-on experience withApache Spark (PySpark) requiredAWS GlueDeveloping scalable data pipelinesExperience supporting analyticsFull Time Employee Benefits

Required

A strategic data engineer who enjoys designing sophisticated systems and solving challenging problems
Strong experience in modern cloud-based solution design
Comfortable balancing business needs with technical constraints and long-term strategy
A strong communicator
Collaborative, proactive, and comfortable navigating ambiguity
Bachelor's degree in Computer Science, Engineering, or a related technical field
15+ years of professional experience in data engineering, data architecture, or related fields
Strong hands-on experience with:
Apache Spark (PySpark) required, Python, SQL (PostgreSQL), and dbt for large-scale data engineering, ETL/ELT development, data transformation, and data modeling.
AWS Glue, Amazon EMR, Amazon MWAA (Apache Airflow), AWS Lambda, AWS Step Functions, Amazon S3, Amazon Redshift, Amazon RDS, AWS DMS, and Amazon CloudWatch.
Developing scalable data pipelines, workflow orchestration, and data integration solutions across enterprise environments.
Working with modern data lake technologies and data formats such as Parquet, ORC, and Avro.

Preferred

Experience supporting analytics, data engineering, or modernization initiatives for financial regulators, capital markets, or other highly regulated environments is a plus.
Experience with Apache Iceberg and modern data lakehouse architectures.
Experience working with unstructured data processing, including document/text processing, embeddings, vector search, and LLM-based data solutions.
Exposure to integrating LLMs and generative AI capabilities into enterprise data pipelines and platforms.
Experience designing data architectures that support both structured and unstructured data at scale.
We are committed to offering a competitive salary for this position, with an estimated range of $150k to $200k annually. Please note that this range is intended to provide a general idea of what to expect; however, the final offer may vary based on experience, skills, and other factors. The stated range is not a guarantee and is subject to change.
Full Time Employee Benefits
Remote Work (Hybrid roles will be specified in the job post)

Lead Data Engineer at Capitaltg · LumiJobs

Lead Data Engineer

Responsibilities

Qualification

Required

Preferred

Benefits

Company

Capitaltg

More roles at Capitaltg