-
Design, develop, and maintain scalable batch data pipelines for transaction data processing and integration.
-
Build, enhance, and support data pipelines using Python, Spark, AWS Glue, and Databricks.
-
Develop complex SQL queries to extract, transform, validate, and analyze data.
-
Work with Snowflake for data storage, transformation, and analytical processing.
-
Use Apache Airflow for workflow orchestration, scheduling, and monitoring of data pipelines.
-
Troubleshoot pipeline failures, investigate data quality issues, and optimize performance.
-
Support the integration of one client's data into current client's existing data ecosystem.
-
Collaborate with engineers across multiple initiatives within a small, agile team structure.
-
For API-focused initiatives, develop and maintain APIs to serve data to downstream applications and systems.
-
Follow established engineering standards, data governance practices, and best practices for reliable data processing.
-
Data Engineering: 5 7 years of hands-on experience building and supporting data pipelines.
-
Programming: Strong Python programming skills.
-
Big Data: Hands-on experience with Apache Spark.
-
AWS: Experience with AWS Glue and related data engineering services.
-
SQL: Strong SQL skills, including complex queries, joins, aggregations, and data analysis.
-
Databricks: Experience developing and processing data using Databricks.
-
Snowflake: Experience working with Snowflake for data processing and warehousing.
-
Orchestration: Experience with Apache Airflow for scheduling and managing data workflows.
-
Data Processing: Experience with batch pipelines, ETL/ELT processes, data transformation, and pipeline support.